Common DFOS tools:
Documentation

make printable | back to: phoenix description · process instances

Find here a detailed overview of the characteristics of the MUSE PHOENIX process. The focus is on specific properties for MUSE. The standard PHOENIX process is not covered here, find it here. Find the phoenix tool documentation here.

Phases of phoenix process:

• pre-processing
• processing (phoenix -P)
• certification (phoenix -M)
• post-processing

Pre-processing

MUSE datacubes are big (3-5 GB for single, 3-50 GB for combined ones). Their production is complex, it requires significant resources and compute time. In order to maximize the usefulness for their users, a certain amount of non-standard features is part of the PHOENIX process. The most prominent feature is the non-standard association scheme (called "smart association" with non-OCA rules), combined with the non-standard creation of the science cascade (going beyond what is possible with createAB). You may want to read more about this feature here before you read this section.

The smart association scheme is implemented with the tool phoenixPrepare_MUSE. The non-standard creation of science ABs is implemented with the phoenix pgi called pgi_phoenix_MUSE_AB.

phoenixPrepare_MUSE

This is a semi-automatic, command-line tool. It is called on the command line, either by date or by month:

phoenixPrepare_MUSE -d <date>

phoenixPrepare_MUSE -m <yyyy-mm>

For a given date, it analyzes the entire set of science headers, evaluating the keys container_id, container_type, obs_id, tpl_start, obs_start, the dpr keys, and keys related to the differential offsets and the rotator angles. It analyzes all datasets defined by the container keys, the obs_id and tpl_start.

First, it checks for containers of type C (concatenated). These might contain two or more linked OBs which would then be evaluated together (this goes beyond the OCA rules). Then it checks for repetitions of the same OB (same night of course), which sometimes happens (both for VM and SM). Finally, it analyzes multi-template OBs, for the operator to decide if they should get combined (stacked) or not (mapping) (more on these patterns here ...).

The relative acquisition pattern of each OB (or container), based on the differential offset values, is visualized by a small MIDAS script.

Display of acquisition pattern for a given OB, to decide about 'smart association'. The plot displays the centre of each 1x1 arcmin^2 FOV (red dot), plus its boundaries. The example OB has two templates, therefore two ABs. Based on this plot, they get merged into one AB because there is sufficient overlap.

The design of the tool is such that it takes the science ABs as created by the DFOS account, from http://qcweb.hq.eso.org/MUSE/logs/2014-12-14/. These are either scibasic ABs (single file) or exp_combine ABs (tpl).

After interaction with the ABs (with or without changes), the resulting ABs are uploaded to a new branch, http://qcweb.hq.eso.org/MUSE_smart/logs/2014-12-14/, for permanent storage. This is where they are later expected by the phoenix tool. Any modification done to the ABs applies to the input datasets only (set of RAWFILEs). All associations to master calibrations are left unchanged. The MUSE_smart directory is a "buffer" to become independent of the DFOS storage, while having a permanent storage of the modified ABs for future reference. All science ABs are stored in the MUSE_smart tree, no matter if they are modified or not. When phoenix is called, it will download the ABs from the MUSE_smart directory. After finishing it will store the ABs under the final tree http://qcweb.hq.eso.org/MUSE_R/logs/2014-12-14/ .

The tool tries to analyze the dataset patterns and offers to the operator a number of (context-aware) actions, among them

• merging of datasets (multiple tpl_start from one OB: could be useful to merge);
• splitting of datasets (depending on the offsets, splitting of a tpl-based AB into individual ABs might be reasonable);
• re-sorting datasets (two executions of the same OB, both have partly good, partly not so good input files);
• individual deletion or editing of ABs;
• candidates for 'hidden sky' OBs and editing of the OBs according to fixed rules.

These are the most common actions:

All edits are logged in a simple-minded logging system for future reference, in phoenixPrepare_MUSE_2014-12-14.log.

The tool is designed as an "expert system", with more options to be added relatively easily if needed.

As the last step, the tool will transfer, after confirmation, all ABs and ALOG files (no matter if edited or not) to the MUSE_smart tree on qcweb.

Options:

Note:

• You can exit the tool anytime before the last step, without impacting the MUSE_smart tree.
• Normally you will run the tool in monthly mode. This is safe since then no date can be missed, and your treatment of the possible cases is likely more consistent than in daily mode. However this might take a while.
• This pre-processing step is mandatory for MUSE IDP processing. If it is not done, there is no output in the MUSE_smart tree. The phoenix tool expects all ABs there and will assume that no science ABs are available for that night!
• It is recommended to use the option -B to display the data report; you can then use the filter function on the data report to display a particular OB, or CONTAINER.
• If the tool runs an error (file with AB list is empty), it is likely that the headers are incomplete. Then, delete $DFO_HDR_DIR/<date> and call the tool again.
• It might be helpful to check for already processed OBs for a given run_ID. Use the QC1 plotter http://archive.eso.org/qc1/qc1_cgi?action=qc1_plot_table&table=muse_sci_combined for a selected run_ID, select RA and DEC, and see all so far processed pointings for that run.

SMART association

The principle of SMART association is to review the DFOS ABs, in order to maximize the usefulness of the datacube product. This feature is provided by the pre-processing tool phoenixPrepare_MUSE which must be run before phoenix.

In many cases the standard input data set (RAWFILE) which is based on TPL_START is not changed, but there are typical patterns when it is better to optimize the ABs. Here is an overview of the most common cases.

	Case #1: 2 OBs linked by the same CONTAINER ID (type C).
	Case #2: 1 OB, more than 1 template. Depending on jitter offsets, one might want to merge all tpl ABs.
	Case #3: 1 OB, 1 template. Depending on jitter offsets, one might want to split the tpl AB into smaller datasets, or even delete it (provide single datacubes only) if there is no overlap at all.
	Case #4: 1 OB, 1 template. Data come in twins. One may want to split by RA and/or DEC. (This option is likely to be obsolete. With the option to process up to 16 input files, an N=16 combined datacube might still be preferrable over say 8 combined datacubes, each of them made from 2 input files.)
	Case #5a: 1 OB, 2 executions, the SM case. Based on OB grades and comments, it is often possible to identify the bad file(s) and the good files. By editing, one can create one good tpl AB and one not so good (but still useful) (tpl or single) AB.
	Case #5b: 1 OB, 2 executions, the VM case. Typically no OB comments are available. A combination of all executions into a single tpl AB is likely the best option.

Processing: pgi_phoenix_MUSE_AB

When working with phoenix for MUSE, always work on single dates. Typical execution times for a full (data) date are fractions of a full (calendar) day. The typical call is

Because of the interactive certification step for the MUSE IDP process, it is not possible to call the phoenix tool without option (like phoenix -d 2014-12-14).

The phoenix tool works with standard dfos components plus with instrument-specific PGIs. For MUSE the first PGI is

It is needed to create the full science AB cascade with 5 recipes and processing steps. In particular:

• all single SKY ABs (extension .ab, recipe muse_scibasic) are duplicated into a scibasic AB (extension .ab) and a create_sky AB (extension _sky.ab);
• all single OBJECT ABs (extension .ab, recipe muse_scibasic) are duplicated into a scibasic AB (extension .ab) and a scipost AB (extension _pst.ab);
• furthermore, all single SKY ABs are copied to a scipost AB (extension _pst.ab);
• all tpl ABs (only OBJECT) are duplicated into an exp_align AB (extension _tal.ab) and an exp_combine AB (extension _tpl.ab);
• if applicable, tpl ABs are created for SKY files and then duplicated into an exp_align AB (extension _tal.ab) and an exp_combine AB (extension _tpl.ab).

The different types of ABs can be recognized by their extension.

Depending on the number of input frames, and on their type, there are four different cases:

a) single OBJECT frame, no SKY:
	Output of PGI: one scibasic AB, and one scipost AB.
b) N=4 input OBJECT frames, no SKY:
	Output of PGI: 4 scibasic ABs, 4 scipost ABs, one exp_align AB, one exp_combine AB.
c) N=4 input OBJECT frames, 1 SKY:
	Output of PGI: 4+1 scibasic ABs, one create_sky AB, 4+1 scipost ABs, one exp_align AB, one exp_combine AB. The additional SKY scipost AB will create a shallow datacube.
d) N=4 input OBJECT frames, 2 SKY frames:
	Output of PGI: 4+2 scibasic ABs, 2 create_sky ABs, 4+2 scipost ABs, 2 exp_align ABs, 2 exp_combine ABs. The SKY ABs will create a shallow combined datacube.

The creation of scipost, exp_align and exp_combine ABs for SKY files is based on the idea that these are valid pointings and contain science photons ("serendipitous science").

In the process of AB creation (from the "simple" set of ABs delivered by phoenixPrepare_MUSE), the PGI manages

• the recipe name,
• the input dataset (knows if the dataset is a set of raw files or of intermediate products),
• checks for certain static calib files and adds them if not found,
• for others it checks the existence and sends a notification if something is missing,
• manages the processing parameters.

It also manages the content of the ALOG files (for consistency only), and detects and suppresses configured cases of ultra-short exposure times (<10 sec) or excessively high number of input files (> 16 currently).

It does not create new associations.

After execution all ABs are in the standard $DFO_AB_DIR, ready for execution.

Processing: pgi_phoenix_MUSE_stream

The second important PGI is pgi_phoenix_MUSE_stream which is configured as JOB_PGI. This PGI manages the orchestration of the AB execution. Like the AB_PGI implements association functions that are not available in createAB, this JOB_PGI implements dependency functions that go beyond what is possible with createJob.

MUSE AB processing executes without condor. Instead, the pipeline uses internal parallelization, with 24 threads equivalent to the 24 fits file extensions (24 spectrographs). On a powerful machine like muc10 or muc09 (with 48 or 32 physical cores each), it is possible to run 2 such ABs in parallel ("2 streams"). On muc10 one can even go for 3-4 streams. Because of the logical dependencies, the execution scenario is the following:

For performance reasons, independent ABs are executed in 2 or more streams whenever possible (and safe). For a complete daily batch, the ABs are sorted and executed in the following manner (time from left to right):

type of ABs:			scibasic OBJECT .ab	scipost _pst.ab
	scibasic SKY .ab	create_sky _sky.ab			exp_align _tal.ab	exp_combine _tpl.ab
			scibasic OBJECT .ab	scipost _pst.ab
job file:	execAB		JOB_MANAGER		execAB
time			[split]		[merge]

If there are no SKY AB, we start with the OBJECT ABs in 2 or more streams. If there are no combined ABs, we finish with the streams of scipost ABs. There are always the 2 or more streams for the OBJECT ABs.

There are two critical points in the flow: the splitting into multiple streams after the _sky ABs ('split point'), and the merging into one stream before the _tal ABs ('merge point', see above). This is controlled in the following way:

• the initial SKY ABs are listed sequentially in the execAB file;
• the first 2 scibasic OBJECT ABs are called with the first one in the background (stream #1) and the second one in the foreground (stream #2); this is the split point; if more streams can be supported, the schema is extended correspondingly;
• once the first one finishes, there is a post-PGI of processAB that calls the next similar AB from a work list of ABs called JOB_MANAGER_<date>; before calling it, its entry in the JOB_MANAGER file is removed;
• for the second AB the same happens: it calls the next AB from the JOB_MANAGER list (deleting its name before starting to process);
• this continues in 2 (or more) streams, until the JOB_MANAGER file is found empty;
• the streams are sent to sleep until the last stream reaches this point; then the merge point is reached;
• only then the execAB list takes over again and finishes the tal and tpl ABs sequentially.

This last transition to single-stream processing is done for safety: we do not precisely know how many input pixeltables are going to be combined in the last step. Per input pixeltable, about 20-25 GB memory is required. For combing N=4 pixeltables (the standard case) we would need up to 100 GB which would be safe in 2 or even 3 streams (both mucs have 512 GB each). But there are occasionally larger jobs, hence we stay safe and process only one stream. (If the machine runs out of memory the job eventually fails, after hours of swopping.)

As said above, the number of supported parallel streams differs for the operational machines. It is configured in config.phoenix as 'MULTI_STREAM' and is 2 on muc09, and 3 on muc10 (but check for the correct values there).

In order to get familiar with the processing pattern, call 'phoenix -d <date> -C' which not only gives you the complete set of ABs, but also the execAB and the JOB_MANAGER files.

The sequence of QC jobs is listed in the standard way in the execQC file. There is one job per pst and tpl AB. They are executed in sequence. Their execution time is about a minute each and does not need optimization.

What if:

-- ... the tal (exp_align) AB fails, causing the tpl (exp_combine) AB to fail:

• comment out the MCALIB and SOF_CONTENT sections in the tal AB (this to document the failure);
• comment out the OFFSET_LIST in the MCALIB and SOF_CONTENT sections in the tpl AB;
• if you want to be super-correct, do the same in the ALOG files;
• call the tpl AB again, on the command line (processAB -a <ab>);
• don't forget the QC report afterwards (processQC -a <ab>);
• this is the correct procedure since the alignment is actually very often not needed, so that dropping it very likely has no negative impact; inspect the result: if you see double sources, delete the product datacube and deliver the individual datacubes instead (will happen automatically if you reject the tpl AB).
  
  
• If this happens to a SKY AB, it's probably not worth retrying: just reject the tpl AB.

-- ... a set of scibasic or scipost ABs failed because of NGAS access problems:

• do a trial download to check ngas availability; if it fails again, notify AOG;
• if it works, go to $DFO_JOB_DIR and start the script 'mcalDown_SCIENCE_<date>'. It is managed by phoenix and controls the mass download of mcalibs for a given date. It goes through all download calls and downloads the missing files. It recognizes if a file is already available and doesn't download it again.
• If there was an issue with raw files, do the same with the script 'rawDown_SCIENCE_<date>'.
• Once finished successfully, start the AB processing again; this is a non-standard scenario and might involve manual calls since the JOB_MANAGER might be empty already; use the execAB file as reference: it has all processAB calls, some of them commented out;

Don't:

• Don't execute a tpl AB on the command line while another phoenix processing job is running. This could exhaust your memory and cause both processes to fail.
• Don't call processAB of scibasic and pst ABs while the corresponding JOB_MANAGER file is still filled: this would call a stream of ABs being processed after your AB. If necessary, move the JOB_MANAGER file to a different name, execute your AB, and then move back.
• Don't call a pst AB without previously calling its corresponding scibasic AB. Think of both ABs as 'coupled particles' because the output of the scibasic AB (24 single pixeltables) is fed into the pst AB and then deleted (by the post-processing PGI in processAB, to save space). Hence they can only be processed sequentially.
• If you run a pst AB manually (on the command line), make sure to delete any JOB_CONTINUE files in $DFO_JOB_DIR afterwards. These are used to manage the 2-stream processing and mark a merge point (see above). They are created by the post-processing pgi of processAB and make proper sense only within the normal phoenix calls. In the most extreme case they could delete a merge point and cause a running phoenix instance to start the tpl ABs without waiting for the pst ABs to finish.
• Don't call processQC of a tpl AB without previous calls of processQC for the individual pst ABs. The QC script needs some QC parameters of the pst ABs to determine the PATTERN. It will likely become 'UNKNOWN' (which is not an issue, just a bit nasty).

Certification: phoenixCertify_MUSE

After finishing the 'phoenix -P' call, the products are (as usual) in $DFS_PRODUCT, waiting for certification. This step is special for MUSE, no other IDP process has a strict review phase. It is deemed necessary because there are potential issues:

• the algorithm of the alignment recipe might fail, or might be confused by spurious sources, with false alignment and double sources (this happens rarely and more often to SKY datacubes than to OBJECT datacubes);
• occasionally an input exposure has lost its guide star, with no sources visible or with trails; this symptom becomes immediately visible by visual inspection but is impossible to detect numerically.

The certification tool phoenixCertify_MUSE is configured as CERTIF_PGI. It is a clone of certifyProducts, customized to the MUSE IDP needs. It can be called stand-alone on the command line, but normally it is called by the third step of phoenix, after processing:

It has the same workflow as the dfos tool. It has some hard-coded configuration, like RANDOM mode (20%) for the SCI_SINGLE products (from pst ABs), and ALL mode for the PIXTABLE_REDUCED products (from tpl ABs). Since the review is score-supported, the RANDOM mode means exactly the same as under DFOS: at least 20% of all products are reviewed, plus the non-zero scores, plus the failures. There is the usual comment mechanism. Note that the products of the tpl ABs have already auto-edited their PATTERN as QC comment (acquisition pattern). It is not possible to jump, but QC comments can be entered anytime using the 'pgi_phoenix_MUSE_getStat' link on top of the AB monitor.

To assist with the review, the OB comments and grades are displayed throughout the certification.

Apart from the PATTERN, there are auto-comments on the AB monitor if the pipeline log contains WARNINGs or ERRORs, and if the datacube is a SINGLE one. (These auto-comments are meant for information; they don't get propagated but are overwritten by the certification procedure.) Please check the logs for better understanding. The following standard ERRORs are known:

• failure of master sky fit (ERROR): this happens during scipost, for OBJECT frames without external SKY. If the recipe then runs into numerical problems (usually the iteration does not converge), it aborts that step and continues. The QC script catches this error and marks it in the score. There is nothing to do.
• no valid seeing values found (WARNING): headers sometimes have the seeing values missing, the pipeline then assumes a dummy value and marks this by a warning. This is not an issue.

If you encounter a PATTERN 'unknown', the QC script has not found a proper value for the observed pattern. Though this is not an issue, you may want to improve the script and make the new pattern known to it:

• check the coversheet HTML file for the values of COEFF_ALPHA, COEFF_DELTA, POSANGLE;
• add the code for recognizing the new pattern (use ROTATE/DEEP/MAP);
• edit the .tab file of the AB: remove the last entry XYZUnknXYZ by COMMENT_NONE;
• call 'processQC -a <ab>', should be ok now;
• also call the comment PGI on the AB monitor and check the entry (remove UNKN).

If you want to enter a comment about a specific pst AB, use the 'pgi_phoenix_GEN_getStat' button on top of the AB monitor. Because of the hard-coded 20% fraction of reviewed ABs, there is no guarantee that the certification tool will offer that specific AB for review (unless it scores red of course).

Certification: the QC system

The MUSE QC system has the following components:

• QC procedures;
• QC parameters, QC databases;
• Scoring system;
• QC plots.

All follow the standard DFOS scheme.

QC procedures. There are two QC procedures (both under $DFO_PROC_DIR):

• qcpar_PIXTABLE_OBJECT.sh (for products of pst ABs),
• qcpar_PIXTABLE_REDUCED.sh (for products of tpl ABs)

(note: the names are historical, they have no proper meaning). Both are largely similar but differ when dealing with the additional properties for the combined datacubes (products of tpl ABs). They feed the QC parameters into the QC1 databases, derive the score bit, and call the python procedure for the QC plots. The qc1Ingest calls for the QC1 parameters are stored locally, under $DFO_PROC_DIR, as qc1_PIXTABLE_OBJECT/REDUCED.dat and qc1_sources.dat. There is also a technical table qcpar_PIXTABLE_OBJECT.dat that is used for the PATTERN recognition (originally it had other functions but these faded away).

QC parameters, QC databases. The QC parameters are generally speaking all kinds of parameters that might be useful, not only for QC, but also for process monitoring (e.g. PATTERN, score_bit), research on science cases (PROG_ID, abstract links) etc. They are fed into three QC1 databases:

• muse_sci_single
• muse_sci_combined
• muse_sci_sources.

The first two are rather similar in structure and collect data for the single and combined datacubes. The third one collects parameters for every pipeline-identified source and could be used to monitor alignment quality etc.

Scoring system. The are two aspects of the scoring system:

• the scores as known in the DFOS system
• the score_bit string as written into the product headers and available to the users.

There are 7 scores for single datacubes (pst ABs):

• the size of the datacube (3 values; any anomaly here indicates that the recipe run was somehow disturbed);
• num_sat: number of saturated pixels;
• lshift_max: maximum amount of differential wavelength correction applied from the sky lines (read from the product headers of the scibasic products which are not available in subsequent products);
• qc_deltatime: indicates if the associated wavelength solution is not from the same day;
• TPL_NEXP: number of frames in the input dataset (alert if 1 or >16).

There are 4 scores for combined datacubes (tpl ABs):

• offset: indicates if no offset correction was applied at all, or if the correction was larger than normal;
• qc_deltatime: see above;
• TPL_COMPLETE: indicates if template was aborted or resumed (anomaly in the dataset);
• PROC_COMPLETE: indicates if all files in the input dataset have actually been processed.

Since the MUSE pipeline is very "tolerant" with failures at intermediate steps (where other pipelines would just fail), it seems useful to monitor e.g. the size of the output datacube, or check for complete processing.

While the output of the scoring system is used for certification, and is stored locally but not exported to the user, the score_bit is used to give the user some indication about the data quality. It consists of 11 binary values (0 or 1) coding product properties in a very concise way. They are documented here and in the release description. They are automatically derived by the QC system and written into the product headers as key QCFLAG.

QC plots. Each product datacube gets its QC plot which is ingested into the phase3 system and gets delivered to the user. They are created by the python routine muse_science.py under $HOME/python.

Post-processing: pgi_phoenix_MUSE_moveP

After certification the standard phoenix workflow takes place (calling moveProducts). There is a pgi within moveProducts, pgi_phoenix_MUSE_moveP, that controls the complex final distribution of the datacube products:

• Combined datacubes (no matter if from OBJECT or SKY) go to $DFO_SCI_DIR, to become IDPs with names MU_SCBC (OBJECT) or MU_SCBY (SKY). Their associated collapsed white-light views (type IMAGE_FOV) also go there (files MU_SIMC or SIMY). All individual datacubes that made it into a combined datacube are NOT delivered as IDP, but they get stored in a separate tree, $DFO_SCI_DIR/DPING/<date>, for later ingestion with dpIngest. Associated graphical and log files are stored in the standard way. All other related files (intermediate SKY products, and PIXELTABLES) are deleted.
• If for a single datacube no COMBINED datacube exists (SINGLE by design, or SKY single datacube, or the combination failed or the COMBINED datacube was rejected), the SINGLE datacube becomes an IDP. Then it also goes to $DFO_SCI_DIR, to become an IDP with name MU_SCBS (OBJECT) or MU_SCBX (SKY). Its associated collapsed white-light view (type IMAGE_FOV) also goes there (files MU_SIMS or SIMX). Nothing goes to $DFO_SCI_DIR/DPING/<date>. Associated graphical and log files are stored in the standard way. All other related files (intermediate SKY products, and PIXELTABLES) are deleted.

This delivery plan makes sure that all final datacubes (last processing step) are delivered as IDPs, that all combined datacubes have the individual datacubes downloadable on demand, and that all IDPs have a preview file (IMAGE_FOV).

After phoenix: conversion tool idpConvert_mu and ingestProducts

After phoenix has finished, all science products are in $DFO_SCI_DIR, all logs in $DFO_LOG_DIR, and all graphics in $DFO_PLT_DIR. This is the standard architecture also known from DFOS.

The final call is 'ingestProducts -d <date>' which for MUSE has three parts:

• ingestion of files in $DFO_SCI_DIR/DPING/<date> with dpIngest,
• conversion of IDP headers with idpConvert_mu,
• phase3 ingestion of IDPs and ancillary files with IngestionTool@call_IT.

In order to prepare for IDP ingestion, the single datacubes need to be dpIngested (if applicable), and IDP headers need to be modified. This is the task of idpConvert_mu. (Every IDP process has such an instrument specific tool, with the instrument coded in its name).

The tool checks first if

• there are single datacubes in DPING to be ingested;
• if yes: have they already been ingested? YES --> ok, skip; NO --> start calling dpIngest for each datacube; each datacube takes roughly 2 minutes to dpIngest.

The ingestion log is in the usual place known from dfos: $DFO_LST_DIR/list_ingest_SCIENCE_<date>.txt.

The files get ingested as 'S.' files. You can check their existence by calling 'productExplorer -d <date>'. They will display in blue.

These files are ingested 'just in case', for an archive user who discovers issues with the combined datacube and then wants to inspect the individual ones. Their names get registered in the header of the IDP, before ingestion.

Next, the conversion tool adds ASSOC, ASSON and ASSOM keys to define the IDP dataset, consisting of the IDP (datacube), its ancillary fits file (IMAGE_FOV), ancillary text file (pipeline log), and ancillary graphics (QC reports). See more in the release description.

After finishing, the tool has

• created $DFO_SCI_DIR/<date>/conv,
• moved all IDP fits files and ancillary files (IMAGE_FOV) there,
• ingested the single datacubes if existing in $DFO_SCI_DIR/DPING,
• filled a text file CONVERTED which is used by ingestProducts to recognize the data as converted.

The products are then ready to be phase3-ingested with ingestProducts.

The call of ingestProducts and, afterwards, cleanupProducts is usually done in the same way as known from DFOS_OPS:

• there is the job file JOBS_INGEST (linked to the dfoMonitor) that is filled by phoenix with all new ingestion jobs; just call it;
• after successful ingestion, there is a new entry in the cleanup job file which you then also call, upon your convenience.

The successful ingestion is displayed on the phoenixMonitor.

Note that for MUSE there is the CLEANUP_PLUGIN needed (pgi_phoenix_MUSE_cleanup) that

• manages the proper replacement of all ingested fits files (IDPs, ancillary, S-dot files) by their headers;
• manages the transfer of the headers from the SLAVE machine to the MASTER machine in case of multi-machine processing.

Other little tools

There is a number of little tools for all kinds of special tasks. If possible they were configured with standard mechanisms in the standard DFOS tools or in phoenix. Their naming scheme follows a standard when they are specific for phoenix: they start with pgi_phoenix, then follows either MUSE or GEN. Unless otherwise noted, they are stored in $DFO_BIN_DIR.

Multi-instance processing (MASTER/SLAVE model)

Even for the 48-core machine muc10, the MUSE datacube production line is very time-consuming and demanding. The data processing for a day worth of MUSE science data might take half a day or more of machine time. This is very different to all other IDP processes (the other extreme being UVES where the whole IDP dataset could in theory be reprocessed over a long weekend).

Hence a multi-instance processing model has been developed which uses more than one muc machine (muc10 and muc09; at times also muc11, muc12):

The accounts work largely independently (you can of course run a phoenix job on muc09 and another one on muc10). But when it comes to store the output (headers of fits files, logs, plots, statistics), we use a MASTER/SLAVE model: all information is ultimately stored centrally, on the MASTER, so that the phoenixMonitor running there collects the information about the entire process (no matter if processed on the MASTER or on any SLAVE), just like on any other 'normal' IDP account. This means that information is scopied from the SLAVE to muc10 (MASTER) (but never in the other direction).

Software. All accounts share the same tools and configuration files (with a few keys filled with account-specific information). The accounts are coupled via the ssh mechanism to easily exchange information.

All accounts run a daily cronjob of dfosExplorer. It is the responsibility of the operator to keep them up-to-date in the same way. The non-dfos tools (MUSE-specific pgi's) are not covered by this mechanism. They need to be carefully kept in sync by the operator. Good practice is to use muse_ph2@muc10 as the logical MASTER: any changes done here are then scp'ed to the SLAVE account.

Both accounts run a weekly cronjob of dfosBackup.

Output. The phoenix processes run independently up to the step of finishNight, after which the SLAVE scopies logs and plots to the MASTER (same directories on both accounts). This is important for central bookkeeping via phoenixMonitor: only the instance running on the MASTER has the full knowledge and is allowed to export to qcweb.

Ingestion. All fits files are kept in the respective $DFO_SCI_DIR directories (copying would take a long time), and ingest from there. Once ingested, the fits files are ultimately deleted by cleanupProducts. At that point the headers are transferred from the SLAVE to the MASTER.

The config file of phoenix is used to define the roles of the account (keys IS_SLAVE and MASTER_ACCOUNT).

Scheduling. Unless for other IDP instances, there is no explicit scheduling mechanism for the phoenix jobs of MUSE (because the monthly mode cannot be used for MUSE). A MUSE phoenix job is always launched manually. It is the responsibility of the operator to decide which dates are to be launched. There is a notification mechanism implemented on the MUSE DFOS machine muc09, that adds a JOBS_NIGHT entry on muc10 when a night with SCIENCE data is dfos-finished. This is inteneded as a reminder mechanism that eventually the corresponding phoenix should be executed.

Other links

History: http://qcweb/MUSE_R/monitor/FINISHED/histoMonitor.html

Release description: under http://www.eso.org/sci/observing/phase3/data_streams.html.

Monitoring of quality: automatic scoring system, with a dedicated qc1 database table. See also here.

Terms and keywords: