make printable	New:	see also:
	v4.0: - supporting COMPLETE_MODE as a special IDP mode (create all mcalibs as needed, and then process SCIENCE) v4.1: - supporting SCIENCE AB creation instead of qcweb AB download, in normal IDP mode, beyond a certain DATE [documented in red]	databases none dfos tools calling createAB, createJob, processAB, processQC, moveProducts, ingestProducts output pipeline products, QC plots upload/download down: ABs from qc@qcweb up: logs and plots to qc@qcweb
topics: description | workflow | technical implementation | log | output | usage | pseudo dates | OCA and data pool | release | QC procedures | history scores, qualityChecker, autoCertifier | job execution | AB pre-selection | QC job concentration | COMPLETE_MODE | SCIENCE AB creation | ingestion | installation | config | PGI's | workflow | data directories | qcweb | notification | DON'T

Description and process

The phoenix tool is the workflow tool to support the automatic and efficient processing of large homogeneous science datasets into IDPs. (IDP is the acronym for science data products produced in-house, in contrast to the science data products produced externally by PIs which are called EDPs.)

The phoenix tool supports the following modes:

• classical OB-based IDPs,
• reprocessing of master calibrations,
• DEEP IDPs (combined across OBs),
• reprocessing of CALIBs as needed for IDPs, plus the IDPs (COMPLETE mode).

Depending on mode and configuration, the processing is based on

• certified and ingested master calibrations which went through the classical QC processing, scoring and certification workflow (separate processing that might have occurred years before the IDP processing); OR
• new master calibrations created together with the IDPs (COMPLETE mode);
• existing associations (stored on the qcweb QC server); OR
• new associations (created by calSelector).

The benefits of phoenix processing are:

• the homogeneous processing of a large dataset with a mature and reviewed pipeline;
• best-possible processing parameters;
• extraction of QC parameters for quality assessment.

DEEP mode: The meaning of DEEP combination is "to combine datasets across OBs". This is usually intended by the PI for going deeper than possible in a single OB (which is constrained by the 1 hr rule). A pre-selection of input datasets needs to be done by special preparation tools, because this is conceptually not part of the standard (DFS or DFOS) workflows. Usually the datasets need to be calibrated one by one, before they can then be combined in a final step ("step-2 processing"). This puts some requirements on the architecture of the pipeline. The second step might also be provided by a generic pipeline (like stacking by the HDRL pipeline).

The internal organization of the processing is also different from the normal (OB-based) processing, because it needs to be done by runID and not by date.

Workflow description (see also 'Operational workflow' here)

The IDP process has three components:

• the preparation (pre-processing) component: download or create the ABs, scan them, define/re-define input datasets (currently for MUSE and MUSE-DEEP only), covered by phoenix and its pgi's;
• the processing component, covered by phoenix;
• the post-processing (ingestion) component covered by a conversion tool (for header manipulation, or DFS reformatting tools like uv2p3) and the DFS ingestionTool, both plugged into the standard DFOS ingestProducts.

The phoenix processing workflow has many similarities with the autoDaily workflow.

A) For IDPs (OB-based or DEEP), the phoenix workflow has the following components:

While this is the basic processing step, the typical processing pattern in IDP mode is by month. An exception is the daily mode for MUSE IDPs. The DEEP mode is run by run_ID, encoded as pseudo-date.

B) For MCALIBs, the phoenix workflow has the following components:

C) In COMPLETE mode, the phoenix workflow has the same logical steps as under A), with the modified behaviour that all AB-based workflow steps are executed first for the CALIBs and then for the SCIENCE ABs. The SCIENCE ABs are recreated with calSelector/createAB. The CALIB ABs are created and executed only for those calibrations that are required for processing the SCIENCE ABs. Other than in daily QC, the creation of the CALIB ABs is not strictly day by day, but governed by the date of the SCIENCE ABs.

Find below a flow diagram that might also help to understand the processing workflow [click for full-size]:

Technical implementation

A) Workspace.

B) Data directories

C) Tools

The PHOENIX process requires a tool set which consists of a stripped-off version of the DFOS tools, and of some tools dedicated to the PHOENIX workflow. The most important key in the .dfosrc file of a PHOENIX installation is the key THIS_IS_PHOENIX which is set to YES for a PHOENIX installation (while it does not exist for a DFOS installation).

For a PHOENIX MCALIB installation, you also need the key THIS_IS_MCAL set to YES.

The installation is monitored by the standard tool dfosExplorer which is reading THIS_IS_PHOENIX and, if YES, downloads a reference file appropriate for PHOENIX (and different from a DFOS installation). The DFOS tools required for a PHOENIX installation are all evaluating the key THIS_IS_PHOENIX.

Only a subset of DFOS tools (blue in the following table) is needed for PHOENIX, plus some special tools (red):

D) Multi-instance processing (MASTER/SLAVE model).

For heavy processing tasks, it might be useful to have multiple phoenix accounts on different machines. This is currently the case for MUSE, where historically up to 4 accounts on 4 machines were used. All accounts use the same tools and configuration (with the exception of a few keys that specify accounts).

All accounts can run independently, from AB downloads up to ingestion. Only when it comes to the final step (cleanupProducts), it is important to join the bookkeeping information (so far by account) on a central installation. This is called the MASTER. The other account are called the SLAVEs.

To define the SLAVE, use the keys IS_SLAVE, MASTER_ACCOUNT, SLV_ROOT and MST_ROOT in config.phoenix.

To define the MASTER, no additional phoenix configuration is necessary (but for config.phoenixMonitor).

Logs

The process log is created per standard execution (-d <date>). It goes to the command line and is stored under $DFO_MON_DIR/AUTO_DAILY/phoenix_<date>.log (in analogy to autoDaily logs).

Output of phoenix tool

How to use

Type phoenix -h for on-line help, and phoenix -v for the version number.

IDP mode:
phoenix -d <date>	process all SCIENCE ABs for a specified date
phoenix -m <month>	process all SCIENCE ABs for a specified month (in a loop over all dates) (steps 1-3 not available for MUSE)
phoenix -d <date> -C	download & filter all ABs (step 1 only)
phoenix -d <date> -P	download & filter & process all ABs (steps 1 and 2)
phoenix -d <date> -M	call moveProducts (step 3 only, requires steps 1 and 2 executed before)
	Note: all three options C|P|M also work with -m <month>
phoenix -d <date> -M -n	same, but no cleanup of mcalibs done (useful in dry runs)
phoenix -d <date> -p	process all ABs as filtered with config.phoenix; no update of processing statistics (aiming at partial reprocessing of data after an issue has been discovered and fixed)
finishNight -d <date> -c	cleanup all remnants of date (only needed in manual use, if options -C or -P have been called before)
MCALIB mode:
phoenix -X -m <month>	create and process all CALIB ABs for a specified month (for a given pool), steps 1-3
phoenix -X -m <month> -C [-x]	create all CALIB ABs (step 1 only) (with -x you control that the HISTO information is kept in $DFO_AB_DIR/HISTO, otherwise it gets removed at the end of a month; useful if you process multiple months; for the HISTO information see here)
phoenix -X -m <month> -P [-x]	create & process all CALIB ABs (steps 1 and 2) (with -x option as before)
phoenix -X -m <month> -M	call moveProducts (step 3 only, requires steps 1 and 2 executed before)
finishNight -d <date> -c	same as above
phoenix -X -d <date> [-C|-P|-M]	exceptionally you can also call the tool in X mode by date, but then you might end up with incomplete ABs because, in mode -X, the VCAL directory always needs to be erased before a new call
DEEP mode:
phoenix -r <run_ID>	process all SCIENCE ABs for a specified run_ID (e.g. -r 094.D-0116A)
phoenix -r <run_ID> -C	download & filter all ABs for the specified run (step 1 only)
phoenix -r <run_ID> -P	download & filter & process all ABs (steps 1 and 2)
phoenix -r <run_ID> -M [-n]	call moveProducts (step 3 only, requires steps 1 and 2 executed before); flag -n: no cleanup of mcalibs done (useful if several ABs need to processed one after the other)
phoenix -r <run_ID>   -a <ab name> -C|-P	same as above, for one deep AB only (one target); options C|P also work with -a <ab> and can accumulate ABs (for very large runs); option -M will ask user then whether to work on the last AB only or on all accumulated ones

Use the various options on the command-line. In mass production, when you use phoenix for reprocessing of historical data, you may want to use it typically like that:

Pseudo dates

OCA rules and data pool

RELEASE definition

QC procedures

While the original concept of phoenix did not include certification and scoring, it turned out that it is extremely useful to create QC reports, mainly for the purpose of quick look. This is true for all kind of PHOENIX projects. QC reports could also be used to e.g. quickly check that the reduction strategy is the correct one, that the flux calibration looks reasonable etc. These QC reports can likely be easily derived from existing, historical QC procedures for science data.

It is also extremely useful to maintain a set of QC1 parameters and store them in a QC1 database table. This is needed for process control (for instance for proving statistically that the SNR is correctly computed) and also for the scoring. To run a PHOENIX project without QC1 parameters, reports and scores, means essentially flying blind!

The existing IDP streams all have scoring, refer to these examples for gathering more information. Generally the scoring for IDPs, and also for MCALIBs, should focus on key parameters relevant for science reduction. They should check for saturation, negative fluxes etc. It is also a good idea to measure and score association quality, i.e. proximity in time of an IDP to a key calibration.

Technical hints:

• If you use scoring, don't forget to set the TMODE configuration key to HISTORY in your config.scoreQC, to have the red and green buttons returning useful database queries.
• Note that it is extremely useful to make the QC procedures executable in parallel, then handled by condor in the same way as the AB pipeline jobs. Otherwise phoenix execution times will be dominated by the QC procedures (running one after the other in an extremely inefficient way). Actually it is unclear whether phoenix will run at all with QC procedures not enabled for parallel execution.

History scores, qualityChecker, autoCertifier (MCALIBs; for IDPs see below)

These are the possible combinations:

* in order to be auto-certified, otherwise one-by-one assessment needed

The auto-certifier is another embedded procedure of phoenix that uses the history score, plus the current score if available, to automatically decide about certification.

These are the concepts behind the auto-certifier:

For IDPs (both normal and DEEP ones), there is an optional step for certification (set CERTIF_REQUIRED=YES), just before calling 'moveProducts'.

Job execution patterns

Normally the DRS_TYPE to be chosen is CON (for condor) which can use all available cores for parallel processing, both for AB processing and for QC reports. The execution pattern is then identical to the one known from the autoDaily DFOS processing.

The following special cases exist (because standard condor processing would be too inefficient):

2. Multi-stream processing: this is special for MUSE. The MUSE pipeline uses an internal parallelization using 24 cores at certain phases of a recipe. With a custom JOB_PGI a multi-stream processing can be set up, with N=2 (muc10) or even 3 (muc11). N ABs can run in parallel, using all available cores in parallel and delivering corresponding efficiency gains.

3. Multi-instances processing (for MUSE only): a phoenix process can be implemented on several different accounts, for gaining efficiency. While both instances can execute independently, at the end one wants to bring all relevant information to a central account, which is then the MASTER, while the other account is called the SLAVE. See here for more.

AB pre-selection and creation

1. Normal situation before 2020-12-01:
Normally, the AB source is the operational log directory on qcweb, created by the DFOS tools (in particular by createAB and verifyAB for SCIENCE data). For reprocessing projects, the AB source is the previous IDP project. In those cases there is no need for AB creation within phoenix.

2. Normal situation after 2020-12-01:
After that date, no SCIENCE ABs have been created any longer by the DFOS tools, due to the reorganization of QC1. The normal (IDP) mode of phoenix creates the SCIENCE ABs, using calSelector and DFOS_OPS OCA rules. This only applies if the AB source is the operational log directory on qcweb. It does not apply to reprocessing projects. The date 2020-12-01 is the default date for this breakpoint. It can be changed or configured to any date with the config key DFOS_NOAB_DATE.

3. In certain special cases the ABs are not "good enough" and need additional modifications. For DEEP projects, the deep ABs need to be created from the existing tpl ABs. This needs to be done before the PHOENIX process can start.

Find more under the phoenix operational pages, e.g. here.

QCJOB_CONCENTRATION (IDPs only)

[This section is currently only relevant for IDPs on the giraffe_ph@muc08 account.] In normal cases the execution time for pipeline jobs (ABs) is longer than for the QC jobs, and the number of daily jobs is high enough to saturate the multi-core machine:

In that regime all phoenix jobs are executed day after day, and within a day batch there is first the AB call and then the QC call. Let's call this regime the "normal condor regime".

For giraffe_ph, the situation is different: typically there are only a few jobs (let's say 2) per day, meaning the muc machine cannot saturate, most cores are idle and the processing is not optimal. Furthermore, the QC jobs take much longer than the AB jobs (because a single QC jobs actually loops over up to 130 MOS products). For this regime (let's call it the "MOS condor regime") there is a special operational setup needed to optimize the processing. It is possible only for monthly processing (because then a sufficient number of AB/QC jobs exist to saturate the muc machine). In a first step all AB jobs are processed in the "normal" way (day by day, without saturation). This is suboptimal but because of the short execution time not critical for optimization. The daily QC jobs are collected but not executed. Only after all AB jobs of the month are done, the QC jobs are executed in one big call, thereby saturating the muc machine and optimizing the processing scheme. Let's call this regime the "QC job concentration regime".

In some test runs, the effective processing time for a month worth of GIRAFFE IDP processing with phoenix went down from more than 6 hours for "normal condor processing" to about 2 hours for the "QC job concentration processing".

The switch to this processing scheme is made via configuration (key QCJOB_CONCENTRATION). It can be used for any PHOENIX account, but currently makes sense for giraffe_ph only.

COMPLETE_MODE (IDPs only)

This mode is an extension of the classical IDP mode for those cases where a significant pipeline improvement merits the reprocessing of calibrations. If for certain reasons a MCAL reprocessing project is not feasible, or not desired, a more pragmatic approach is available with the COMPLETE_MODE. Here, in a first step all SCIENCE ABs for a specified DATE are retrieved. But instead of downloading the master calibrations listed in these ABs, the SCIENCE file names are extracted from the ABs, and then exposed to calSelector in raw2raw mode. This will deliver xml files with calib file names. The headers of those CALIB files are then downloaded, exposed to createAB. Finally the SCIENCE files are exposed to createAB.

Thereby, only the calibrations required for the new IDPs are re-created, and the SCIENCE ABs all point to these new masters. All products (CALIB and SCIENCE) are created with the same association scheme and latest pipeline version.

Technically, the DATE is applicable to the SCIENCE ABs only, while the CALIBs might come from other dates. They are all collected under the same 'processing date', appear on the same AB monitor. The final distribution of the new master calibrations respects the original dates.

After processing, the new masters need to be reviewed before they can be accepted (certified). As far as possible, the historical processing, scoring, and certification information is used, in the same schema as for the MCAL mode of phoenix.

If desired, the new masters can be renamed and ingested.

Since the COMPLETE_MODE is designed in particular for older dates, it supports historical versioning of the OCA and of the calSelector association schemes.

Normal mode with SCIENCE AB creation

In the classical IDP mode the ABs are downloaded from qcweb. This has several advantages: besides simplicity, this approach provides ABs from the daily workflow which have been checked for consistency with the OCA rules. After the date 2020-12-01, these ABs are no longer created and uploaded for operational reasons. Therefore, ABs as needed for IDPs have to be created within phoenix then. This applies only to IDP projects using the DFOS directory on qcweb (i.e. public_html/<instr>/logs).

The approach is rather similar to the one for the COMPLETE_MODE, but it is simpler since we don't want to recreate CALIB ABs. Instead, it is based on existing master calibrations for the desired DATE. If such masters do not exist, you cannot use phoenix in the IDP mode but you must create the masters first (either in an MCALIB project or in COMPLETE_MODE).

For the SCIENCE AB creation in normal mode, you need to prepare the configuration file config.phoenix:

• set the appropriate DATE for the breakpoint (DFOS_NOAB_DATE), from when on SCIENCE ABs must be created instead of being downloaded (default: 2020-12-01);
• specify CALSEL_OCA_RULE and make sure it is downloaded from the DFOS_OPS system into the phoenix system into the proper directory $DFO_CONFIG_DIR/CALSELECTOR;
• specify DFOS_OCA_RULE and make sure it is downloaded from the DFOS_OPS system into the phoenix system into the proper directory $DFO_CONFIG_DIR/OCA; this is the association.h file, only;
• optionally, you can set the config key CREATE_TXT_FILE to YES or NO which will create the calSelector txt files for you; these are not needed by the tool, but you may want to check them initially, and later turn them off since their creation costs some time.

It is possible to have several versions of CALSEL_OCA_RULEs and DFOS_OCA_RULEs in your system, separated by validity dates. Technically, don't forget to also have a copy of the CSConfiguration.properties file in $DFO_CONFIG_DIR/CALSELECTOR.

Also, make sure to edit config.createAB such that TOOL_MODE=AUTO, and N_MCAL_LIST=1 (because your MCAL pool is only one "pseudo-day" deep, namely the $DATE that you are using in the phoenix call).

The SCIENCE AB creation has been implemented for purely operational reasons. The process of creating SCIENCE ABs is closely connected to the QC1 process for calibrations. As such, it is migrated to the QC-v2 schema and not maintained in this form any longer. Therefore, SCIENCE ABs for the IDP production have to be created from the DFOS_NOAB_DATE on (2020-12-01).

This is achieved by the following steps:

• first, all SCIENCE files for a specified DATE are downloaded as headers;
• then they are exposed to calSelector in raw2Master mode; this will deliver xml files with mcalib file names;
• then the headers of those mcalib files are downloaded into $DFO_CAL_DIR/$DATE;
• then createAB for SCIENCE files is used to create the SCIENCE ABs.

Technically, the DATE is applicable to the SCIENCE ABs only, while the CALIBs might come from other dates.

After the SCIENCE ABs have been created, the workflow for phoenix is the same as if the ABs would have been downloaded. In particular, the next step is the application of the AB_PGI. In many cases, this important pgi might be simplified a lot once all SCIENCE ABs are created in this way and no fixes for historical content are needed anymore.

The creation of SCIENCE ABs for normal IDPs after DFOS_NOAB_DATE applies only if the ORIG_AB_DIR (the AB source) is configured to point to the DFOS ABs (public_html/$DFO_INSTRUMENT/logs). Only then we want to create ABs. If the AB source is already a previous IDP project, these ABs remain the valid source, and no creation is done within phoenix.

Ingestion

Each phoenix call results in an entry in the $DFO_JOB_DIR/JOBS_INGEST file, in the traditional DFOS way. The standard dfos tool ingestProducts is aware of running in the phoenix environment if

• INGEST_ENABLED is set to YES in config.phoenix
• PATH_TO_IT and CONVERTER are properly filled in config.ingestProducts.

If INGEST_ENABLED=YES, the tool is enabled to call the converter tool (optional) and the ingestionTool.

Find more about the IDP ingestion process here.

After ingestion, don't forget to call the tool cleanupProducts to replace the fits files with their extracted headers, in the standard DFOS way. For SLAVE accounts, the headers are transfered to the MASTER account, for central bookkeeping.

Installation

phoenix is required as the central tool for the PHOENIX process.

The PHOENIX process requires a customized file .dfosrc and a standard file .qcrc. The following keys are needed in .dfosrc in addition to the one used in DFOS:

The PHOENIX process requires the dfos tools listed above. That list might evolve. Please always use dfosExplorer to control your current PHOENIX installation.

The workspace as defined in the .dfosrc file is defined by $PROC_INSTRUMENT.

Configuration files

phoenix can run several configuration files:

• the standard one, config.phoenix, is used for IDPs (always needed);
• an optional second one, either config.phoenix_mcal, or config.phoenix_deep, for the MCALIB and DEEP modes, to replace the standard one. Both have the same structure:

The name of the MCALIB config file is not hard-coded, you might use another name but it needs to be registered in the main config file under MCAL_CONFIG. In MCALIB mode, the config file config.phoenix is only read to find the MCALIB config file, no other information is used. The MCALIB config file has the same structure as the IDP config file. (This has the effect that it is very easy to use the same account for either MCALIB or IDP production, by simply switching on or off the MCAL_CONFIG key in the IDP config file.)

In DEEP mode, the same is true for the config.phoenix_deep file.

In normal IDP mode, you can turn on the COMPLETE mode by using the COMPLETE_MODE=YES key.

In the normal IDP mode, there is the additional possibility to switch to a second PHOENIX installation with OTHER_RESOURCE. This is useful e.g. for the XSHOOT_TELL stream which is a stream additional to the XSHOOT_R IDP stream. Both are running from the xshooter_ph account. The configured OTHER_RESOURCE file (if enabled) defines the directories $DFO_MON_DIR, $DFO_SCI_DIR, $DFO_LOG_DIR, $DFO_PLT_DIR to be different from the ones used for the normal IDP stream.

Both configuration files go to the standard $DFO_CONFIG_DIR. The following configuration keys exist (a few keys are specific for a mode, they are marked below in the corresponding colours):

In addition to the phoenix configuration file(s), there are also the config files for the DFOS tools called. For the modes using calSelector (COMPLETE_MODE and normal IDP mode with DFOS_NOAB_DATE set), don't forget to provide the CSConfiguration.properties file in $DFO_CONFIG_DIR/CALSELECTOR (take it from your DFOS account). For the normal IDP mode with DFOS_NOAB_DATE set, don't forget to set N_MCAL_LIST=1.

Operational plug-ins (PGIs)

phoenix is a standard workflow tool. Usually instruments have a certain range of peculiarities which need to be handled via plug-ins (PGI's). This concept has proven useful for the daily workflow with autoDaily, and it is also used here. Some PGIs are configured directly in config.phoenix (or config.phoenix_mcal), others are PGIs for a specific dfos tool and are configured in their respective config files. All PGI's are expected under $DFO_BIN_DIR. Here is a list of PGIs directly related to phoenix:

Operational workflow description

See also here.

Self-management of $DFO_CAL_DIR and $DFO_RAW_DIR, and the other DFO directories with permanent data

Although the data disks on the phoenix machines are big (7-10 TB), it is reasonable to constrain the size of these data directories. Here is a description of the applied strategies (same strategy applies for IDPs, MCALIBs and DEEP unless otherwise noted):

$DFO_RAW_DIR:
The <date> directory is always deleted after processing.

$DFO_LOG_DIR and $DFO_PLT_DIR:
The usual content of these directories is preserved and copied to qc@qcweb. All logs and plots are accessible from the exported phoenixMonitor. On MASTER accounts, also logs and plots from the SLAVE account are collected.

Data structure on qcweb

The phoenixMonitor tool creates an overview of the phoenix processing history, similar to histoMonitor for DFOS operations.

Special notes

Don't do

• Never call 'moveProducts -m SCIENCE -d <date>' on the command-line in a PHOENIX environment. The tool will export all log and plot information to qc@qcweb:public_html/${DFO_INSTRUMENT}/logs/<date> and plots/<date> where it will a) get mixed up with dfos information (calibrations, historical science processing), and b) get used as AB source the next time you call the phoenix tool again on that date (if ever). There is no way to prevent this safely (setting DFO_INSTRUMENT to the value of RELEASE would also cause issues). Only if wrapped in 'phoenix -d <date> -M' (or 'phoenix -d <date>', the tool sets the target directory properly to qc@qcweb:public_html/${RELEASE}/logs/<date>.

Notification about phoenix on operational account

In the following we assume that each PHOENIX account has an operational counterpart on the mucs. (This will likely be modified, or removed, with DFOS-v2.) For the stream part of a PHOENIX process (after the bulk processing has been done), it is desirable to get a signal from the operational account that a certain set of master calibrations has been finished and is available for phoenix. This 'batch unit' for phoenix is one month, for no other than pragmatic reasons. For 'PHOENIX-enabled' instruments, the dfos tools dfoMonitor and histoMonitor on the operational account can be configured to become aware of the respective PHOENIX account. If the configuration keys PHOENIX_ENABLED and PHOENIX_ACCOUNT are set in $DFO_CONFIG_DIR/config.dfoMonitor, then the following happens: a) histoMonitor, when encountering a new month upon being called from 'finishNight', sends a signal (email) to the QC scientist that a new month has started, meaning that a set of certified master calibrations is available for the previous month. At the same time, a new status flag 'phoenix_Ready' is written into DFO_STATUS, along with the previous month (format YYYY-MM). b) This flag is catched by dfoMonitor and used to flag that month on the main output page: PHO ENIX: 2013-06   If no new PHOENIX job is on the ToDo list, this field is empty: PHO ENIX: none   c) It is left to the QC scientist, when and how this new PHOENIX task is launched. There is no automatic job launch, and there is no communication between the operational machine and the PHOENIX machines. The standard way is to launch 'phoenix -m 2013-06' as sciproc@muc08 (or xshooter_ph@muc08 etc.). Depending on the data themselves, and on the data volume, this might take a few hours. Since in principle it might be that in the future there are several concurrent phoenix jobs, or an operational account in addition to sciproc, there is no automatic mechanism to launch PHOENIX jobs. d) When this step has been done, the QC scientist can confirm this PHOENIX job to be done, by pushing the button 'done'. This triggers a dialogue, where the user is asked to confirm the execution, and then this month is removed from the DFO_STATUS file. If there is more than one month, all months will offered for confirmation, one after the other.