Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO

make printable

new:

see also:

v2.6.5:
- new optional key CHECK_SOF_EXIST (for PHOENIX)

createJob; AB monitor: getStatusAB;
batch queue system: CONDOR

While processAB calls the pipeline, processQC calls QC reports.

v2.7:
- enabled for OPSHUB environment

databases
dfos tools	esorex; ngasClient
output used by	pipeline products: to be QC-processed and certified
upload/download	download fits files with ngasClient

processAB

Description

	enabled for parallel execution
	enabled for condor execution

This tool is the standard interface between an AB and the pipeline/DRS. It extracts from the AB the information which is relevant for pipeline processing: the type of DRS (CON, CPL or INT), the recipe name, the recipe parameters, the set of input frames and associated calibration product names, the location and predicted names of products. This information is written into a set of frames file (sof). The DRS system is invoked to process the AB. Finally processAB checks for success/failure of the recipe, and (if configured) renames and moves the products from the workspace to the final certification area.

The tool checks first for availability of the required raw files and the calibration products, and downloads them if not found. Thereby a valid, historical AB stored in the DFOS system can be executed anytime later. Only exception is if the recipe, as coded in the OCA rules and listed in the AB, is "none" or "NONE": then data download is suppressed. If a download error occurs (e.g. because a file is not yet available in NGAS, or NGAS is temporarily unavailable), the tool reports this just like a processing error, i.e. the AB is flagged as 'FAILED'.

The tool supports the following types of DRS (see also the help page for createJob):

CPL (calling CPL-based pipelines, using esorex)
CON (CONDOR batch queue system, calling CPL recipes)
INT (like CPL but with the pipeline doing internal parallelization).

The DRS_TYPE is read from config.createAB.

How to avoid data downloads for unsupported modes. Per default, processAB always first downloads the raw and mcalib files listed in the SOF part if they are not already available in the system. Then it attempts pipeline processing. This is usually what you want, but in case of pipeline-unsupported modes, this does not make sense. If you use the reserved tags "none" or "NONE" as recipe name in the OCA association file, processAB interprets this as a signal to not download the files listed in the SOF part.

Other options. In addition to these standard features, the tool offers extended functionalities:

it gives the user full control over the product naming scheme
it offers the option to use plugins before and after the standard recipe call (primarily intended to deal with pipeline imperfections)
it returns an AB process status which is written into the AB
ACCEPT_FZ = YES --> in the sof, fits file names are replaced by fits.fz (relates to tile compression) which can be directly processed by the pipeline

esorex parameter configuration file. In section 2, you can enter the name of an esorex pipeline configuration file. This is currently useful for the generic detmon pipeline.

MAXN_RAW

To protect against processing "unusual" ABs having much more than the usual number of raw frames, thereby potentially slowing down the incremental processing system to an unacceptable degree, there is a parameter called MAXN_RAW which is by default 10. Any AB having more than MAXN_RAW input files is not processed and fires an email alert. Once you are out of the autoDaily regime, you can process that AB by calling the tool on the command line with option -F (force), see below. Or you can fine-tune the default, global MAXN_RAW for a specific RAW_TYPE to a more appropriate number, by using the section 3.3 in the configuration file.

Output

pipeline products and logs; the products are delivered in the product path specified in the AB, and with names generated according to the PIPEFILE convention with indexes as defined in the configuration file.

Example

processAB -a FORS1.2024-07-12T06:59:13.223_tpl.ab

How to use

Type processAB -h for on-line help, processAB -v for the version number,

processAB -x

for help about esorex and your pipeline (see below);

processAB -a FORS2.2024-07-12T06:59:13.223_tpl.ab

to process a particular AB;

processAB -a FORS2.2024-07-12T06:59:13.223_tpl.ab -D

to do the same in DEBUG mode (prepare the environment, but leave the execution to a manual call);

processAB -a HAWKI.2013-01-06T23:55:39.189_tpl.ab -F

force the execution of an AB which was refused by the automatic system (because of NRAW > NRAW_MAX).

Note that the tool logs per AB into <AB>.rblog which is created under $DFS_LOG. (The name rblog is used for historical reasons only, this was once the "reduction block log".)

For execution by condor (the standard way), the option -u <user> has to be added, like e.g. -u vircam, in order to make condor find the home directory and source the environment properly. This is done automatically by createJob.

For execution by DRS_TYPE=INT, the option -l (no argument) has to be added (l stands for LIKWID), to mark the AB as being executed by autoDaily. This is done automatically by createJob.

For execution by CONDOR in the OPSHUB environment, the option -i <instr> has tobe added. This is done automatically by createJob.

processAB and esorex

The tool calls the DFS tool esorex as the standard interface to the CPL-based pipelines:

esorex --time --output-dir=$WORK_AREA --suppress-prefix $ESOREX_PARAMS $RECIPE $PARAMS sof

No other esorex parameters are specified explicitly. If wanted, the user can configure those in $HOME/.esorex/esorex.rc (e.g. modify the log level). Also, the processAB config key NO_CHECKSUM has direct impact on the esorex parameters (see below).

For generic pipelines (e.g. the detmon pipeline), you can configure a set of parameters in a configuration file $HOME/.esorex/detmon_{ir|opt}_lg_{instr}.rc. You need then to define these in section 2 of the configuration file. These parameters are extracted by the tool and called as $ESOREX_PARAMS. If no such configuration file is defined in section 2, the variable $ESOREX_PARAMS remains empty.

For DRS_TYPE=INT, the esorex call is embedded in a call of the tool likwid-pin, to enable internal parallelization:

$LIKWID_PATH/likwid-pin -c $LIKWID_CALL esorex --time --output-dir=$WORK_AREA --suppress-prefix $ESOREX_PARAMS $RECIPE $PARAMS sof

The parameters $LIKWID_PATH and $LIKWID_CALL need to be configured in config.processAB.

For DRS_TYPE=INT, you may also need to define OMP_NUM_THREADS=24 in your .qcrc (corresponding to the value 0-23 for LIKWID_CALL).

Because of the close logical connection between processAB, esorex and the pipeline, the tool provides esorex help and pipeline recipe help, invoked as processAB -x.

DEBUG: The DEBUG mode can be invoked as -D. It sets up the $WORK_AREA, extracts the sof etc., so that you can start the esorex command from the command line. This may be useful if you want to check esorex parameters, products before renaming etc. Products will not be renamed or shifted. Don't forget to delete the $WORK_AREA after your tests.

Installation

Use dfosExplorer or dfosInstall.

Since the tool is designed for parallel processing of ABs, it uses $WORK_AREA/$AB as recipe work area. These subdirs are deleted after execution.

Configuration file

config.processAB defines:

Section 1: general
WORK_AREA	/data28/visir/CONDOR_WORK	root directory where pipeline recipes are executed; make sure it exists!
RENAME_SHIFT	YES \| NO	enable/disable renaming and shifting of pipeline products; NO makes sense for debugging only.
ACCEPT_FZ	YES \| NO	YES: fits file names are replaced by fits.fz in the sof; if found, fits.fz files are not downloaded by ngasClient; default: NO
# for DRS_TYPE=INT only:
LIKWID_PATH	/opt/likwid/bin	path to likwid tool (since this is not a standard dfs tool)
LIKWID_CALL	0-23	translated into '-c 0-23': split by 24 detectors
# for PHOENIX only:
DRS_FROM_AB	YES\|NO	for PHOENIX only: read DRS_TYPE from the AB (instead of config.createAB); this to support special situation for MUSE IDPs that some ABs need a different DRS_TYPE than others; default: NO
NO_CHECKSUM	YES\|NO	for PHOENIX only: adds --no-checksum and --no-datamd5 in esorex call; for better performance in well-defined exceptional cases (like MUSE IDPs); default: NO
CHECK_SOF_EXIST	YES\|NO	for PHOENIX only: esorex checks all input files for existence before starting the recipe; default: NO
Section 2: esorex configuration file name(s) for generic pipeline(s) List here the name(s) of esorex configuration file(s) of general pipeline(s) (e.g. detmon)
RAW_TYPE	DETMON_FLAT	usually just one, but could be more
DET_ID	ANY	required only if you need to specify config files per detector (avoid if possible), one line per detector; if not needed, just enter ANY
	det1 etc.
CONFIG	detmon_ir_lg_vircam.rc	name of config file for generic pipeline, under $HOME/.esorex
Section 3: plugins 3.1 GENERAL plugins There are three such plugins. They are optional and of type 'post-processing'.
COMMON_PLUGIN	my_plugin	optional post-processing plugin to be applied to all raw_types
ERROR_PLUGIN	my_error_plugin	optional post-processing plugin to be applied if a recipe fails
FINAL_PLUGIN	&&my_final_plugin $AB&&	optional plugin to be called at the end of RENAME_SHIFT, accepts $AB as argument, to be enclosed as &&...&&
3.2: PRE/POST plugin names of optional plugins specific to a particular RAW_TYPE. They have to reside under $DFO_BIN_DIR; PRE and POST means before and after recipe execution. Syntax: - RAW_TYPE: e.g. BIAS, as defined in the OCA files - PRE_PLUGIN: name of plugin (can be NONE) - POST_PLUGIN: name of plugin (can be NONE)
SPEC_PLUGIN	DARK pgi_preDARK pgi_postDARK	only one plugin per RAW_TYPE!
3.3: MAXN_RAW section optional configuration for maximum number of accepted raw frames in the AB, this to override the tool default (10). Syntax: - RAW_TYPE: e.g. BIAS, as defined in the OCA files - MAXN_RAW: e.g. 15
Section 4: Define numbering scheme for products That section defines the final product names. The pipelines deliver internal names. The DRS may or may not rename them into the PIPEFILE scheme. Hence it is better to have the final names created by the user. While the root name is defined in the AB, the index and the extension need to be defined here to make the final name complete. Index and extension are defined per PRO.CATG. - RAW_TYPE is needed since some recipes may work on multiple RAW_TYPEs - Recipe: pipeline recipe name - internal: internal name of product - index_ext: user-specified product index and extension - PRO.CATG: pro.catg key - RAW_TYPE: as read from the AB
PRO_CATG	gimasterbias \| master_bias.fits \| 0000.fits \| MASTER_BIAS \| BIAS
etc.

Status information

The tool writes status information into the AB (PROCESS_STATUS = SUCCESSFUL or FAILED), based on the availability of products and on the recipe return code. It also writes the total execution time in seconds into the AB (under TEXEC), to be evaluated later by the statistics tool extractStat. It writes the corresponding updates into the .tab file.

Operational aspects

Typical call for processAB: within the execAB job file. You can launch it on the command line.
Note that the key DPR_CATG in the created AB actually refers to the tool mode; in CALIB mode, it includes data with DPR.CATG=CALIB, TECHNICAL, and ACQUISITION (if configured in the OCA rules).
The config key DRS_FROM_AB is used within PHOENIX to switch the standard DRS_TYPE to another value for certain ABs. This is usually not relevant, and the key can be ignored in most cases (default is NO).
The config key NO_CHECKSUM prevents the checksum calculation for the products. This is usually not a good idea (default: NO) since the products can then not be ingested into NGAS. For certain PHOENIX projects, e.g. for MUSE IDPs, the checksum calculation within a recipe may take up to 10 minutes and is useless at that point, since the product is not yet final. In such cases NO_CHECKSUM prevents calculating the checksum and datamd5.
The config key CHECK_SOF_EXIST is used within PHOENIX to check every file in the SOF for existence before starting processing. This is useful for some pipelines which check for existence only after a long executing time and then would fail if a file does not exist. If a file is missing and the key is set, the tool does not call the recipe and marks the AB as failed.

Workflow description

1. Create a unique work directory as $WORK_AREA/$AB

2. Preparation

2.1 parse the AB for pipeline processing information (recipe, sof, parameters etc.); ACCEPT_FZ=YES: replace in the sof fits by fits.fz for those files which exist already as fits.fz files
2.2 check for MAXN_RAW (either as configured, or default); if actual number is higher, do not execute the AB but send an information mail to $OP_ADDRESS
2.3 read DRS_TYPE (from config.createAB)
2.4 call PRE_PLUGIN
2.5 Check for existence of MCALIBs, download with ngasClient if necessary (downloaded mcalibs are listed in $DFO_MON_DIR/MCAL_DOWNLOAD, for later deletion with cleanupProducts); if download impossible, send out warning, set AB status to "FAILED", finish
2.6 Check for existence of target directory ($DFS_PRODUCT/$RAW_TYPE/$DATE); create if not existing

3. Launch recipe

4. Verify results

4.1 no products in $WORK_AREA: send out warning, set AB status to "FAILED", apply ERROR_PLUGIN, finish
4.2 products found: set AB status to "SUCCESSFUL"
4.3 insert the correct value for PIPEFILE
4.4 apply configured common plugin
4.5 apply POST_PLUGIN
4.6 move all products to final certification area, rename them to pipefile name (if RENAME_SHIFT is enabled)
4.7 if not: leave them in $WORK_AREA for inspection (note: this means keep the WORK_AREA subdirectories, which will accumulate quickly!)
4.8 update TEXEC, T_QCEXEC and AB_status in the AB
4.9 apply FINAL_PLUGIN.

Last update: April 26, 2021 by rhanusch

Common DFOS tools: Documentation