make printable	new:	see also:
	v2.3: - supports OPSHUB environment (no functional changes)	documentation about CONDOR AB concepts, AB structure
	v2.3.5: - cascadeMonitor removed	databases none dfos tools none output execAB and execQC files; entries in job file upload/download none

createJob

Description

This tool reads the list of created ABs and transforms it into an executable list of process jobs. It creates an execution file for recipe calls ($DFO_JOB_DIR/execAB_<MODE>_<DATE>). It also creates an execution file for QC report jobs ($DFO_JOB_DIR/execQC_<MODE>_<DATE>).

The tool presently supports one type of DRS (data reduction system) for pipelines:

• CPL (calling CPL-based pipelines through the esorex interface)

and two types of batch queue systems (BQS):

• CON = CONDOR (the standard BQS for QC)
• INT (internal parallelization, used for MUSE).

For historical reasons, all three are valid options for the configuration key DRS_TYPE. Effectively, CPL means serial processing, CON means batch queue processing (many independent jobs in parallel), and INT means serial processing (one AB at a time) but internal splitting into N jobs (N spectrographs in the case of MUSE). In operations, INT is the standard mode of the MUSE pipeline, while CON is the standard mode for all other cases.

For DRS_TYPE=CON the scheduler tool condor is called with the dfs script vultur_exec_cascade. For INT, the linux tool likwid-bin is used.

For the QC report calls, three cases of BQS are configurable in config.processQC (!):

• SER = serial processing (the standard way: one by one)
• PAR = parallel processing, using CONDOR (and requiring a corresponding architecture of the QC procedures)
• IMPLICIT = QC reports are created implicitly right after pipeline processing, by processAB using a plugin.

Subcascades. The tool supports creation of subcascades from a pre-existing cascade. This might be useful if a certain AB has failed, or was not certified, and all depending ABs need to be reprocessed again.

Batch queue system. A batch queue system is able to recognize and manage a set of ABs and their dependencies within a cascade, and to feed the cascade into a processing platform. Presently QC is using a farm of 9 multi-core blades called 'muc<nn>', and the open-source software HTCondor (commonly called CONDOR) is used for scheduling jobs for parallel execution.

CONDOR addresses all available CPUs (up to 8 for muc01-muc07, up to 30 for muc08 and muc09), hence increasing the efficiency of pipeline processing.

The QC jobs can also be executed in parallel, here even without the need to respect dependencies. Since QC reports are coded by the QC scientist, they do not strictly follow a common architecture. Depending on the way they are written, they might or might not execute in parallel.

The standard combination for DFOS (autoDaily) processing is BQS=CON. Sequential processing without CONDOR is also possible on the command line but should be used operationally only in exceptional cases (like for MUSE). Check here for more information about CONDOR.

The tool writes the name of the execution file (execAB) into the off-line job file (JOBS_NIGHT for off-line processing, JOBS_AUTO for automatic processing with autoDaily). JOBS_NIGHT can be launched anytime off-line.

Find its place in the daily workflow here.

Execution mode for QC report jobs

The tool writes an execution file execQC containing processQC calls. These calls can be managed in three ways:

	sequential scheduling	scheduling through a batch queue system	implicit scheduling (new with v1.6)
configuration in config.processQC (QCBQS_TYPE):	SER	PAR	IMPLICIT
executing:	one AB at a time	multiple ABs at once	within processAB, by plugin
sequence controlled by:	execQC file which is filled by createJob (but sequence does not matter)	same (no dependencies for QC jobs)	execAB file
levels to QC report creation	processQC>> MIDAS/python/... (whatever is the platform for the QC procedure)	BQS >> processQC>> MIDAS/python/...	BQS >> processAB >> >> post_plugin >> MIDAS/python/...
invoking AB:	processQC -a <AB>	processQC -a <AB>	none

Note that using QCBQS_TYPE = PAR requires that your QC procedures can handle parallel execution correctly!

Preparation for updating HC plots. In incremental processing mode, the tool writes a call of the executable file execHC_TREND into the jobs file. This file is filled by autoDaily from two sources: qc1Parser (for the most recent opslog entries), and scoreQC (for the most recent QC1 parameters). After execution of execAB and execQC, it finally collects all trendPlotter calls triggered by any of these data sources. Thereby it ensures that all HC reports are updated in due time to feed back product quality information to Paranal SciOps.

Calls of getStatusAB. To have an updated view of the AB monitor, the tool getStatusAB is called during execution. Execution time and load of the tool is non-zero (but it has become much faster with v3.0), so it is not possible to run it permanently. But as a reasonable compromise, a tool refresh is called every 60 sec, and the web page is auto-refreshed every 300 sec.

Output

The tool produces the following list of calls in $DFO_JOB_DIR/JOBS_AUTO (if called by autoDaily) or $DFO_JOB_DIR/JOBS_NIGHT (if called on the command line) (<mode> and <date> are the command-line parameters):

Bold: files created by createJob (and deleted by finishNight):

• rawDown_CALIB_<DATE> in $DFO_JOB_DIR
• mcalDown_CALIB_<DATE> in $DFO_JOB_DIR
• execAB_<MODE>_<DATE> in $DFO_JOB_DIR
• execQC_<MODE>_<DATE> in $DFO_JOB_DIR

There is always only one call execAB_<MODE>_<DATE> per mode and date in the job file (same for execQC).

The rawDown_CALIB_<DATE> and mcalDown_CALIB_<DATE> files are created only for mode=CALIB.

rawDown. The rawDown_CALIB_<DATE> files have the following structure:

They are intended as protection against the muc blades firing too many download jobs at once. Otherwise it could happen that the condor dagman releases many (currently 8) download requests at once, potentially overloading ngasClient and also the local decompression, and HOTFLY, processes. The current download file allows up to 8 parallel downloads (this number has been empirically tested) but not more. For the OPSHUB, we download one file after the other (MAX_CORES=1) because we are bandwidth-limited, and having more than one transfer process at a time does not offer advantages.

The mcalDown file has a related but different purpose. It provides a controlled download of master calibration files which had already been transformed into headers (using cleanupProducts). With condor on the muc blades, it might happen that many master_flat ABs are fired at the same time and, under unfortunate circumstances, all attempt to download at the same time the same master_bias. This is avoided by checking all ABs for their MCALIB section, and then download the required mcalibs for all ABs first, before the condor processing is then started. The mcalDown files are only created if there is something to download, under current standard conditions (incremental processing) only rarely.

For the OPSHUB, there is a special feature for the download jobs, the progress monitor. It displays in a simple graphical way the total amount fo files to be downloaded, and the current status of downloads (one dot per downloaded file). This helps following the download status in a situation which is bandwidth-limited.

Monitor calls. In automatic calls (within autoDaily), the tool enters into JOBS_AUTO:

In command-line mode, the tool enters into JOBS_NIGHT:

How to use

Type createJob -h for on-line help, createJob -v for the version number,

createJob -m CALIB -d 2024-07-12

to create a pipeline processing job for all CALIB ABs from 2024-07-12,

createJob -m CALIB -d 2024-07-12 -a

to create a subcascade.

There is the option -F used if the tool is called by autoDaily (not on the command-line!) and only for mode CALIB:

createJob -m CALIB -d 2024-07-12 -F

This triggers the getStatusAB tool to be run with option -F (immediate transfer to qcweb).

Check the recipe call entries in $DFO_JOB_DIR/execAB_CALIB_2024-07-12, and the job submission entry in $DFO_JOB_DIR/JOBS_NIGHT.

Subcascade. A subcascade is created interactively: the flag -a causes the tool to ask the user about the AB name, and then creates a new job file from the existing execAB file. The new job file is called execAB_<mode>_<date>_sub and contains only those ABs and the corresponding waitfors which depend upon the specified parent AB. That job is ready to be launched in the same way as the main job file.

The typical process calls in execAB_CALIB_2024-07-12 look like follows (for DRS_TYPE = CPL):

processAB -a <ab_name1>
processAB -a <ab_name2>
etc.

The typical QC report calls in execQC_CALIB_2024-07-12 look like follows (for QCBQS_TYPE = SER):

processQC -a <ab_name1>
processQC -a <ab_name2>
etc.

The typical entry in the job file, for the above call and DRS-TYPE = CPL, looks like:

# pipeline jobs for CALIB, 2024-07-12
getStatusAB -d 2024-07-12 -r &
$DFO_JOB_DIR/execAB_CALIB_2024-07-12

$DFO_JOB_DIR/execQC_CALIB_2024-07-12
getStatusAB -d 2024-07-12 -F &

The DRS type is read from config.createAB, the QCBQS_TYPE from config.processQC. The getStatusAB calls provide an up-to-date view of the processing situation every 60 sec as long as the batch is executing. A final call of getStatusAB occurs with flag -F, meaning the results (the AB monitor and all log and plots files) are written to the QC web server qcweb.

Re-creation of ABs. For support of the re-create mode of createAB, the tool createJob has an option -r <job_id> where <job_id> is the name of the job file to be created. This option is used only within createAB, not in normal operations. The added functionality is rather small, the tool works on a job_id (with suffix recreate) that is not determined by the tool itself but by createAB.

Installation

Use dfosExplorer, or type dfosInstall -t createJob .

Configuration file

config.createJob defines:

Status information

Operational aspects

• Typical call for createJob: right after createAB, with the same set of parameters. This is done automatically by autoDaily for the incremental CALIB processing.
• The jobs resulting from several interactive calls (for different dates) are all written sequentially into JOBS_NIGHT, ready for off-line execution. The job file is cleaned later by moveProducts.
• The tool 'getStatusAB' is called in recursive mode during processing, to update the AB status on the AB monitor. A final call is done to export all content to the QC web server qcweb. Check out here for more details.
• The job file for the subcascade is useful if a certain AB, and with it all other ABs depending upon it, needs to be re-executed.
• Creation of subcascades makes sense only if the main job file with the main cascade exists already.
• The optional keys MAX_JOBS, FULL_RAWDOWNLOAD and MAX_CORES are evaluated for the OPSHUB only, and are controlled by the distillery workflow tool. MAX_JOBS controls the maximum allowed number of jobs for parallel execution (to constrain e.g. memory consumption). FULL_RAWDOWNLOAD = YES accounts for large files with long download times, to avoid multiple downloads in the processAB environment. MAX_CORES is forced to 1 in the OPSHUB environment. All three parameters are controlled in config.distillery and propagated to config.createJob.

Workflow description

1. Find the AB list as $DFO_MON_DIR/AB_list_<DATE>

2. read DRS_TYPE from config.createAB and QCBQS_TYPE from config.processQC

3. loop on ABs:

3.1.1 mode=CALIB: check the ABs for files in the RAWFILE section which need to be downloaded before processing; if found, write the download calls into rawDown_CALIB_<DATE>, write call of rawDown into JOBS_NIGHT or JOBS_AUTO
3.1.2 mode=CALIB: check the ABs for files in the MCALIB section which need to be downloaded before processing; if found, write the download calls into mcalDown_CALIB_<DATE>, write call of mcalDown into JOBS_NIGHT or JOBS_AUTO

3.2 DRS_TYPE=CON: determine job IDs and waitfors (dependencies); DRS_TYPE=CPL/INT: determine dependencies and write processAB calls in proper sequence

3.3 create command-line recipe calls:

3.3.1 CPL: processAB -a $AB
3.3.2 CON: <job_id> || processAB || -a $AB -u $USER || <dependencies>
3.3.3 INT: processAB -a $AB -l

3.4 write result into execAB_<mode>_<DATE>

3.4.1 call PLUGIN if configured
3.4.2 write call of getStatusAB into JOBS_NIGHT or JOBS_AUTO (in recursive mode, -r)

3.5 write call of execAB into JOBS_NIGHT or JOBS_AUTO

3.6 create command-line QC report calls:

3.7.1 SER: processQC -a $AB
3.7.2 PAR: <job_id> || processQC || -a $AB -u $USER || NONE
3.7.3 IMPLICIT: no call

3.7 write result into execQC_<mode>_<DATE>
3.8 write call of execQC file into JOBS_NIGHT or JOBS_AUTO
3.9 write call of getStatusAB into JOBS_NIGHT or JOBS_AUTO (with option -F if called by autoDaily, incremental mode )
3.10 (only if called within autoDaily, incremental mode: write call of execHC_TREND into JOBS_AUTO)

3.11 if CAL_CLEANUP=YES and mode=CALIB: write command into JOBS_NIGHT or JOBS_AUTO to clean up all raw files after QC processing

4. set DFO status to 'cal_Queued' or 'sci_Queued'

5. If flag -a is set (subcascade):

5.1 query for the AB name
5.2 extract the subcascade
5.3 exit

Operational hints

• In automatic, incremental mode, all steps are executed in their own workspace; the jobs file is called JOBS_AUTO in the end
• In incremental mode, the tool is called by autoDaily. The execution of JOBS_AUTO is done automatically.
• In interactive mode, launching JOBS_NIGHT is responsibility of the QC scientist. In general, JOBS_NIGHT may have several different calls of execAB and execQC.
• The configuration key CAL_CLEANUP should be used if disk space is an issue.
• The optional PLUGIN can use $DATE and $MODE as exported environment variables .
• The tool uploads the config.processAB file to the $DFO_WEB_SERVER, in order to make the PRO_CATG section available to the OPSHUB tool.