make printable	new:	see also:
	v1.2.6: handles files with same product root and having the same PRO_CATG	- embedded in QC procedures - before scoreQC databases qc1..<configured table> (read and write) dfos tools processQC, scoreQC, qc1Ingest output used by none upload/download none

writeQC

Overview

QC1 parameters are calculated by pipeline recipes and are written as fits keys into product headers. There are frequently also parameters that are not available from headers, e.g. civil date, and parameters that are not calculated by pipeline recipes. These parameters exist as local variables within QC procedures using MIDAS, Python, etc. QC1 parameters are ingested into the QC1 database using qc1Ingest which expects all parameter values as command line arguments. Therefore, QC procedures have to collect these data before calling qc1Ingest.

The tool writeQC fills the gap between header keywords, local script variables, and qc1Ingest. It

• collects from fits headers and local script variables all data necessary for qc1Ingest calls,
• ingests parameters into the QC1 database,
• optionally calculates average and rms values for QC parameters across detectors in case of multi-detector instruments.

The tool is called per AB. It expects pipeline products within the certification area (i.e. in the $DFS_PRODUCT directories) as specified in the AB. The connection between PRO.CATG and database tables and between fits key names and database columns is defined in the configuration file config.writeQC.

Usually all pipeline products with file names starting with the same product root (e.g. r.KMOS.2014-01-01T00:00:00.000_tpl) have different PRO.CATGs. New with v1.2.6: In exceptional cases, there can be product files with the same product root and with the same PRO.CATG. Then, writeQC reads QC1 parameters only from the first file in alphabetical order. E.g. if there are files r.KMOS.2014-01-01T00:00:00.000_tpl_0000.fits, r.KMOS.2014-01-01T00:00:00.000_tpl_0001.fits, and r.KMOS.2014-01-01T00:00:00.000_tpl_0002.fits and all have PRO.CATG=MASTER_DARK then the QC1 parameters for MASTER_DARK would be read from r.KMOS.2014-01-01T00:00:00.000_tpl_0000.fits.

The tool is usually called from the QC script invoked by processQC, and before scoreQC is executed. In the QC script, three lines of code are needed for proper operation. The first line calls writeQC in mode 'PREPARE'. This creates a small script that is executed in the second line. The script depends on the scripting language used for the QC script (either MIDAS, Python, or Unix Shell) and writes the values of header keywords and local variables into a temporary file. This approach is necessary for sharing the values of local variables between the QC script and writeQC while supporting different scripting languages with only one version of writeQC. The third line calls writyQC in mode 'INGEST' which finally uses qc1Ingest for writing the QC1 parameters into the database.

The three lines of code can be written in a way so that they are identical for each QC procedure. All information for reading and writing QC parameters is configured within the configuration file. An optional fourth line can be added for multi-detector instruments calling writeQC in mode 'AVERAGE'.

The following graph illustrates the place of writeQC within the QC processing scheme:

How to use

Type writeQC -h for help and writeQC -v for the version number.

The tool is invoked as a UNIX command line. The syntax is:

writeQC -m PREPARE | INGEST | AVERAGE -a <AB> [ -e <EXTENSION> ]
        [ -l MIDAS | PYTHON | SHELL ] [ -c <CONFIG_FILE> ]

Calling sequence:

writeQC -m PREPARE -a <AB> [ -e <EXTENSION> ] [ -l MIDAS | PYTHON | SHELL ]
<TMP_DIR>/<AB>_<EXTENSION>/writeQC.<SCRIPT_TYPE>
writeQC -m INGEST -a <AB> [ -e <EXTENSION> ]

where <SCRIPT_TYPE> is either 'prg' for MIDAS, 'py' for Python, or 'sh' for shell scripts. The language used for QC scripts can be specified with option '-l' in mode 'PREPARE' or within the configuration file (see below). The command line option is useful if the QC scripts for an instrument are written in more than one language. If option '-l' is given then the value for 'LANG' in the configuration is not evaluated. Only mode 'PREPARE' needs to know the scripting language.

The <EXTENSION> must be specified in case of MEFs. For SEFs, the '-e' option can be omitted (which is equivalent to '-e 0').

For multi-detector instruments, writeQC also supports average and rms calculation of QC parameters. The above sequence can optionally be followed by:

writeQC -m AVERAGE -a <AB> .

This requires that the QC1 parameters for all detectors have been ingested before. They are read again from the database and identified via MJD-OBS as given in the AB (with a precision of 1E-5). This supports true MEF cases like CRIRES or VIRCAM and multi-detector instruments with SEFs (like VIMOS). For each QC parameter, the average of the values for all detectors and the rms is calculated. Two additional entries are inserted into the database where the detector name is replaced by 'AVG' and 'RMS', respectively. The names of the parameters for averaging and the keys denoting the detector are configured in config.writeQC.

Shell. Example for UNIX shell usage (4 extensions MEFs, additional averaging plus scoreQC):

for EXT in 1 2 3 4
do
    writeQC -m PREPARE -a $AB -e $EXT
    ${TMP_DIR}/${AB}_${EXT}/writeQC.sh
    writeQC -m INGEST -a $AB -e $EXT
done
writeQC -m AVERAGE -a $AB
scoreQC -a $AB

Note: all local shell variables that contain QC1 parameter values must have been exported (e.g. 'export CIVIL_DATE=2007-07-01' in Bash).

MIDAS. Example for MIDAS (single extension files, no averaging):

$writeQC -m PREPARE -a {AB_NAME}
@@ {TMP_DIR}/{AB_NAME}_0/writeQC.prg
$writeQC -m INGEST -a {AB_NAME}
$scoreQC -a {AB_NAME}

Python. Example Python calls for MEF products (16 extensions):

for ext in range(17, 1):
    os.system('writeQC -m PREPARE -a ' + ab_name + ' -e ' + str(ext))
    execfile(tmp_dir + '/' + ab_name + '_' + str(ext) + '/writeQC.py')
    os.system('writeQC -m INGEST -a ' + ab_name + ' -e ' + str(ext))
os.system('writeQC -m AVERAGE -a ' + ab_name)
os.system('scoreQC -a ' + ab_name)

Operational hints. Automatic ingestion into the QC1 database can be switched off (see configuration). This is useful for testing the tool and it is highly recommended to use this feature for configuring writeQC. All temporary files can be found in the directory <TMP_DIR>/<AB_NAME>_<EXT>; the qc1Ingest calls are within the files <TABLE_NAME>.sh.

It is possible to run writeQC in parallel for different ABs. It is also possible to work on separate extensions of MEFs in parallel.

Configuration file

The tool configuration file (config.writeQC) defines the relation between PRO.CATG and database tables and between fits key names and database columns. The option -c <CONFIG_FILE> allows to use a different configuration file (which must also be in DFO_CONFIG_DIR). This is useful if QC1 scripts are written in different scripting languages; the syntax for value formatting (see table) can then be adapted.

Keyword names. The keyword names given in the configuration file must not contain blanks. A dot '.' is interpreted as a placeholder (i.e. the UNIX grep command is used internally). This feature is useful if keyword names vary as is the case e.g. for some VIMOS names which depend on the quadrant.

Formatting parameter values. It is possible to format the values of (numerical) header keywords and script variables before writing to the QC1 database. This is done with

• SET/FORMAT in case of MIDAS,
• the printf command of awk in case of SHELL,
• or Python string formatting using the % operator.

The syntax of the format strings in Sects. 2 to 4 depends on the scripting language specified in Sect. 1.1 and must be compatible with these commands or operators. In case of MIDAS, a 'C' has been added to the list of allowed formats which is denoting an unformatted character string.

If formatting is not wanted for a parameter coming from a fits header then '%s' or 'C' can be chosen. This also functions for numerical header keys which are then interpreted as character strings and written into the database with exactly the same precision as in the header. This can often be a good choice.

It is recommended to use a numerical format string for floating point variables in QC procedures.