Include detmon as a generic pipeline

home

dfos

administration

documentation

procedures

concepts

external: QC

2010-12-01: online

2010-12-03: migration put on hold, waiting for new processAB, comments by BW and SM included

2011-02-22: migration continued after upgrade of processAB to v1.10

2011-11-07: step on config.processAB added

For reasons of better and more efficient maintenance of the pipeline recipes, the SDD pipeline group introduces generic pipeline recipes. Generic pipelines collect data processing tasks like detector monitoring which are not specific to an instrument. SDD offers now both lingain recipes, the optical CCD and the NIR array detector version in one generic pipeline. In the future, SDD plans to remove the detmon recipes from the instrument pipelines and will only maintain the generic pipeline version.

This page describes the procedure how to migrate the operational detmon pipeline recipe as part of the instrument pipeline (IP) to the generic detmon pipeline recipe (GP). This page does not describe, how to put the detmon recipe into operations for the first time. The detmon recipe is currently implemented in several instrument pipelines. Either the version for optical CCDs or the version for NIR arrays is included. Both recipes are also available in the new generic detmon pipeline. In the long term it is planned to use the recipe from the generic pipeline instead of the recipe implemented in the instrument specific pipeline. The migration procedure is explained in the detmon pipeline user manual vers 1.0.1, 2010-11-02 (see bottom of that page: detmon pipeline manual ) and is repeated on this page. Additional information is given on the esorex web page. The migration has to be done only once. The procedure does not has to be repeated for generic pipeline upgrades.

IP stands for instrument pipeline (from where the migration starts)
GP stands for generic pipeline (to where the migration follows)

step by step

In case you operate the current IP detmon recipe with a recipe config file in .esorex, (e.g. /home/crires/.esorex/crires_detmon.rc) backup this file. Otherwise this step can be omitted.

To adapt the GP detmon recipe, to the instrument specific needs, the default GP recipe command line parameters have to be overwritten by the instrument/detector specified parameters. Copy the recipe config file into the directory ${HOME}/.esorex of your operational processing machine.

The recipe config values in detmon_...rc files are first guess values provided by SDD and might differ from the operational values.

It is recommented to include .esorex directory in your backup scheme.

Only in case your ${DFO_CONFIG_DIR}/OCA/${DFO_INSTRUMENT}_association.h file contains recipe parameters, check that they are part of the ${HOME}/.esorex/detmon_{opt|ir}_lg_{instrument}.rc.

Type ls /home/quality/pipelines to verify that detmon-1.0.7 pipeline is installed and that there is a soft link from detmon to detmon-1.0.7

Add the detmon pipeline to the instrument specific pipeline:
in .esorex/esorex.rc add :/home/quality/pipelines/detmon to esorex.caller.recipe-dir={instrument pipeline path}.
E.g. for VIRCAM the line looks like:
esorex.caller.recipe-dir=/home/quality/pipelines/vcam-1.0.7:/home/quality/pipelines/detmon

Verify the last step by typing esorex --recipes
In addition to the recipes of the instrument pipepline, there should occur the following two lines,

           detmon_ir_lg          : Linearity/Gain recipe for the IR domain
           detmon_opt_lg         : Linearity/Gain recipe for the optical domain

that indicate that both versions of the detmon recipe are available.

You can test the recipe offline by calling

esorex --recipe-config=${HOME}/.esorex/detmon_{opt|ir}_lg_{instument}.rc detmon_opt_lg SOF

where SOF can be the set of files extracted from an operational linearity AB. Note that the DO_CLASS values (the raw type tags, which appear after the raw file name in the SOF, which are specified in {instrument}_classification.h are ON_RAW and OFF_RAW and might differ from tags used by the IP version. Note also that the PRO.CATG values of the GP products might differ from the IP version. The --recipe-config=${HOME}/.esorex/detmon_{opt|ir}_lg_{instument}.rc tells esorex to read the instrument specific parameters for recipe detmon_opt_lg (or detmon_ir_lg, whatever version is applicable) from the specified file under ${HOME}/.esorex.

Send your final detmon_{opt|ir}_lg_{instument}.rc to the detmon pipeline developer to reasons of maintenance and to include it in the GP used on Paranal.

Make the generic recipe call operational:
Change the recipe name in $DFO_CONFIG_DIR/OCA/${DFO_INSTRUMENT}_association.h. The result could look like:

  recipe detmon_ir_lg ;

edit config.processAB accordingly for the new recipe name and the new PRO.CATG (ed. 2011-11-07)

Generic pipeline detmon recipe

The generic pipeline detmon recipe is usable by many instruments with different detectors. Upgrades of this recipe will affect all users. Bug reports and tickets should not be addressed to the IP but to the detmon pipeline.

On the handling of fits extensions:

If the raw frames have no detector extensions, the default detmon recipe configuration --exts=0 allows to process them.

If the raw frames are multi-extension fits files the user can either

1) (monolithic processing mode) use --exts=-1 to process all extensions and generate products with same number of extensions as the raw frames.

2) (detector splitting processing mode) use --exts=n to get a product file with one extension corresponding to extension n.

detector specific configuration, meaning processing each detector with a different setup, is possible with esorex only in detector splitting processing mode but not in monolithic processing mode.

    esorex --recipe-config=${HOME}/.esorex/detmon_ir_lg_vircam_4.rc  detmon_ir_lg --ext=4  SOF
    esorex --recipe-config=${HOME}/.esorex/detmon_ir_lg_vircam_9.rc  detmon_ir_lg --ext=9  SOF
    esorex --recipe-config=${HOME}/.esorex/detmon_ir_lg_vircam_16.rc detmon_ir_lg --ext=16 SOF

meaning for each detector a separate detector specific configuration file has to be maintained. The DFOS tool processAB version 1.10 supports recipe specific configuration and detector specific configuration. See extraction of config.processAB :

# ======================================================================
# 2. esorex configuration file names for generic pipelines
# List here the name(s) of esorex configuration file(s) of general pipeline(s) (e.g. detmon)
# RAW_TYPE:     usually just one, but could be more
# DET_ID:       required only if you need to specify config files per detector (avoid if possible)
#               if not needed, just enter ANY, will be ignored
# CONFIG:       name of config file for generic pipeline, under $HOME/.esorex
# ============================================================================================
#GENERAL        RAW_TYPE        DET_ID  CONFIG
#Example for arm-specific configuration:
GENERAL         DETMON_IR_ARM   ANY     detmon_ir_lg_xshooter.rc
GENERAL         DETMON_UV_ARM   ANY     detmon_opt_lg_xshooter.rc

#Example for detector-specific configuration, with a default config file and
# several detector specific configuration files:
# GENERAL       DETMON          1       detmon_ir_lg_vircam_DET01.rc
# GENERAL       DETMON          2       detmon_ir_lg_vircam.rc
# GENERAL       DETMON          3       detmon_ir_lg_vircam_DET03.rc
# GENERAL       DETMON          4       detmon_ir_lg_vircam_DET04.rc
# GENERAL       DETMON          5       detmon_ir_lg_vircam_DET05.rc
# GENERAL       DETMON          6       detmon_ir_lg_vircam.rc
# GENERAL       DETMON          7       detmon_ir_lg_vircam_DET07.rc
# GENERAL       DETMON          8       detmon_ir_lg_vircam.rc
# GENERAL       DETMON          9       detmon_ir_lg_vircam.rc
# GENERAL       DETMON          10      detmon_ir_lg_vircam_DET10.rc
# GENERAL       DETMON          11      detmon_ir_lg_vircam_DET11.rc
# GENERAL       DETMON          12      detmon_ir_lg_vircam_DET12.rc
# GENERAL       DETMON          13      detmon_ir_lg_vircam_DET13.rc
# GENERAL       DETMON          14      detmon_ir_lg_vircam_DET14.rc
# GENERAL       DETMON          15      detmon_ir_lg_vircam_DET15.rc
# GENERAL       DETMON          16      detmon_ir_lg_vircam_DET16.rc
# ======================================================================