Common DFOS tools:
|
dfos = Data Flow Operations System, the common tool set for DFO |
![]() |
new: | see also: | ||||||||
v2.0.1: v2.1: v2.1.1: Please call the tool once, without parameters, to make this happen. |
tools related to calSelector: general overview of calSelector here - Find all calibration maps here. |
|||||||||
|
|
|||||||||
topics: description | syntax rules | types of rules | validity | output | consistency checks | how to use | tool configuration | comments | operational hints |
[Historical: For the transition from the old to the new version of calSelector, check the migration description here.]
For a given instrument, the calibration cascade is the fundamental relationship between the data types (calibrations and science). The cascade is coded in OCA syntax (organisation, classification, association). In the dfos system, this information is structured in three main configuration files, with further files containing macros. These files are compiled by the OCA tools (createAB, calChecker) into a single OCA rules file, named <instr>.RLS.
Several dfos applications use OCA syntax. There is dfos daily operations, using createAB to generate ABs for processing and packing. There is calChecker, creating ABs to analyze their completeness. There is calSelector, an application running behind the archive web interfaces and creating association information for the delivery of calibration data sets for a user-selected science file.
Each of these tools uses a dedicated OCA rule set, coming in different flavours: raw-to-master (dfos and calSelector), raw-to-raw (calChecker; see here for more). Apart from the modes, the rulesets also differ by completeness. For instance, Health Check calibrations are included in dfos but not in calChecker.
All data types are handled together in one single rule set, although every instrument has many sub-cascades: modes, arms, settings. These sub-cascades are sometimes loosely connected, via detector raw types, sometimes not at all. The single ruleset corresponds to the OCA rules file. This file is usually big, machine but not human readable. A key concept in this situation is visualization. Without visualization of the rules file, it is almost impossible to properly understand, check, and modify the calibration cascade. This visualization is the purpose of createCalibMap. Its output is called a calibration map.
The ruleset is structured top-down such that we start with the most fundamental calibrations (detector calibrations) and then add the more complex ones following the calibration cascade (e.g. flat fields, arclamp, standard stars). At the bottom the science data follow. The calibration map displays this arrangement turned by 90 degrees, ordered from left to right, with the simplest detector calibrations at left and the science data at the right end. The map is ordered as a table, where each raw type is sorted in a column, while its properties (raw_type, do_class, products, recipe name etc.) can be found in rows. Further to the cells, the map has also relations between cells, corresponding to the input products from other raw types as required for processing a given raw type.
Find an example of a calibration map here (produced by createCalibMap).
The tool createCalibMap parses the (gcc-compiled) rule set with name <inst>.RLS and works out the individual components of the calibration map and its relations: raw_types, grouping rules, products etc. These components are then turned into graphical elements to constitute the calibration map.
[gcc compilation is the process needed to go from the individual OCA files (<instr>_association.h etc.) to the rules file <instr>.RLS, with macro expansion and removal of comment lines. This is done within createAB.]
Most of, if not all, the parsing can be provided in an automatic way by the utility tool dfosRLSMassage.py that is called in option -C and also by createAB. Hence the sentence below is likely to be obsolete.
[The parsing cannot be done in a completely automatic way but needs some help in the form of conventions being followed. In most cases they result in some changes to be done in the OCA files. The modifications are not very challenging. These changes need to be done in the respective OCA files, not in the rules file directly (they would be overwritten by createAB or calChecker quickly!). This has also the advantage that these contain the macros which are easier to edit (if necessary at all) than the expanded strings.]
Note that these modifications are transparent to the OCA tools, they are required only for createCalibMap, with the modified OCA files still working fine with the OCA tools.
Here is an overview of the conventions for the OCA files (most if not all conventions are controlled by the utility tool dfosRLSmassage.py):
General rules:
- raw types will appear colour-coded by their CATG (ACQUISITION, CALIB, SCIENCE)
-
an acquisition should
have CATG = "ACQUISITION" (not:
CALIB or SCIENCE)
classification.h:
if
DPR.CATG=="CALIB" and DPR.TYPE=="BIAS,DETCHECK" and DPR.TECH=="IMAGE" then
{
RAW.TYPE = "IFLT";
DO.CLASS = "LINEARITY_BIAS";
PACK.DIR = "NONE";
CATG = "CALIB";
}
-
a classification record is recognized by the starting 'if',
it must be the first element [the parser uses (if $1=="if")]
- the classification rule must be one line [everything
until 'then']
- the CATG
must be the second-last row of each record [the last one being '}'].
An exceptional EXPOSE_TO_CALSELECTOR=T is recognized by the tool, if it follows the line with the RAW.TYPE definition.
organisation.h:
select
execute(ACTION_DETMON_BIAS)
from inputFiles where RAW.TYPE=="DETMON_BIAS"
group
by DET.READ.SPEED,TPL.START as (TPL_A,tpl);
- an organisation record is recognized by the starting 'select
execute' [the parser uses this as a marker]
-
it must
be one line up to RAW.TYPE==... [otherwise only the
first line will be recognized by the parser]
-
the 'group by' clause must be one line [otherwise
only the first line will be recognized by the parser]
action ACTION_SCIENCE_ARG { minRet = 1; maxRet = 1; select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.READ.SPEED==DET.READ.SPEED ; minRet = 1; maxRet = 1; select file as NF_LOCWIDTH from calibFiles where PRO.CATG=="NF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED ; minRet = 1; maxRet = 1; select file as FF_LOCWIDTH from calibFiles where PRO.CATG=="FF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED ; minRet = 1; maxRet = 1; select file as DISPERSION_SOLUTION from calibFiles where PRO.CATG=="DISPERSION_SOLUTION" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED ; minRet = 1; maxRet = 1; select file as STD_ARGSPECTRA from calibFiles where PRO.CATG=="STD_ARGSPECTRA" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.INS1.OPTI1.POS==INS1.OPTI1.POS and inputFile.DET.READ.SPEED==DET.READ.SPEED ; }
-
an association record is recognized to start with 'action' [the
parser uses (if $1=="action")]
- its end is recognized by the last '^}' [avoid
having ^} before (e.g. as part of the recipe section!) where ^} means } being the first character
in the line]
- select file as: must be one line [otherwise
tooltips with match rules get confused; the semicolon belongs to the next line]
- the label "as ... from" must be the same as the one in "PRO.CATG==..."
For CALSELECTOR rules, there is the additional requirement to normally have the validity of the association encoded as tryBetween/between pair:
select file as DISPERSION_SOLUTION from calibFiles where PRO.CATG=="DISPERSION_SOLUTION" and ... and MJD-OBS between inputFile.MJD-OBS - <ext_val> and inputFile.MJD-OBS + <ext_val> and MJD-OBS tryBetween inputFile.MJD-OBS - <val> and inputFile.MJD-OBS + <val> ;
where <val> is the calibration plan validity (for migrated rules: taken from the DELTAT_RULE in the DFOS_OPS rules), and <gen_val> an extended (relaxed) validity, used for performance and to deliver a master calibration with a disclaimer if no instance could be found within the <val> range. Both <val> and <gen_val> can be asymmetric.
Find more under 'validity', and here.
If you have applied changes to your RLS file (other than the format changes), call createCalibMap -r <rules file> -C, in order to gcc-compile your RLS file properly.
Supported
types of OCA rule sets.
There are three supported types of OCA rule sets. These types are configured in the respective tool configuration file (see below). These are the types:
types | association | DO_CLASS |
VALIDITY supported |
complete description of all raw types? |
static calibrations used? | versioning supported? | $RLS_PATH |
DFOS_OPS | raw to master | yes | no | yes | yes | no | $DFO_CONFIG_DIR/OCA |
CALCHECKER | raw to raw | no | yes | no | no | no | $DFO_CONFIG_DIR/CALCHECK/OCA |
CALSELECTOR | raw to master | yes | yes | yes* | yes | yes | $DFO_CONFIG_DIR/CALSELECTOR/<start_date> |
You have the option to provide comments to the calib map. Comments are collected in a comment file called info.createCalibMap (in the same path as the config file). There are the following types of comments:
comment type | displayed where? |
whole calib map | top table, "General comment" |
sub-cascade | same |
raw_type | bottom part of calib map, in raw_type column |
Find the description of the comment file here.
Validity is the time range in which a calibration can (likely) be applied. It describes the empirical timescale of changes in the parameters to be measured by the calibration. This concept makes sense only in the framework of associations, i.e. in the relation between two files (science and calibration). An individual calibration is always valid if it has measured the target property in a valid way.
The concept of validity is implicit in the DFOS_OPS rules: there is the DELTAT_RULE part in the <instr>_associations.h files which however is not evaluated by OCA itself but is used by createAB to flag the ABs in case of violation. For calChecker, validity is evaluated by the tool itself. For calSelector v2.0, the same validity information as for the DFOS_OPS rules is required, but it must be coded within OCA. This is done in the automatic migration by parsing the DELTAT_RULEs and translating them into a combined between and tryBetween statement.
The between statement (-30...30) gets all candidates matching between -30 and 30 days; from those candidates, the closest-in-time is taken. Depending on whether the match is within the inner boundaries (defined by tryBetween) or only the outer boundaries, calSelector then returns a flag "calib_plan" (for the tryBetween match being satisfied) or "extended" (if not). In that way we can convey the QC concept of 'OK' or 'NOK' for time matches and avoid delivering no return data sets at all if there is only a mild violation of the formal validity. The logical justification of the two statements is that in DFOS operations, we have the calibration plan matches (corresponding to tryBetween) but we also accept (at least in many cases) a mild violation if no closer match is possible. With calSelector v2 we cannot convey the message that this particular exception is acceptable, but we can generally offer those candidates, with a disclaimer.
For static calibrations, usually the PREVIOUS time match should be applied.
createCalibMap has functions to check and display the validity (time-match) information in the calibration map, see here.
The tool creates the following output:
Here, <name> is the configured type of OCA rule set (one of DFOS_OPS, CALCHECKER, CALSELECTOR).
Find all calibration maps from http://www.eso.org/qc/ALL/OCA/oca_rule_sets.html .
There are some consistency checks performed by the tool. They are marked in the product
match section of the HTML output file:
Consistency
checks
in the product match rule (meaning minRet=0; maxRet=0):
obsolete match, could be removed from <instr>_association.h
this cell has multiple match rules between a given
raw_type and a given product pro_catg. This could indicate a logical problem although
it does not fire an OCA error. It creates a display issue for the calibration
map, which is why only the first match is displayed.
unconstrained
minRet, maxRet undefined in <instr>_association.h:
could be risky
as background of a cell with a match rule (and tooltip "none"):
no match rule found, could be a violation of the "one-line
rule" (see above under association.h) or just a very simple match
by PRO.CATG only .
CALSELECTOR: a time-match rule is missing for a general calibrations
(GT) or a normal calibration (T)
The tool also checks the completeness of the RLS file:
- raw_types without products are flagged in the ANALYSIS section, first line, for OCA ruleset type DFOS_OPS and CALCHECKER (CALIB only); this indicates that these data never produce virtual products and cannot be associated to other data types, which is ok for Health Check calibrations but worth checking;
- products listed for matching but not listed as products of raw_type (type DFOS_OPS; ANALYSIS section, second line): these could either be static calibrations, or configuration errors;
- unconstrained matches (minRet, maxRet): these could be potentially dangerous if more than one instance could exist.
The tool assumes that there is exactly one action per raw_type. In those cases where more than one action is defined, the tool raises a warning and takes the first defined action.
There are legal cases with multiple raw_type definitions (most likely when there is the need for multiple DO_CLASSes, as e.g. for the IR instruments with ON/OFF definitions). That raw_type is then marked as M, the multiple definitions are connected by 'OR', and the DO_CLASSes listed comma-separated.
Additional markings
Certain properties are highlighted in the calibration maps:
- type DFOS_OPS: those products which are listed but not used further are marked in italics
- type DFOS_OPS: MASSOCs are marked
- types CALCHECKER and CALSELECTOR: validity in days
- type CALSELECTOR: proper time-match rule for gencalibs (<); valdity for between and tryBetween (see the codes on the result page)
- type CALSELECTOR: the property EXPOSE_TO_CALSELECTOR=T is marked.
Type createCalibMap -h | -v for on-line help and version.
Type
createCalibMap -r <path>/<your_instr>.RLS [-m ALL]
(complete pathname!) to create the calibration map for all <your_instr> modes, and
createCalibMap -r <path>/<your_instr>.RLS -m IMG
to create the calibration map for <your_instr> mode IMG. The default is $DFO_CONFIG_DIR/OCA/<your_instr>.RLS.
The type of rules is configured in the corresponding tool configuration file config.createCalibMap under the same path as the OCA rule set. For the operational DFOS rules, the type is DFOS_OPS, and the path is $DFO_CONFIG_DIR/OCA.
Type
createCalibMap -r <path>/<your_instr>.RLS -D [-m <mode>]
to check all raw_types and all pro_catgs defined (or for a specified mode) in the rules file.
Type
createCalibMap -r <path>/<your_instr>.RLS -A
to create the calibration maps for all modes configured in the tool config file, and
createCalibMap -r <path>/<your_instr>.RLS -E <selected science raw_type >
to get a listing of the dependencies for the specified pro_catg.
Replace <path>/<your_instr>.RLS in the above examples by the pathname of your operational rules file (e.g. for the calChecker calibmap of HAWKI: -r $DFO_CONFIG_DIR/CALCHECK/OCA/HAWKI.RLS).
createCalibMap -X
to enter a dialog for bulk creation of calib maps. The tool offers a menue. You select an option (one of DFOS_OPS, CALCHECKER, CALSELECTOR) and then the tool creates all calib maps for that option and the complete overview, without further asking.
The tool has its configuration file config.createCalibMap which comes in one version per rules file, and resides in the same path. For instance, the configuration file for the calChecker calibration map resides under $DFO_CONFIG_DIR/CALCHECK/OCA.
For the CALSELECTOR configuration, there is the need for versioning. This is solved by having one subdirectory for each version, with the start-date (first date of validity) being the label for the version and the subfolder: $DFO_CONFIG_DIR/CALSELECTOR/<start_date>. To manage the various versions, there is the configuration file config.createCalibMap.calsel in $DFO_CONFIG_DIR/CALSELECTOR.
DFOS_OPS | CALCHECKER | CALSELECTOR | ||
$DFO_CONFIG_DIR/OCA | $DFO_CONFIG_DIR/CALCHECK | $DFO_CONFIG_DIR/CALSELECTOR | ||
config.createCalibMap | config.createCalibMap | config.createCalibMap.calsel | ||
<date1> (version) | <date2> | <date3> | ||
config.createCalibMap | config.createCalibMap | config.createCalibMap |
The configuration file config.createCalibMap has the following structure:
1. General parameters | ||
NAME |
DFOS_OPS | type of this cascade (one of DFOS_OPS, CALCHECKER, CALSELECTOR) |
#INSTRUMENT | GIRAFFE | optional, default: $DFO_INSTRUMENT |
KEEP_SHORT | YES | YES|NO; YES: tags will be shortened to the value of SHORT_CHAR default: YES; if you mouse over, the full value will display |
SHORT_CHAR | 15 | value to which tags will be shortened if $KEEP_SHORT=YES; default: 15 |
2. Sub-cascades |
||
ALL_MODES | DET |
DET (=detector) is a reserved flag for calibrations assumed to be shared among all modes. (You could also include ACQ here.) |
ALL_MODES | MEDUSA | List of all SCIENCE modes |
ALL_MODES | etc. | |
2.2 All RAW_TYPEs per mode The ones listed here will appear on the output. The ones not listed here will show up only for mode ALL. one block per mode; list all raw_types which constitute the sub-cascade: ACQ, CAL, SCI |
||
MODE (tag) | RAW_TYPE | MODE |
MODE | FFLAT | IFU |
MODE | STD_IFU | IFU |
The configuration file config.createCalibMap.calsel has the following structure:
1. Versions for CALSELECTOR configuration |
There is one row per version. A version is defined by the following keys: |
START_DATE 2008-05-01 &&applicable after CCD upgrade&& |
The comment file is optional. It is called info.createCalibMap (in the same path as the config file).
Comments: 1. MODE_COMMENT: their qualifier is either 'ALL' or the mode |
||
TAG |
Qualifier | Content |
MODE_COMMENT
|
ALL | &&A comment applicable to the full cascade&& |
MODE_COMMENT | MEDUSA | &&A Medusa commment&& |
|
||
RT_COMMENT | DARK |
&&HC calibs; used for cascade with old CCD&& |
RT_COMMENT | FFLAT | &&normal fibre flats&& |
RT_COMMENT | etc. |
Syntax.
Sub-cascades.
createCalibMap supports sub-cascades. The concept of sub-cascades is not part of OCA but very useful for calib maps. It helps reducing complexity and makes dependencies much more obvious.
Configuration of raw types.
Finishing
References
Use the already existing calibration maps under http://www.eso.org/observing/dfo/quality/ALL/OCA/oca_rule_sets.html for reference.
Concluding remarks
Yes, it's kind of complicated but certainly pays off.
Last update: April 26, 2021 by rhanusch |