Common DFOS tools:
Documentation

make printable	new:	see also:
	v2.0.1: - supports complex between/tryBetween timematch rules v2.1: - supports EXPOSE_TO_CALSELECTOR=T v2.1.1: - uploads list_static_mcalibs to $DFO_WEB_SERVER for OPSHUB tool Please call the tool once, without parameters, to make this happen.	tools related to calSelector: calselManager | createCalibMap | verifyAB | writeBreakpoint general overview of calSelector here OCA: rules | syntax - Find all calibration maps here. - The CALCHECKER and DFOS_OPS calib maps are linked to other pages.
		databases none dfos tools dfosRLSMassage.py output html pages under http://www.eso.org/qc/ALL/OCA upload/download html pages and RLS files to http://www.eso.org/qc/ALL/OCA
topics: description | syntax rules | types of rules | validity | output | consistency checks | how to use | tool configuration | comments | operational hints

createCalibMap

[Historical: For the transition from the old to the new version of calSelector, check the migration description here.]

Description

For a given instrument, the calibration cascade is the fundamental relationship between the data types (calibrations and science). The cascade is coded in OCA syntax (organisation, classification, association). In the dfos system, this information is structured in three main configuration files, with further files containing macros. These files are compiled by the OCA tools (createAB, calChecker) into a single OCA rules file, named <instr>.RLS.

Several dfos applications use OCA syntax. There is dfos daily operations, using createAB to generate ABs for processing and packing. There is calChecker, creating ABs to analyze their completeness. There is calSelector, an application running behind the archive web interfaces and creating association information for the delivery of calibration data sets for a user-selected science file.

Each of these tools uses a dedicated OCA rule set, coming in different flavours: raw-to-master (dfos and calSelector), raw-to-raw (calChecker; see here for more). Apart from the modes, the rulesets also differ by completeness. For instance, Health Check calibrations are included in dfos but not in calChecker.

All data types are handled together in one single rule set, although every instrument has many sub-cascades: modes, arms, settings. These sub-cascades are sometimes loosely connected, via detector raw types, sometimes not at all. The single ruleset corresponds to the OCA rules file. This file is usually big, machine but not human readable. A key concept in this situation is visualization. Without visualization of the rules file, it is almost impossible to properly understand, check, and modify the calibration cascade. This visualization is the purpose of createCalibMap. Its output is called a calibration map.

The ruleset is structured top-down such that we start with the most fundamental calibrations (detector calibrations) and then add the more complex ones following the calibration cascade (e.g. flat fields, arclamp, standard stars). At the bottom the science data follow. The calibration map displays this arrangement turned by 90 degrees, ordered from left to right, with the simplest detector calibrations at left and the science data at the right end. The map is ordered as a table, where each raw type is sorted in a column, while its properties (raw_type, do_class, products, recipe name etc.) can be found in rows. Further to the cells, the map has also relations between cells, corresponding to the input products from other raw types as required for processing a given raw type.

Find an example of a calibration map here (produced by createCalibMap).

Business rules for syntax.

The tool createCalibMap parses the (gcc-compiled) rule set with name <inst>.RLS and works out the individual components of the calibration map and its relations: raw_types, grouping rules, products etc. These components are then turned into graphical elements to constitute the calibration map.

[gcc compilation is the process needed to go from the individual OCA files (<instr>_association.h etc.) to the rules file <instr>.RLS, with macro expansion and removal of comment lines. This is done within createAB.]

Most of, if not all, the parsing can be provided in an automatic way by the utility tool dfosRLSMassage.py that is called in option -C and also by createAB. Hence the sentence below is likely to be obsolete.

[The parsing cannot be done in a completely automatic way but needs some help in the form of conventions being followed. In most cases they result in some changes to be done in the OCA files. The modifications are not very challenging. These changes need to be done in the respective OCA files, not in the rules file directly (they would be overwritten by createAB or calChecker quickly!). This has also the advantage that these contain the macros which are easier to edit (if necessary at all) than the expanded strings.]

Note that these modifications are transparent to the OCA tools, they are required only for createCalibMap, with the modified OCA files still working fine with the OCA tools.

Here is an overview of the conventions for the OCA files (most if not all conventions are controlled by the utility tool dfosRLSmassage.py):

General rules:
- raw types will appear colour-coded by their CATG (ACQUISITION, CALIB, SCIENCE)
- an acquisition should have CATG = "ACQUISITION" (not: CALIB or SCIENCE)

classification.h:

if DPR.CATG=="CALIB" and DPR.TYPE=="BIAS,DETCHECK" and DPR.TECH=="IMAGE" then
{
  RAW.TYPE = "IFLT";
  DO.CLASS = "LINEARITY_BIAS";
  PACK.DIR = "NONE";
  CATG = "CALIB";
}
- a classification record is recognized by the starting 'if', it must be the first element [the parser uses (if $1=="if")]
- the classification rule must be one line [everything until 'then']
- the CATG must be the second-last row of each record [the last one being '}'].

An exceptional EXPOSE_TO_CALSELECTOR=T is recognized by the tool, if it follows the line with the RAW.TYPE definition.

organisation.h:

select execute(ACTION_DETMON_BIAS) from inputFiles where RAW.TYPE=="DETMON_BIAS"
   group by DET.READ.SPEED,TPL.START as (TPL_A,tpl);

- an organisation record is recognized by the starting 'select execute' [the parser uses this as a marker]
- it must be one line up to RAW.TYPE==... [otherwise only the first line will be recognized by the parser]
- the 'group by' clause must be one line [otherwise only the first line will be recognized by the parser]

action ACTION_SCIENCE_ARG
{
  minRet = 1; maxRet = 1;
  select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.READ.SPEED==DET.READ.SPEED
  ;
  minRet = 1; maxRet = 1;
  select file as NF_LOCWIDTH from calibFiles where PRO.CATG=="NF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED
  ;
  minRet = 1; maxRet = 1;
  select file as FF_LOCWIDTH from calibFiles where PRO.CATG=="FF_LOCWIDTH" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED
  ;
  minRet = 1; maxRet = 1;
  select file as DISPERSION_SOLUTION from calibFiles where PRO.CATG=="DISPERSION_SOLUTION" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.DET.READ.SPEED==DET.READ.SPEED
  ;
  minRet = 1; maxRet = 1;
  select file as STD_ARGSPECTRA from calibFiles where PRO.CATG=="STD_ARGSPECTRA" and inputFile.INS.SLIT.NAME==INS.SLIT.NAME and inputFile.INS.EXP.MODE==INS.EXP.MODE and inputFile.INS1.OPTI1.POS==INS1.OPTI1.POS and inputFile.DET.READ.SPEED==DET.READ.SPEED
  ;
}

- an association record is recognized to start with 'action' [the parser uses (if $1=="action")]
- its end is recognized by the last '^}' [avoid having ^} before (e.g. as part of the recipe section!) where ^} means } being the first character in the line]
- select file as: must be one line [otherwise tooltips with match rules get confused; the semicolon belongs to the next line]
- the label "as ... from" must be the same as the one in "PRO.CATG==..."

For CALSELECTOR rules, there is the additional requirement to normally have the validity of the association encoded as tryBetween/between pair:

 select file as DISPERSION_SOLUTION from calibFiles where PRO.CATG=="DISPERSION_SOLUTION" and ... and MJD-OBS between inputFile.MJD-OBS - <ext_val> and inputFile.MJD-OBS + <ext_val> and MJD-OBS tryBetween inputFile.MJD-OBS - <val> and inputFile.MJD-OBS + <val>  ;

where <val> is the calibration plan validity (for migrated rules: taken from the DELTAT_RULE in the DFOS_OPS rules), and <gen_val> an extended (relaxed) validity, used for performance and to deliver a master calibration with a disclaimer if no instance could be found within the <val> range. Both <val> and <gen_val> can be asymmetric.

Find more under 'validity', and here.

If you have applied changes to your RLS file (other than the format changes), call createCalibMap -r <rules file> -C, in order to gcc-compile your RLS file properly.

Supported types of OCA rule sets.

There are three supported types of OCA rule sets. These types are configured in the respective tool configuration file (see below). These are the types:

• DFOS_OPS: this is the classical rule set in $DFO_CONFIG_DIR/OCA/<instr>.RLS used for creating operational ABs. The association type is raw-to-master, where products can be either virtual (predicted at the time of AB creation, real but uncertified at AB execution time) or real (certified master calibrations, or static calibrations). DO_CLASS is defined and displayed by the tool.
• CALCHECKER: this is the rule set in $DFO_CONFIG_DIR/CALCHECK/OCA/<instr>.RLS used for creating calChecker ABs and evaluating their completeness information. The formal association type is raw-to-master where all products are virtual (predicted). The ABs have no permanent value, hence no DO_CLASS is needed. On the other hand, the validity information (although strictly speaking not supported by OCA) is displayed by the tool. The cascade is simplified in comparison to DFOS_OPS, no completeness of raw types is aimed at since calChecker concentrates on calibrations needed for science reduction (hence no HC calibrations are required).
• CALSELECTOR: this is the rule set in $DFO_CONFIG_DIR/CALSELECTOR/<start_date>/<instr>_raw2master.<start_date>.RLS. It is used for the same cases as DFOS_OPS, and is written in raw-to-master syntax. The rules file is read by calSelector, a tool not directly used by QCG, but QCG has the task to provide and maintain those rules. For this type, versioning is supported, via the <validity_date> (the config directory splits by these dates). With calSelector v2, the current CALSELECTOR ruleset can be identical to the DFOS_OPS ruleset.

Comments

You have the option to provide comments to the calib map. Comments are collected in a comment file called info.createCalibMap (in the same path as the config file). There are the following types of comments:

comment type	displayed where?
whole calib map	top table, "General comment"
sub-cascade	same
raw_type	bottom part of calib map, in raw_type column

Find the description of the comment file here.

Validity

Validity is the time range in which a calibration can (likely) be applied. It describes the empirical timescale of changes in the parameters to be measured by the calibration. This concept makes sense only in the framework of associations, i.e. in the relation between two files (science and calibration). An individual calibration is always valid if it has measured the target property in a valid way.

The concept of validity is implicit in the DFOS_OPS rules: there is the DELTAT_RULE part in the <instr>_associations.h files which however is not evaluated by OCA itself but is used by createAB to flag the ABs in case of violation. For calChecker, validity is evaluated by the tool itself. For calSelector v2.0, the same validity information as for the DFOS_OPS rules is required, but it must be coded within OCA. This is done in the automatic migration by parsing the DELTAT_RULEs and translating them into a combined between and tryBetween statement.

The between statement (-30...30) gets all candidates matching between -30 and 30 days; from those candidates, the closest-in-time is taken. Depending on whether the match is within the inner boundaries (defined by tryBetween) or only the outer boundaries, calSelector then returns a flag "calib_plan" (for the tryBetween match being satisfied) or "extended" (if not). In that way we can convey the QC concept of 'OK' or 'NOK' for time matches and avoid delivering no return data sets at all if there is only a mild violation of the formal validity. The logical justification of the two statements is that in DFOS operations, we have the calibration plan matches (corresponding to tryBetween) but we also accept (at least in many cases) a mild violation if no closer match is possible. With calSelector v2 we cannot convey the message that this particular exception is acceptable, but we can generally offer those candidates, with a disclaimer.

For static calibrations, usually the PREVIOUS time match should be applied.

createCalibMap has functions to check and display the validity (time-match) information in the calibration map, see here.

Output

The tool creates the following output:

• HTML file called cascade_<ins>_<mode>.html under $RLS_PATH
• exported version of that file on http://www.eso.org/qc/ALL/OCA/<name>
• exported RLS file (same URL)

Here, <name> is the configured type of OCA rule set (one of DFOS_OPS, CALCHECKER, CALSELECTOR).

Find all calibration maps from http://www.eso.org/qc/ALL/OCA/oca_rule_sets.html .

Consistency checks

There are some consistency checks performed by the tool. They are marked in the product match section of the HTML output file:

0,0	in the product match rule (meaning minRet=0; maxRet=0): obsolete match, could be removed from <instr>_association.h
M 0,1	this cell has multiple match rules between a given raw_type and a given product pro_catg. This could indicate a logical problem although it does not fire an OCA error. It creates a display issue for the calibration map, which is why only the first match is displayed.
unconstrained	minRet, maxRet undefined in <instr>_association.h: could be risky
	as background of a cell with a match rule (and tooltip "none"): no match rule found, could be a violation of the "one-line rule" (see above under association.h) or just a very simple match by PRO.CATG only .
GT or T	CALSELECTOR: a time-match rule is missing for a general calibrations (GT) or a normal calibration (T)

The tool also checks the completeness of the RLS file:

- raw_types without products are flagged in the ANALYSIS section, first line, for OCA ruleset type DFOS_OPS and CALCHECKER (CALIB only); this indicates that these data never produce virtual products and cannot be associated to other data types, which is ok for Health Check calibrations but worth checking;

- products listed for matching but not listed as products of raw_type (type DFOS_OPS; ANALYSIS section, second line): these could either be static calibrations, or configuration errors;

The tool assumes that there is exactly one action per raw_type. In those cases where more than one action is defined, the tool raises a warning and takes the first defined action.

There are legal cases with multiple raw_type definitions (most likely when there is the need for multiple DO_CLASSes, as e.g. for the IR instruments with ON/OFF definitions). That raw_type is then marked as M, the multiple definitions are connected by 'OR', and the DO_CLASSes listed comma-separated.

- type DFOS_OPS: those products which are listed but not used further are marked in italics

- type DFOS_OPS: MASSOCs are marked

- type CALSELECTOR: proper time-match rule for gencalibs (<); valdity for between and tryBetween (see the codes on the result page)

- type CALSELECTOR: the property EXPOSE_TO_CALSELECTOR=T is marked.

How to use

Type createCalibMap -h | -v for on-line help and version.

Type

createCalibMap -r <path>/<your_instr>.RLS [-m ALL]

(complete pathname!) to create the calibration map for all <your_instr> modes, and

createCalibMap -r <path>/<your_instr>.RLS -m IMG

to create the calibration map for <your_instr> mode IMG. The default is $DFO_CONFIG_DIR/OCA/<your_instr>.RLS.

The type of rules is configured in the corresponding tool configuration file config.createCalibMap under the same path as the OCA rule set. For the operational DFOS rules, the type is DFOS_OPS, and the path is $DFO_CONFIG_DIR/OCA.

Type

createCalibMap -r <path>/<your_instr>.RLS -D [-m <mode>]

to check all raw_types and all pro_catgs defined (or for a specified mode) in the rules file.

Type

createCalibMap -r <path>/<your_instr>.RLS -A

to create the calibration maps for all modes configured in the tool config file, and

createCalibMap -r <path>/<your_instr>.RLS -E <selected science raw_type >

to get a listing of the dependencies for the specified pro_catg.

Replace <path>/<your_instr>.RLS in the above examples by the pathname of your operational rules file (e.g. for the calChecker calibmap of HAWKI: -r $DFO_CONFIG_DIR/CALCHECK/OCA/HAWKI.RLS).

Type

createCalibMap -X

to enter a dialog for bulk creation of calib maps. The tool offers a menue. You select an option (one of DFOS_OPS, CALCHECKER, CALSELECTOR) and then the tool creates all calib maps for that option and the complete overview, without further asking.

Configuration files

The tool has its configuration file config.createCalibMap which comes in one version per rules file, and resides in the same path. For instance, the configuration file for the calChecker calibration map resides under $DFO_CONFIG_DIR/CALCHECK/OCA.

For the CALSELECTOR configuration, there is the need for versioning. This is solved by having one subdirectory for each version, with the start-date (first date of validity) being the label for the version and the subfolder: $DFO_CONFIG_DIR/CALSELECTOR/<start_date>. To manage the various versions, there is the configuration file config.createCalibMap.calsel in $DFO_CONFIG_DIR/CALSELECTOR.

The configuration file config.createCalibMap has the following structure:

The configuration file config.createCalibMap.calsel has the following structure:

Comment file

The comment file is optional. It is called info.createCalibMap (in the same path as the config file).

Operational hints.

Syntax.

• Make sure that the above syntax rules for the RLS files are followed, otherwise the parsing might be incomplete, or even fail.
• If you modify the syntax: always do this in the applicable original OCA file (like<instr>_classification.h etc.) and then call
    createCalibMap -r <rules_file> -C
  to compile the rules file. Any direct modification in the rules file itself would eventually get lost.

Sub-cascades.

createCalibMap supports sub-cascades. The concept of sub-cascades is not part of OCA but very useful for calib maps. It helps reducing complexity and makes dependencies much more obvious.

• Start with the full cascade, this is the default mode ALL. It will display all defined raw types, including stand-alone health checks. Control the sequence of raw types in the classification.h file. (Note that for complex cascades the analysis of the full cascade can take long [dozens of minutes]. You might be surprised to see how many raw types and products there are in your rules file).
• Usually the ALL cascade is not very useful for any practical purposes. It might be more than a hundred raw_types wide (127 for XSHOOTER) and more than a hundred pro_catg's high (162 for XSHOOTER). It is however the complete picture.
• Then start thinking in sub-cascades. Since OCA does not know sub-cascades, you have to define them. There are no general rules, but the following are good criteria for the definition of sub-cascades:
  • by instrument modes (e.g. IMG, MOS, LSS)
  • by instrument arms (e.g. UBV, VIS, NIR)
  • by observational techniques (STARE, NOD, JITTER)
• If your sub-cascades are not very obvious, start exploring dependencies. This you can do with the option -E <science raw_type>. It supports exactly one science raw_type and gives you all associated CALIB raw_types. Configure the mode under ALL_MODES, and the raw_types under MODE in config.createCalibMap. There are situations when this defines already a mode, but there could also be the situation that you need to combine several such raw_type's under one mode (generally there is not a 1:1 correspondance between -E and -m). You then call the option -E several times and configure the aggregated results. Call the tool with option -m <mode> and check the result until you are satisfied.
• Define DET for the 'detector' calibrations, and/or for ACQuisitions. These data types will then appear in every sub-cascade, without the need to define them again.
• Do not confuse modes with setting parameters like filter, grating etc.: these do not establish sub-cascades (these would then all be identical!).
• A good guideline for sub-cascades and their completeness is that generally one (or a few) SCIENCE raw_type(s) in the DFOS_OPS cascade define(s) one sub-cascade.
• For type CALCHECKER, you should use the raw_type names as defined on the calChecker interface (column "data types"). In simple cases, each of them defines a mode. But in many cases several of them can be grouped together into a mode, so in general the business rule for CALCHECKER calib maps is: use the same SCIENCE raw_types as for calChecker, but feel free to combine them into modes. In that way it becomes possible to cross-link the calChecker "data types" column directly to the calib maps.

Configuration of raw types.

• Explore your selected science raw_type:
    createCalibMap -r <rules_file> -E <selected science raw_type>
  which delivers a listing of the dependencies for that science raw_type. Configure that raw_type in config.createCalibMap under a suitable MODE. Register all MODEs under ALL_MODES. Ignore all static calibrations, they will be found by the tool.
• Continue with the other sub-cascades. Write their raw_types and MODEs into the section ALL_MODES of the tool configuration file, in the same order you want them in the output.
• Call
    createCalibMap -r <rules_file> -D [-m <mode>]
  to check all raw_types and all pro_catgs defined (for a specified mode) in the rules file.
• This helps to decide which of those you may want to include in the tool configuration:
  • if a mode is specified, only the configured raw_types are displayed in the sub-cascade.
  • if no mode (or -m ALL) is specified, all possible raw_types are displayed, including the ones without any dependencies like HC calibrations.
• This is a good strategy to focus on the important components. After all, the output of this tool is used for visualization and for consistency checks.
• Create and control the output by calling
    createCalibMap -r <rules_file> -m <mode>
• The output is displayed on http://www.eso.org/observing/dfo/quality/ALL/OCA . The wrapper page http://www.eso.org/observing/dfo/quality/ALL/OCA/oca_rule_sets.html will be edited to contain all calibration maps.

Finishing

• Do this for all sub-cascades, and do not forget to list them (in the order of your choice) in the ALL_MODES section.
• Call
    createCalibMap -r <rules_file> -m <mode>
  to create the calibration map for that mode, and
    createCalibMap -r <rules_file> -A
  to create all sub-cascades defined in ALL_MODES (except for the DET part).
• Once you have done the initial maps, updates are easiest managed by calling
    createCalibMap -X
  with the appropriate selection.
• Beware of traps:
  having a valid config file in the CALSELECTOR path but having a wrong NAME (e.g. DFOS_OPS since you probably derived it from there) will result in strange effects (like the CALSELECTOR calibmap overwriting your DFOS_OPS calibmap).

Further hints

• For DFOS_OPS and CALCHECKER calib maps, the "grouping rules" should contain the setup keys as per original OCA configuration. Although they are not needed for the OCA grouping, they are evaluated as setup keys by tools like scoreQC and the AB monitor (getStatusAB).
• There is no CALCHECKER CAL4CAL map. The corresponding OCA file is in general not complete, since it contains only the raw_type definitions required for the triggering calibs (having CATG=SCIENCE), while the classifications of the dependent CALIBs are just copied from the general calChecker part.
• There are sometimes multiple definitions for raw_types in DFOS_OPS rules. These are included in the calib map as just one single column, with all found definitions linked by an OR, and all corresponding DO_CLASSes linked by a comma.
• For DFOS_OPS, it is quite common that CALIB products required for SCIENCE raw_types are not defined in the OCA rule set (to avoid undesired association of virtual mcalibs to science data). The cascade is then incomplete, which has no direct consequences for DFOS processing. But for the migration to CALSELECTOR rules, this information is missing, cannot be reconstructed by a tool, and leads to the consequence that the corresponding raw files cannot be associated. Therefore, all required pro.catgs should always be included in the DFOS_OPS OCA rule set.
• For option -C, the tool calls the utility tool dfosRLSMassage.py which provides the correct edits for an input RLS to comply with the business rules from above. The same tool is also called by createAB.
• The CALCHECKER and DFOS_OPS calib maps are linked to the calChecker and the AB monitor pages, resp. Make sure to keep them up-to-date.

References

Use the already existing calibration maps under http://www.eso.org/observing/dfo/quality/ALL/OCA/oca_rule_sets.html for reference.

Concluding remarks

Yes, it's kind of complicated but certainly pays off.