Common DFOS tools:
Documentation

make printable	new:	see also:
	v3.2: - added '-m VALIDATE'	tools related to calSelector: calselManager | createCalibMap | verifyAB | writeBreakpoint general overview of calSelector here OCA: rules | syntax
	v3.1: - tool modified to support CalSelector v3 (breakpoint support added); no changes to functionality, only to internal call syntax	databases associations for the rules dfos tools dfosRLSMassage.py output <instr>_raw2master.${VAL_DATE}.RLS in $DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE} upload/download upload: RLS file to http://www.eso.org/qc/ALL/OCA
topics: description || task 1: goal | alignment rules | rules editing | consistency checks | modify rules || task 2: rule management || config file | operational hints | tips and tricks
Find a database overview (offered to the public) of all current calSelector rules here. The internal overview of all OCA rules (including the calSelector ones) is here. The public documentation is here. Modifications to OCA rule file if you migrate from DFOS_OPS to CALSELECTOR:: each SCIENCE raw_type needs (at least) one product defined (in order to define the dataset, see below) 'between' and 'tryBetween' time match to be added (auto-migrated from DELTAT_RULE but needs a check) CLOSEST time match is obsolete (you can remove the macro if used, it is the default anyway, and its definition in OCA_macro.h is in conflict with the 'between' time match) for RASSOC files: make sure to use no macros when you use their raw_types in the association section

calselManager

Description

This tool manages the association rule files (OCA rules) for calSelector. It supports two tasks:

• the alignment of the DFOS_OPS OCA rules with the CALSELECTOR OCA rules, and
• the management of the database ingestion of the CALSELECTOR rules set.

Testing of OCA rules is done with verifyAB.

Task 1: Align DFOS_OPS and current CALSELECTOR

Goal of alignment; basics about calSelector

calSelector is the archive tool to select calibrations for input science data. It is generally driven by science data, with a few exceptions where standard stars are also supported. It dynamically searches, for each input file, all master calibrations necessary for the reduction process. This mode is called "Raw2Master (R2M)". The use case in mind is a user who wants to reduce a certain science file, using the archived master calibrations (to run the science recipe only).

If one or more required master calibrations cannot be found, the tool switches to the Raw2Raw (R2R) mode and delivers the raw data from the entire cascade up to the science reduction. (The same happens if the archive user has decided to go for raw calibrations.) In that use case, all steps in the calibration cascade need to be executed. Typical examples for this use case are: users deliberately want to start from scratch; users want to reduce data from epochs for which no, or not all, master calibrations exist in the archive; users want to reduce data from very recent phases ("ToO regime") for which masters have not yet been certified; users want to process data from pipeline-unsupported modes, with their own software. More about the calSelector tool here ...

calSelector follows the following strategy:

• find the proper OCA rule to apply (per instrument and mjd_obs period);
• if R2M is chosen by the user: find all required master calibs, plus add MASSOC/RASSOC ones;
• if the result set is complete, the task is done;
• if not, go R2R and search for all raw calibrations needed to process the entire cascade;
• if R2R is chosen by the user: search for all raw calibrations needed to process the entire cascade.

In R2R mode (no matter if chosen by the user or by the tool), the tool gives preference to raw calibrations with the 'certified' flag (these are the ones which have been used in a master calibration and are certified by definition). Only if none are found, any raw calibration is associated (i.e. calibrations with unknown quality).

It is important to note that the CALSELECTOR OCA rules must contain all data types (in the above sense) and not only the ones needed directly for science reduction. If the tool returns an incomplete dataset for the R2M case, it switches automatically to the R2R mode (but still based on the same R2M OCA rule!), evaluates "virtual products" previously stored in the archive, and returns a raw file dataset. For this to be complete, the whole calibration cascade needs to be known to calSelector.

It is important to note that all SCIENCE raw_types should have at least one product defined. (The DFOS_OPS OCA rules have this usually implemented.)

The calSelector rules are formulated in R2M syntax. Hence there is no big difference between the OCA rules for the DFOS_OPS and the (current) CALSELECTOR cases.

With v3, calSelector also respects the breakpoints as entered in the corresponding database. Associations between science files and calibration files are not allowed to cross a defined breakpoint (either for all raw types or for the specified ones).

In exceptional cases you can also force calSelector in the OCA rule to go Raw2Raw. See here for more.

Business rules for alignment.

The tool calselManager provides support for keeping the DFOS_OPS OCA rules and the CALSELECTOR OCA rules aligned.

The alignment cannot be achieved in a completely automated way but needs some editing. The tool creates a copy of the operational OCA rules, pre-edits it, leaves additional checks to the user, and then continues with the edited file. The user can at any time repeat the pre-editing (it will not overwrite the user-edited version), or repeat or iterate and then continue with the automatic part.

There are a few business rules:

• static master calibrations normally need a PREVIOUS time match rule (but there could be exceptions)
• other master calibrations should not contain time match rules before the migration, these are added automatically from the DELTAT_RULEs (but there could be exceptions).

Rule editing: align DFOS_OPS and CALSELECTOR rule

Here is an overview of the pre-editing done by the tool, along with a description of what needs to be hand-edited. This description is also available on-line, type calselManager .

Step 1: Call the tool

Call

calselManager [-r <pathname to rules file>]

The default for the rules is the DFOS_OPS rule under $DFO_CONFIG_DIR/OCA/<instrument>.RLS. If you launch the tool without pathname specification, it asks for confirmation.

The tool starts a dialog, checking for the OCA rules to contain a product definition for every raw type (this is required to later create the virtual products).

If you decide to modify your DFOS_OPS OCA rules, you should keep in mind that the rules file is actually always recreated from scratch by createAB. In order to have permanent modifications, you need to edit one of the source files (e.g. <instr>_association.h) and then gcc-compile (the dialog explains how).

Then the tool

• removes the obsolete PACK_DIR;
• checks for missing time match rules for gencalibs;
• checks for redundant or missing (no DELTAT_RULE) time match rules for master calibs.

It puts the output to $TMP_DIR/<instr>_raw2master.RLS1 for final inspection. [It uses the utility tool dfosRLSmassage.py for some automatic parsing.]

Step 2: Manual checks

- copy $TMP_DIR/<instr>_raw2master.RLS1 to $TMP_DIR/<instr>_raw2master.RLS
- edit $TMP_DIR/<instr>_raw2master.RLS
That simple mechanism protects you from unintentional overwrites.

Review and edit if necessary:

• section 2 (from organisation.h):
  if you have TPL-like entries other than TPL_A or TPL_D (e.g. TPL_I), you may want to remove them by hand for simplification.
  
• time-match rules: normally the gencalibs should have a PREVIOUS time match rule MJD-OBS < inputFile.MJD-OBS.
  This is required for logical reasons: if there is more than one version, this is due to an "intervention" (an instrument intervention, or a pipeline modification), defining a breakpoint, and you don't want to associate across breakpoints.
• for the other master calibrations, the tool has added the time match rule
  
  MJD-OBS between inputFile.MJD-OBS - 10 and inputFile.MJD-OBS + 10 and MJD-OBS tryBetween inputFile.MJD-OBS - <val> and inputFile.MJD-OBS + <val>
  
  The value of <val> is read from the DELTAT_RULE. The value 10 is hard-coded in the tool and can be changed as required.
  
  The meaning of this complex time match rule is:
  
  a) tryBetween: this is the validity from the calibration plan, it is based on likely timescales for variations of important instrument properties. If a master calibration is found within the 'tryBetween' range, it gets the flag 'calib_plan' in the output datasets (xml file), in order for the file to flag that this match is the best possible one. Logically, a 'calib_plan' match is the same as an 'OK' match for the DFOS system (marked on the AB monitor).
  
  b) between: this is the larger validity range which is used both for performance (all candidate matches are downloaded by the tool using the between time range), and for a relaxed match if no match can be found within the 'tryBetween' range. It is assumed that this match is still better than having the tool switch to the R2R mode where it delivers no master calibration at all. The corresponding flag in the xml file is 'extended' meaning for the user that we deliver this match with some disclaimer about the quality of the match (not of the mcalib!). Logically, such a match corresponds to the red flag "NOK" on the AB monitor.
  
  
• If you have in addition explicit time match rules in your DFOS_OPS rules, like the one based on the ALL macro, you need to check them since there might now be redundant time match rules.
• If you have a gencalib match without time match rule, the migration process discovers this and marks them with the tag CHECK MISSING MJD CONSTRAINT. You need to investigate and decide. You can remove the tag and leave the rules as they are if you think this makes sense.
  

Step 3: Pre-compilation (tool part)

Compile $TMP_DIR/<instr>_raw2master.RLS:

calselManager -C

This is to remove all comments etc.

Step 4: Upload to final directories

Enter a validity date and a comment, and store the precompiled file as <instr>_raw2master.${VAL_DATE}.RLS:

calselManager -U

Now your file will be stored as <instr>_raw2master.${VAL_DATE}.RLS in $DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE}, and exported for reference to
http://www.eso.org/qc/ALL/OCA/CALSELECTOR/<instr>/${VAL_DATE}.

Note: this'upload' is different from storing the rule in the database (using -m WRITE, see below).

The validity date is used to create a directory $DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE} (if not already existing). It hosts, in addition to the rules file, the configuration file for the calib map.

The validity date and the comment are written into $DFO_CONFIG_DIR/CALSELECTOR/config.createCalibMap.calsel. Hence there is a close connection between this tool and the tool createCalibMap.

Any CALSELECTOR directory already existing and containing a R2R rule can still be used. The new R2M rule is added to the directory. It is not planned currently to remove the R2R rules although they are not used anywhere else.

The tool creates the following output:

• <instr>_raw2master.${VAL_DATE}.RLS in $DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE}
• exported version of that file on http://www.eso.org/qc/ALL/OCA/CALSELECTOR/<instr>/${VAL_DATE}/

At that point task 1 is finished, and task 2 is needed to finish the whole operation.

Consistency checks and on-line help

Call

calselManager -I

and get information about the alignment (migration) process. Call

calselManager -M

to get information about the modification process.

Table: Use cases for rules management, workflow steps, and supporting tools

							<unclear!>	local →		← database
Use cases	copy DFOS_OPS to $TMP_DIR		copy CALSEL to $TMP_DIR		modify rule	analyze, verify, confirm, put to CALSELECTOR/<date>	OK? --> copy back to DFOS_OPS	calibMap create/update		CALSELECTOR DB: download/delete	upload
1. develop current OCA rule					( maybe)		(OCA unification)
2. Modify current OCA rule		OR			( maybe)		(OCA unification)
3. Add historical OCA rule		OR					!
4. Modify historical OCA rule							!
5. Make current CALSELECTOR OCA rule a historical OCA rule
supported by:	calselManager --		calselManager -B		[editing]	calselManager -C, -U verifyAB -f <file>	[copy]	createCalibMap -r <rule> [-A]		calselManager -m DELETE	calselManager -m WRITE

calselManager --: call without parameter (-r <rule> is optional)
Current OCA rule: the RLS file used in DFOS_OPS and current CALSELECTOR
Historical OCA rule: the RLS file applicable for CALSELECTOR period

Modify existing CALSELECTOR rule

For the modification of an existing CALSELECTOR rule (use case 2 in the above table), or the creation of a historical rule from the current rule (use case 3), use

calselManager -B

where the tool starts a dialogue for specification of the rule. You then edit the rule as appropriate, follow step 3 and 4 above.

Task 2: Rule management

The second major task of calselManager is the management of CALSELECTOR OCA rules in the database. Rules in this context are the OCA files <instr>_raw2master.${VAL_DATE}.RLS in $DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE}. There might be one (valid for the whole data history of the instrument), or several (to cover periods with significant changes of data types and/or associations).

Technically this task is provided by the rulesManager, a component of the installed calSelector tool suite.

Note that the rulesManager knows instruments by $DFO_FILE_NAME (max. 6 chars), i.e. by the label with which all ARCFILE names start.

The rules management is provided by the parameter -m[ode]. It can have values LIST, READ, CHECK, VALIDATE, WRITE, and REMOVE.

LIST:

calselManager -m LIST [-A]

The tool queries the database and returns all existing rules for your instrument with start_date, stop_date, and comment. If you add option -A[ll], all existing rules are listed.

CHECK:

calselManager -m CHECK

The tool checks the ISTs for the keys contained in the OCA RLS file. There are some trivial results which can be ignored: missing PRO.CATG in obs_metadata, and missing DPR keys in qc_metadata. All other claims about missing keys are significant and could either mean a typo in the OCA rules (like SEP.ARM instead of SEQ.ARM), or an issue with the database tables.

READ:

calselManager -m READ [-d <ref_date>]

The tool queries the database for the rule applicable to the given reference date (which by default is $TODAY), and downloads it to $DFO_CONFIG_DIR/CALSELECTOR/DOWNLOADED/${DFO_FILE_NAME}_calsel.RLS for further reference.

VALIDATE:

calselManager -m VALIDATE -R $RLS_FILE

where $RLS_FILE is the usual pathname of the CALSELECTOR rules file ($DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE}/<instr>_raw2master.${VAL_DATE}.RLS).

The tool calls the --validate-rules=true option for the specified local calselector rule. This can be useful to check for syntax and logical consistency checks. Although formally being part of the ingestion (WRITE) mode (see next), the tool is forced to abort after the validation.

WRITE:

calselManager -m WRITE -R $RLS_FILE

where $RLS_FILE is the usual pathname of the CALSELECTOR rules file ($DFO_CONFIG_DIR/CALSELECTOR/${VAL_DATE}/<instr>_raw2master.${VAL_DATE}.RLS).

The tool writes the specified RLS file to the database. It uses the $VAL_DATE as specified in the name as begin of validity. It starts a dialogue (self-explaining), asking you for confirmation that this file is of proper syntax (which only you know). It will also ask you about the stop-date (either 2999-12-31 or the start-date of another version). You have the option to enter a comment (one line) describing the rule or the version. This is optional and can be dropped (no need to enter trivial comments like "this is the MIDI rule set"). Then, the module RulesManager is called which first executes a syntax check (like the ones done in ABbuilder each time you call createAB). Once passed successfully, the tool asks for confirmation to upload the rules file. If the answer is 'y', the rules file is uploaded. During that process the tool also checks if there is an overlap in time with a pre-existing rules file.

REMOVE:

calselManager -m REMOVE

The tool starts a dialogue in which the currently existing rules are displayed (using the tool in mode LIST). Then the user is asked about start-date and stop-date (which seems redundant but that's the way it is). If the rules to remove is properly identified, the tool asks for confirmation and then removes the rule.

This step is necessary when you want to ingest a modified rules file:

• first use -m REMOVE
• then use -m WRITE.

How to use

Most options have already been discussed above and are repeated here for completeness.

General options:

Type calselManager -h on-line help about the tool, and -v for the version of the calselManager tool and of the calSelector tool.

Type

calselManager -H

for extended help about the components of the calSelector tool.

In most cases below you can add -D for DEBUG, in case you have an issue with the calSelector tool and need the DEBUG logs.

Task 1 options (if new CALSELECTOR rule):

Type

calselManager -I

to obtain information about the process to create the calSelector rules file (essentially the same as in the section "rule editing");

calselManager [ -r <path>/<your_instr>.RLS ]

to start the conversion (step 1) from your default DFOS_OPS OCA file (if none is specified; you then need to confirm the default), or from the specified file (which then needs a confirmation that that file is of DFOS_OPS syntax);

calselManager -C

to continue with the automatic part (always works on $TMP_DIR/<instr>_raw2raw.RLS which is the file that you have edited between step1 and this step);

calselManager -U

to finish the migration, by specifying a validity date and entering a comment describing e.g. why this version differs from the others, if any.

Task 1 options (if modification):

Type

calselManager -M

to obtain information about the process for modification of a calSelector rules file or the creation of a new one.

Type

calselManager -B

to enter a dialog to begin the modification workflow. Then call options -C and -U as before.

Task 2 options:

Type

calselManager -m LIST [-A]

calselManager -m CHECK

calselManager -m READ [-d <ref_date>]

calselManager -m VALIDATE -R $RLS_FILE

calselManager -m WRITE -R $RLS_FILE

calselManager -m REMOVE

See here for details.

Configuration files

The following config files are related to calselManager, all reside under $DFO_CONFIG_DIR/CALSELECTOR:

• CSConfiguration.properties: calselector configuration; used by the dfos tools calSelector,verifyAB and calselManager; described here.
• config.createCalibMap.calsel: contains validity date and comment for the CALSELECTOR rules file, managed by createCalibMap.

[config.calselManager has become config.verifyAB with calSelector v2.0.]

Operational hints

Operational hints for Task 1 (alignment).

Workflow.

The workflow to align the DFOS_OPS rules with the CALSELECTOR rules is organized in four steps (see "rule editing"). The fundamental workflow is matched by the steps 1-4:

1. start alignment
- call calselManager [optional: non-standard rules file -r <pathname>]
- output file is always $TMP_DIR/<instr>_raw2master.RLS1
- repeat as you wish

2. Move $TMP_DIR/<instr>_raw2master.RLS1 to $TMP_DIR/<instr>_raw2master.RLS
- check the pre-edited file $TMP_DIR/<instr>_raw2master.RLS

3. compile $TMP_DIR/<instr>_raw2master.RLS
- calselManager -C
- if necessary, fix $TMP_DIR/<instr>_raw2master.RLS and repeat this step

4. upload (finish)
- calselManager -U
- decide about version by entering validity date and comment
- tool will check if that date exists already; if yes: you can overwrite
- date and comment are also entered in $DFO_CONFIG_DIR/CALSELECTOR/config.createCalibMap.calsel which then knows about the versions
- once you have uploaded a new rules file, you have to return to step 3 (if you want to correct e.g. date or comment) or to earlier steps.

5. It is mandatory to create a calib map for the new rules set:

createCalibMap -r $DFO_CONFIG_DIR/CALSELECTOR/<val_date>/<rules file>

All calibration maps are collected under http://www.eso.org/observing/dfo/quality/ALL/OCA/oca_rule_sets.html for reference.

6. Note on Raw2Raw flag:

You can force the calSelector rules to go Raw2Raw, in the following way: add in the beginning of your rule

force all data to go Raw2Raw	if 1==1 then {   FORCE.RAW2RAW=T; }
force only INS.MODE=IMG data to go Raw2Raw	if INS.MODE=="IMG" then {   FORCE.RAW2RAW=T; }
force no data to go Raw2Raw (just use it as a placeholder)	if 1==2 then {   FORCE.RAW2RAW=T; }

As for all OCA rules within calSelector, it's important to use only real keywords, not "synthetic" keywords that are not yet defined (remember that OCA rules are read sequentially). Best is to have these four lines at the very begin of your OCA rule (if you want it at all).

The Raw2Raw flag could be useful if

• the current public pipeline and the archived master calibrations don't match
• there is public pipeline
• the masters are of very poor quality

In general, you should avoid delivering masters which the pipeline cannot make sense of. In all other cases (occasional quality issues, incomplete coverage) there is no need to control the Raw2Raw mode since this is done by the tool itself.

Tips and tricks

• Errors discovered by 'calselManager -m WRITE -R <RLS>': the calSelector component rulesManager performs a syntax check and discovers syntax errors like typos (Maxret instead of maxRet; unclosed or superfluous brackets; typical editing errors like 'and and' ).
  
  - fix the error in $TMP_DIR/<instr>_raw2master.RLS (or whatever the name of the RLS file is you have migrated and want to ingest).
  - call 'calselManager -C' then ' calselManager -U' then 'calselManager -m WRITE -R <RLS>'
  - repeat these step until no error occurs.
  
  
• createCalibMap can discover other kinds of logical errors, like missing (or inconsistent) time-match rules. Therefore, once a change to an RLS file has been applied and no further error is found, don't forget to call createCalibMap again.
  
  
• Timeout during ingestion of a rule: use the configuration parameter "database.timeout=<val>" with val e.g. set to 60 (CSConfiguration.properties). This error should be rare and the timeout should be reported to dbcm@eso.org.
  
  
• Checklist for migration:
  • all SCIENCE raw_types have a product? (important for definition of virtual products aka datasets)
  • all RASSOCs (if you have any): their definition contains only keywords, no macros? (otherwise expand them)
    
    
• Checklist for time match rules:
  • for all dynamic (master) calibrations: do they have a calib_plan time_match (the narrow one, defined by 'tryBetween'), and a wide one (the 'maybe still acceptable' range, defined by 'between', also important for performance)? The calibmap will tell you!
  • is the narrow one indeed narrower than the wide one? Remember: during conversion, the wider one is hard-coded to +/-10; there could be cases where you decide to go for 'tryBetween' to larger values, then choose appropriate values (maybe x factor 2?) for the 'between' range. The calibmap can't tell you!
  • for all static (generic) mcalibs: do they have a time_match (PREVIOUS or other)? Don't use tryBetween for them (there is no calibration plan for them).
  • You could use asymmetric time match rules (like '... between -10 and +5'; also: 'tryBetween -2 and +1').
  • You can even use the tryBetween and between syntax for non-timematch rules, it works fine for airmass, encoder positions etc. This is experimental!