Common DFOS tools:
Documentation

make printable	New:	See also:
	This documentation refers to calSelector v3.0. It documents the design, architecture and usage of calSelector from the QC perspective.	tools related to calSelector: calselManager | createCalibMap | verifyAB | writeBreakpoint general overview of calSelector here OCA: rules | syntax
topics: basics | rules and versions | testing | calSelector and dfos | syntax | calibration maps | virtual products | certified flag | validity | breakpoints

Introduction to calSelector

Find the design document available here.

Basics

calSelector is the DFS tool linked to the archive interfaces to deliver associated calibrations for user-selected science files. The tool is based on OCA rules which are provided and managed by the QC group. To support this task, and to continuously monitor the results, the calSelector tool is also installed on the QC machines. There are several tools in the DFOS tool suite related to calSelector. This page documents the high-level functionality of calSelector.

The tool is called for a single dp_id, or a set of dp_ids, and looks for associations in a database. The associations are defined by a set of OCA rules. These OCA rules come in Raw2Master (R2M) syntax, similar to the DFOS_OPS rules. The tool returns a result set in form of a XML file. It has the flag 'complete' if all associations can be satisfied.

calSelector v1 had the Raw2Raw functionality. With calSelector v2, the Raw2Master option was added. The current calSelector v3 has the breakpoints added.

By default the users receive master calibrations (mcalibs) only (plus those raw files configured as RASSOC), and only those that are necessary to execute the science recipe (the last step in the cascade).

If one or more required mcalibs cannot be found, the tool switches to the Raw2Raw (R2R) mode. It still applies the R2M rules, now using either the provenance information in those master calibrations that it can find, or the virtual products mechanism (see below). In R2R mode, all raw calibrations are delivered, including the "calibrations for calibrations", thus supporting the entire calibration and reduction cascade. In this mode, the tool returns certified raw files only (plus static calibrations). This certification flag is a feature related to the dpIngest tool: if a master calibration is ingested by QC, its provenance is checked (the list of parent raw files), and these raw files get the certified flag. See below for more information.

It is also possible for QC to configure an OCA rule for calSelector as "R2R_only", meaning that the tool is forced (by this database configuration) to go R2R (while in the above case it has decided by itself that R2M is not possible and therefore goes R2R). This is typically done if a public pipeline is incompatible with archived master calibrations (processed with an earlier pipeline version) and therefore a delivery of master calibrations would not make sense, but the certification flag provides a useful added value.

Static calibrations ("gencalibs") are always delivered as such. Associations marked as RASSOC always deliver raw files.

Rules and versions

The calSelector OCA rules come in R2M syntax, like the DFOS_OPS rules. Actually the DFOS_OPS and CALSELECTOR rules can be merged.

This is the schema for creating CALSELECTOR OCA rules:

• start with the current DFOS_OPS OCA rule; the workflow for the implementation of the CALSELECTOR rules is provided by the dfos tool calselManager;
• mark in the config file config.calselManager those RAW_TYPEs that are of no use for calSelector; they will be ignored in the calibration maps but still be seen by calSelector;
• check the auto-added (by calselManager) BETWEEN statements, remove duplications, edit special cases
• add products for science data (used to generate vproducts for them)

With these modifications, the rules become compatible for both DFOS_OPS and CALSELECTOR environments. The rules are then uploaded to web directories and database, and they are displayed graphically for better overview. These tasks are supported by dfos tools calselManager and createCalibMap.

For historical periods with OCA rules different from the CURRENT one, the corresponding CALSELECTOR version is developed from the CURRENT one.

For future periods with the need of a modification in DFOS_OPS, the CURRENT CALSELECTOR rule is made a historical one, and the new DFOS_OPS rule becomes the CURRENT CALSELECTOR rule.

These steps are supported by calselManager.

The alignment of DFOS_OPS and CALSELECTOR OCA rules is not enforced by any tool, it is a conceptual principle. The QC scientist has to make sure that this alignment is achieved and continued. In particular they need to take care that any change on one side is also reflected on the other side.

Testing

With a new OCA rule set, testing the associations is an important step towards the proper formulation of the rules. Remember they come in a complex syntax and cannot be interpreted intuitively. The dfos tool verifyAB is used to expose dp_ids to calSelector OCA rules which are either already ingested, or are local and under development. The user can define reference datasets for regression tests, or typical test cases, or filelists with many dp_ids for performance tests. Also, the tool is used to check routinely all new science datasets in the DFOS workflow for proper associations (in particular completeness). These continuous tests replace logically the historical harvesting. For their own purpose, QC not only creates but also stores all new science ABs for future reference. The currently only use case for those ABs would be the IDP production.

calSelector and dfos tools

The following dfos tools support the QC scientists in their tasks related to calSelector:

• calSelector itself: installed in the $DFO_BIN_DIR branch, used by calselManager and verifyAB
• creation of new OCA ruleset, modifications of existing OCA rulesets: calselManager and createCalibMap
• testing of OCA rules; routine checking of new science ABs in dfos workflow: verifyAB
• creation of break points in dfos workflow: writeBreakpoint

Syntax

calSelector uses standard and well-known OCA syntax. The only new feature is the 'between' statement which is also understood by ABbuilder. It is important to make sure that all science data types have a product defined since this is needed for the creation of vproducts. While this was in place originally, these product definitions were dropped in many cases with the termination of science processing by QC, and now need to be re-introduced again.

Calibration maps

The dfos tool createCalibMap supports the creation of calib maps. It displays only those raw types that are relevant for the science reduction. See the entire set of calibration maps here.

Virtual products

A core feature of calSelector are the virtual products ("vproducts"). They represent datasets, as defined in the grouping rules in OCA (organization). Datasets define the smallest reasonable unit for data processing (while the traditional single file is the smallest unit for archiving). A dataset could be: a single file (e.g. UVES), all files from the template (e.g. IMAGE stack), or a subset of those (e.g. all XSHOOTER OBJECT+SKY frames from the same template and the same arm). Vproducts can exist for both calibrations and science data.

Vproducts are used by calSelector for the following purposes:

• define a dataset for the input science file: you call calSelector with one dp_id and automatically get the full science dataset that is required for processing. The science vproducts are always evaluated by calSelector.
• define a dataset for the associated calibrations: that dataset is used only if no mcalib is found, otherwise it is ignored. In that sense the calibration vproducts are similar to the virtual calibrations in DFOS, which are also replaced by real mcalibs if existing.

Vproducts are created in several ways:

• New data: there is a cronjob on the archive side that scans the archive input data stream and creates new vproducts.
• Based on mcalibs: For all mcalibs in the archive, vproducts have already been created, to be used for delivery of raw file datasets (on users demand, or if R2M fails). Via dpIngest, vproducts are created for newly ingested mcalibs.
• Since new vproducts (from newly archived science raw data) are defined by the OCA rules, there is a vproduct creation (and deletion) process triggered upon OCA rules ingestion (and deletion). Since this can be time-consuming, and is not always required, the calselManager user can control this option.

The tool for the creation of vproducts is called the qcproducthandler. It is a component of the calSelector jar package. It is not used by QC but is running on the archive side in the background. For information, here is its workflow:

• for each instrument, get all templates between the input dates (which are defined for the various instances of the tool, see above);
• for each template, run the OCA rules and generate the vproducts;
• for each product: if there is a master calib with the same pro.catg, same input files and roughly the same mjd-obs (+/-0.0001), don't do anything;
• else if there is a vproduct with the same name, replace it;
• else save the vproduct.

Certified flag

There is an automatic process in the archive that marks all raw files as 'certified' if they have been used to generate an mcalib that got archived. Raw files without flag are of unknown (not: bad!) quality. They can exist in the very recent data flow, or in old data with no pipeline support, or no good pipeline support (one recipe is known to fail), or no QC support (SM data only, or standard setups only).

calSelector gives preference to certified raw data if it comes to raw data delivery (see above).

The certified flag is set by dpIngest upon ingestion of a new mcalib, and unset if an mcalib is hidden (dpDelete).

The proper mechanism to flag a raw file as having bad quality is to hide it. Then it will always be ignored by calSelector.

The certification flag reflects an important value added by the QC group.

Validity

The OCA rules as maintained by QC do not contain validity information in OCA syntax. For DFOS_OPS and for calChecker, validity is supported in a non-OCA syntax (the commented DELTAT_RULE section) and is evaluated by createAB. That validity concept has three values: OK, NOK, MISS. NOK means a matching calibration exists in the data pool but it is outdated. MISS means that no calibration has been found.

calSelector also supports such a scheme. It is implemented by the use of two between-like statements:

• between <val1> and <val2> is the classical 'between' statement which returns any matches within the two values <val1> and <val2>; for calSelector, it is normally used as a relatively large interval, returning matches which could still be acceptable if one checks carefully; also, the interval serves as a first-guess interval for calSelector to avoid scanning the whole database for the final match;
• tryBetween <val3> and <val4> is a statement whose intervals are based on the calibration plan and are usually tighter than val1/val2. For all matches found, the closest-in-time is chosen.

Usually both statements are used, and calSelector then interprets the matches in the following way:

• if tryBetween returns a match, that match gets the flag "match=calib_plan" in the result xml file, meaning this is the best-possible association;
• if tryBetween does not return a match but between does, that match gets the flag "match=extended" indicating that this match might be acceptable but the user should apply some care.
• if no tryBetween statement is given, the match value in the xml file is "N/A".

In addition there are the classical time match rules for static mcalibs (the PREVIOUS rule in most cases).

Breakpoints

A special case for validities is covered by the concept of breakpoints where associations are not allowed to cross certain well-defined MJD-OBS values. The breakpoints are maintained by QC in a database table calsel_breakpoints (here) and are maintained with the dfos tool writeBreakpoint. They are implemented with calSelector v3.