Common DFOS tools:
Documentation
dfos = Data Flow Operations System, the common tool set for DFO
*make printable   see also:
    ABbuilder documentation
    OCA syntax
    createAB 
    AB concepts, AB structure

AB: organizing and rules

1. Standard events for organizing raw files

An Association Block (AB) generally contains a set of raw files and a set of calibration product files. There are certain rules for grouping, or organizing, raw files into an AB. For instance, the simplemost grouping rule is "one raw file - one AB", resulting in processing one file at a time. Others are "all raw files from a template", or "all raw files from the same chip" etc. All files grouped together are processed together.

For historical reasons, there is also the term 'event' since in the stream-oriented environment on Paranal, a group of raw files is defined by an event, namely a single file event or an end-of-template event.

For QC, events are replaced by grouping rules. These are defined within OCA in the <instr>_organisation.h configuration file. Some pre-defined grouping rules, which are quasi-standard across the instruments, can be refered to by macro name which is defined in OCA_macro.h. Other, not-so-standard rules can be defined by the user.

The grouping rules are defined by a name and listed in the AB as AB_EVENT. The AB name contains a short tag to identify the grouping rule. Although rarely used, this may become important in cases where several grouping rules exist and the ABs carry the same name root. As for all other rules, OCA evaluates FITS keys for the grouping rules. Mostly the TPL keys are used since there is a close logical connetcion between groups and templates.

Here is a list of the most commonly used grouping rules for raw files:

QUALIFIER TAG* Evaluated TPL keys GROUPING RULE DESCRIPTION EXAMPLES PRIMARY FILE (the one to give its name to products, logs, AB name etc.)
SINGLE <none> none single file any single file which has a DO.CLASS in <instr>_classification.h single file
TPL_A _tpl TPL.START all files per TPL_START; undefined DO.CLASSes are suppressed

homogeneous stack: e.g. all BIAS frames

 first file (EXPNO=1)
TPL_M _tpl TPL.START all files per TPL_START; multiple DO.CLASSes are supported; undefined DO.CLASSes are suppressed inhomogeneous stack: e.g. alternating OBJECT and SKY files in IR instruments first file (EXPNO=1)
inhomogeneous stack (with TESTs): initial TEST exposures in many SKYFLAT templates, e.g. FORSes, are suppressed (if not defined); AB carries the name of the first valid input raw frame first unsuppressed file (EXPNO .ne. 1 in general)
TPL_I _tpi TPL.START, plus other keys, e.g. INS multi-parameter template: group files from a template and with additional common properties daytime CALIB data coming with a single TPL.START first file per setup
TPL_B _tpb like TPL_A like TPL_A; if you want the same grouping rule applied for a second time (SEQ2) with a different recipe, use the option TPL_B like TPL_A; used by SINFONI first file (EXPNO=1)
TPL_D _tpl TPL.START, OCS.QUAD all files per TPL_START having the same detector ID homogeneous and inhomogeneous stacks supported; used e.g. by VIMOS to produce ABs per detector first file (EXPNO=1)
COMB_P _cmb none input files are products! This is a grouping rule to combine products (hence the name) from a previous grouping step, so far supported as SEQ2 step only

use case from VIMOS:
SEQ1: create SINGLE ABs for each found STD, to produce preliminary zeropoints
SEQ2: create COMB_P ABs for all products which would then produce the final zeropoint of the night

 first file of the SEQ1 ABs, with qualifier _cmb

*tag: the name of the created AB is formed by:

Examples:

GIRAF.2012-01-30T12:13:14.004.ab is the name of a SINGLE AB (one input raw file).

FORS2.2009-10-23T05:13:21.529_tpl.ab is the name of a stacked AB (a set of input files, all having the same TPL.START; applied rule: TPL_A).

VIMOS.2009-04-30T14:44:87.233_tpl.ab is the name of a detector-stacked AB (a set of input files, all having the same TPL.START and detector ID; applied rule: TPL_D).

VIMOS.2012-04-30T23:12:55.457_cmb.ab is the name of the combined STD AB of that night (applied rule: COMB_P).

2. Time match rules

The association rules select a set of calibration products having all the same PRO_CATG and FITS key values for the defined match key. Formally, any of these calibration products qualifies for association. To select from these match candidates, a time match rule is needed which is configured by the user in the <instr>_association.h file. The most popular ones are defined in OCA_macro.h:

NAME RULE TIME MATCH RULE DESCRIPTION EVALUATION
CLOSEST min (|mjd_diff|) mjd_diff is the difference between modified mjd_obs of the first raw file and mjd_obs of the calibration product; the modified mjd_obs is the mjd_obs read from the header, with EXPTIME/2 added (since mjd_obs refers to the begin of exposure, and we effectively want to use mjd at the middle of the exposure time). We choose the absolute value to have a symmetrical selection (earlier calibrations are in principal as good as later calibrations). The CLOSEST rule is the default choice. It usually guarantees to find the closest daytime calibrations. For a continuous stream of SCIENCE data from the night, it may result in an "association jump" in the sense that part of the data associate to the previous day, and part to the present.
NEXT like CLOSEST, with the additional requirement that mjd_diff must be positive like CLOSEST, but forward association is enforced: nighttime SCIENCE data are not allowed to associate to the previous daytime calibration This may be reasonable if you want to avoid an "association jump ". It may also make sense if you want to enforce that e.g. daytime FLATs and nighttime SCIENCE data find the same MASTER_BIAS. The caveat is you may not find any calibration for the latest downloaded day.
PREVIOUS like NEXT, but mjd_diff must be negative backward association is enforced: e.g. calib product must have been taken before science data  
ATTACHED try to find an attached (night-time) calibration, if none found, apply CLOSEST rule Attached calibrations are user-defined nighttime calibrations taken as part of the SCIENCE OB. The user generally expects them to be associated with the SCIENCE data. Attached calibrations typically are an option but not a requirement, hence it is always a valid situation that no attached calibrations are found, in which case the CLOSEST rule is applied.

This rule makes sense for those instrument modes and data types which have attached calibrations foreseen. They make sense when a time-critical calibration is required (e.g. to measure fringing, measure an instrument property related to the actual pointing etc.). But they may be of lower quality than daytime calibrations in other aspects like S/N.

Hence those calibrations which are found by application of the CLOSEST rule, should be added to the AB under the qualifier RASSOC (hence they are always packed along with the attached calibrations). This is done since in general observatory-provided calibrations are exposure-time controlled which is not always true for user-defined calibrations.
NIGHT matching calibrations are selected only from the same night as the SCIENCE data calibrations are only selected from the same night as the SCIENCE data (defined by the DFO date); minimum and maximum number to be found are defined by MINNUM and MAXNUM The typical application is to find all STD data taken in the same night as a SCIENCE observation. To find *all*, use a sufficiently high MAXNUM parameter. If these files are optional, use MINNUM0.
Last update: April 26, 2021 by rhanusch