AB = association block, the QC equivalent of the reduction block, with additional association information. The AB is the fundamental pipeline processing job.

The AB monitor lists all created ABs and their processing status. If available it also provides historical quality information (scores, certification comments). It links to association and processing logs. It is organized around dates. For the OPSHUB installation, a date is defined by the selected SCIENCE data, and all associated CALIB data are listed, no matter if from the same date or not. If no SCIENCE data are selected for the specified date, then the date applies to all CALIB data.

Apart from the instrument, the AB monitor is always specific to a PROJECT. Depending on the PROJECT definition, the AB monitor for a given date might neither be complete nor might it contain only ABs formally belonging to that date.

Each AB monitor page has two parts: the header and the AB product report.

Header part:

collects project overview information.

OPSHUB processing monitor (project: KMOS_PROJECT_WAVEHC) (instrument: KMOS, date: 2018-07-03) This is the overview of all current processing jobs (ABs) of project KMOS_PROJECT_WAVEHC [...]

last update: 2019-10-19 15:27:04 (UT); machine: muc12; 21 ABs in $DFO_AB_DIR   browser_refresh: on (every 60 sec); tool_refresh: on (every 60 sec + runtime)

number of ABs (all | success | failed | created): 21 | 15 | 0 | 6     pipeline version:   kmos-1.4.3 some statistics and the configured pipeline version KMOS quick links (book-keeping): AB | JOBS | DFS_LOG | MONITOR | CONFIG Data directories for KMOS: RAW | CAL | PRODUCTS Local OCA rules for KMOS: CREATE_CASCADE | CALSELECTOR these links are to directories related to this instrument PROJECT log file: KMOS_PROJECT_WAVE Project definition for KMOS_PROJECT_WAVE: these links are related to this PROJECT: the log file PROJECT_NAME INSTRUMENT MODE DATE SELECT AB_METHOD CALSEL_MODE OCA_RLS_METHOD KMOS_PROJECT_WAVE KMOS CALIB 2018-07-04 ANY CREATE_CASCADE NONE DFOS_OPS the project definition (taken from $DFO_CONFIG_DIR/projects.distillery), links to all DATEs for this PROJECT, and to the HTML page with the configured OCA rule (if any)

Navigation links:

During runtime of a job, the local AB monitor displays additional information if running CONDOR:

AB product report:

The AB product report is the overview of all ABs defined for the given PROJECT and DATE. The table is sortable and can be filtered. The column headers have tooltipps available. The AB processing status, and the AB mode, are colour-coded. For the OPSHUB, ABs can come in mixed modes (CALIB and SCIENCE).

These are the possible entries for CALIB ABs (most of them sortable):

BQS

AB NAME

INDEX

COMPL.

AB LOG

RECIPE

RAW_TYPE

SETUP

AB STATUS

P LOG

T_EXEC

PRODUCTS

SAME DATE?

HISTORICAL SCORE

HISTORICAL CERTIFICATION

created but not yet executed

KMOS.2018-03-24T11:54:06.688.ab

CAL001

compl.

OK

wavecalibration

WAVE

Medusa2_L543.1

...

...

pipeline processed

KMOS.2018-04-29T23:18:39.329_tpl.ab

CAL031

compl.

-37.4

kmos_rec_jitter

PSF

S3_K_0.1

OK

P_LOG

5.7

...

(0/1)

OK

failed

KMOS.2018-03-24T11:51:48.057_tpl.ab

CAL131

compl.

OK

gimasterflat

FFLAT

Medusa2_L543.1

NOK

P_LOG

failed, no flux

For SCIENCE ABs:

BQS

AB NAME

INDEX

COMPL.

AB LOG

RECIPE

RAW_TYPE

SETUP

AB STATUS

P LOG

T_EXEC

PRODUCTS

SAME DATE?

HISTORICAL SCORE

HISTORICAL CERTIFICATION

created but not yet executed

GRAVI.2018-01-11T01:59:13.497.ab

SCI001

compl.

OK X H T

gravity_vis

SCI_SINGLE

MEDIUM_SPLIT_SPLIT_A0

...

pipeline processed

GRAVI.2018-01-11T01:51:43.188_tpl.ab

SCI015

compl.

VIRT X T

gravity_vis

SCI_SINGLE

MEDIUM_SPLIT_SPLIT_A0

OK

PLOG

6.3

products

failed

GRAVI.2018-01-12T02:22:07.265_tpl.ab

SCI320

compl.

VIRT X T

gravity_vis

SCI_SINGLE

MEDIUM_SPLIT_SPLIT_A0

NOK

PLOG

BQS (batch queue system): displays at execution time which ABs are currently processing. It is meaningful for CONDOR batches only. Blocked ABs (the ones which have to wait for others) are marked red, currently executing ones are yellow.

AB NAME: name of processing job. ABs which are contained in other ABs in $DFO_AB_DIR are marked by a little blue square. If these fail, the ones depending on those will also fail. The AB names always are derived from the first input raw file. They might contain suffixes like 'tpl' which indicate that the AB contains many raw files (e.g. all from the same template). No suffix means the AB has one raw file which then gives its name to the AB. Extension of ABs is always '.ab'.

INDEX: starting with CAL or SCI, indicates the AB index within the entire set of ABs for this PROJECT and this date.

COMPL: displays the completeness status of the AB ('compl.' or 'incompl.'). A complete AB contains all calibration files rquired for execution. An incomplete AB has some configured calibration files missing, but - depending on the pipeline recipe - may still be executable.

AB LOG: link to the association log file. It lists all associated calibration files, with time difference information (in fractions of a day). If a time difference warning is found in the association log, it is displayed in red. If a science AB contains associations with VIRTUAL flag (not existing at the time of AB creation), this is indicated in red. The X and T links exist for SCIENCE ABs only, and only if CalSelector has been used for creating the ABs. They link to the corresponding CalSelector xml and txt files. They are colour-coded: green if all ok, red if warnings have been found (i.e. incomplete, outside time window, not certified). Hover your mouse in order to see more information, and click the link to see all details.

RECIPE: name of the associated pipeline recipe.

RAW_TYPE: raw data type, single tag that is built out of the three DPR keys for RAW data and specifies a unique (per instrument) data type in terms of association and processing properties. RAW_TYPEs are linked to the calibration map URL which displays the association cascade and the relations between the data types. Often this map is specific for a given instrument mode.

SETUP: combination of setup keys used to distinguish data sets within a RAW_TYPE.

AB STATUS: created but not yet processed: ... Successfully executed: OK Failed: NOK

PLOG: link to the pipeline processing log.

T_EXEC: recipe execution time in minutes.

PRODUCTS: a link to the product directory, displaying the entire content.

SAME DATE: for DATEs with CALIB ABs created from CALSELECTOR content, the calibration ABs might refer to different dates than the SCIENCE ABs. Then, the CALIB ABs have an entry 'earlier' or 'later'. This cell is empty if the CALIB date is the same as the SCIENCE date.

HISTORICAL SCORE: for CALIB ABs, there are historical scores available from the QC processing. They are displayed here, to provide potentially useful quality information.

HISTORICAL CERTIFICATION: for CALIB ABs, there is historical certification information available from the QC processing. Possible flags are "OK", "AUTO" (for automatic certification), and "REJ" (for rejection). Comments are also displayed if available.