Workflow information system for QC

AB processing

Association Blocks (ABs) are the fundamental data processing unit at QC. They contain all information necessary to process a raw file, or a set of raw files: recipe name, data type, associated calibration products, processing parameters, etc. They also contain additional information relevant during the lifecycle of an AB, like pathnames of products, execution time, setup information, OB ID.

AB handling has two main aspects. One ist the technical aspect, the other one is the human aspect. The technical aspect is related to the requirements on the compute platform: the number of processing jobs, and the AB execution time. The other aspect is monitored here.

The AB metrics contain both CALIB and SCIENCE ABs. They were always created in both modes, but the processing scenario changed over the years (see graphics below). While the CALIB ABs were always fully processed, the SCIENCE ABs were processed in SM only until June 2009, then - until 2011-09 - all SCIENCE ABs were processed, from 2011-10 on all SCIENCE ABs were created and stored but not processed anymore, while since 2020-10 no SCIENCE ABs are created anymore.

An AB does not take into account the size of the files. For metrics related to hardware performance, the size is important though. Therefore the AB number used here refers to ABs per detector (called det.ABs for short in the plots). For instruments with multiple detectors, that number is higher by a factor MEF_FACTOR than the actual number of ABs as counted in the QC system (MEF_FACTOR=4 for CRIRES and HAWKI; 16 for VIRCAM; 32 for OCAM; 3 for KMOS).

The processing time depends on the hardware used. Numbers from different instruments are comparable at any given time and can be useful to identify CPU bottlenecks etc.

Note that each AB is measured only once. Multiple processing of the same AB is not measured, the execution time always refers to the last execution.

Execution times are effective execution times per AB. They include CPU time, disk I/O etc. Between 2008 and 2013, they do include file download times.

Pipeline processing of ABs takes only a certain fraction of the overall process time required for full AB handling. In addition to pipeline processing, there is preparation (data downloads, reporting); AB creation; QC report creation and scoring; post-processing handling like renaming and distribution. Execution times of these subprocesses are not measured (with the exception of QC report creation, see below) and could exceed pipeline processing times. It is nevertheless true that execution times for all other processes roughly scale with the number of ABs, and hence with execution times. This is the real value of these two numbers.

Since the end of 2009, also the QC report creation times are measured and plotted as a third plot.

The plots come per instrument. There is also 'ALL' collecting sums for all instruments. Note that most metrics have been collected since 2004 or later. Comparison numbers from earlier epochs are not available in the database.

You can do more detailed research, e.g. per instrument mode, using the QC1 interfaces under "advanced studies".