AB = association block, the QC equivalent of the reduction block, with additional association information. The AB is the fundamental pipeline processing job.
| topics: AB monitor | QC reports | certification process |
The AB monitor lists all created ABs, their processing status and the quality of their products. It links to the AB content, to processing logs and to detailed QC and scoring information. It is intended to give a complete overview about product quality. These pages are linked to the calChecker and to the Health Check monitor pages and provide detailed "information on demand".
There is one such page for each specified date and instrument. It has two parts: the header and the AB product report.
Header part: collects possible date comments, and statistical information.
| AB product monitor (instrument: GIRAFFE, date:
2011-10-16) This is the product quality monitor, with an overview of the processing status of all processing jobs (ABs) and the quality of the products. [...] For data later than 2011-10-01 only calibrations are processed, no science data. |
|
| monthly DETMON data taken | |
last update: 2011-10-19 15:27:04 (UT); machine: dfo23;
 21
ABs browser_refresh: on (every
300 sec); tool_refresh: on (every 300 sec +
runtime) |
|
|
The date comment field will have an entry like
if there are data in pipeline-unsupported modes. This is to indicate to the Paranal daytime astronomer that such data are not quality-checked by the QC system and need visual inspection on-site.
Links:
| CAL | report | RAW | NLT | refs |
| calChecker results | data report | raw display pages | link to nightlog tool | reference frames |
AB report: The AB product report has tooltips available for each header field. The page comes in two views. You can select sorting by AB NAME (default) or by SETUP.
These are the possible entries for CALIB data on the AB product monitor:
| BQS | AB NAME | COMPL. | AB LOG | RECIPE | RAW_TYPE | SETUP | AB STATUS | P LOG | T_EXEC [min] |
QC CHECK | SCORE | CERTIF | ||
| just created but not yet executed |  GIRAF.2009-03-24T11:54:06.688.ab |
compl. | OK | giwavecalibration | WAVE | Medusa2_L543.1 | ... | ... | ||||||
| pipeline processed but QC report not yet created | SINFO.2009-04-29T23:18:39.329_tpl.ab |
compl. | -37.4 | sinfo_rec_jitter | PSF | S3_K_0.1 | OK | P_LOG! | 5.7 + 4.3 | ... |  |
|||
| pipeline, QC report, scores done |  GIRAF.2009-03-17T21:15:38.110.ab |
compl. | OK | giwavecalibration | WAVE | Argus_H447.1A | OK | P_LOG! | 1.3 +3.9 | DONE |  HC |
(0/4) | OK | |
| unsuccessful execution | GIRAF.2009-03-24T11:51:48.057_tpl.ab |
compl. | OK | gimasterflat | FFLAT | Medusa2_L543.1 | NOK | P_LOG! | failed, no flux | |||||
|
SCIENCE ABs have only one status (created): |
||||||||||||||
| BQS | AB NAME | COMPL. | AB LOG | RECIPE | RAW_TYPE | SETUP | AB STATUS | P LOG | T_EXEC [min] |
QC CHECK | SCORE | CERTIF | ||
| created | GIRAF.2011-10-11T01:59:13.497.ab |
compl. | OK X H T | fors_img_science | SCI_IMG | 2068_2048_COLL_SR_z_GUNN | no proc. | |||||||
BQS (batch queue system) is a field that displays which ABs (or parts thereof, in case of multi-extension files) are currently processing. This field is only useful within the QC environment and can be ignored on the exported version.
AB NAME name of processing job (association block). You can sort by SETUP or by AB NAME. Those ABs which are contained in other ABs in $DFO_AB_DIR are marked by a little blue square. With this marking it is easier to see if the failure, or rejection, of an AB might have consequences for other ABs.
COMPL displays the completeness status of the AB ('compl.' or 'incompl.'). A complete AB contains all associated calibration files as configured. An incomplete AB has some calibration files missing, but - depending on the pipeline recipe - may still be executable.
AB LOG links to the association log under $DFO_AB_DIR. That log lists in a short form all associated calibration files, along with time difference information (in fractions of a day). If a time difference (DELTA_T) warning is found in the log, that leap is displayed in red. If multiple warnings are found, only the absolute largest one is displayed. This is the way how QC implements and monitors the concept of validity of calibrations. The X H T links point to outputs of the Associations Harvester.
RECIPE lists the name of the pipeline recipe associated to the data file under RAW_TYPE.
RAW_TYPE lists the raw data type (which is defined by the QC scientist).
SETUP is the combination of setup keys used to distinguish different data sets. You can sort by SETUP or by AB NAME.
AB STATUS: created but not yet processed (or processing): ... Successfully executed: OK Failed: NOK
P LOG links to the processing (pipeline) log. A '!' marks a WARNING entry (which is visible on mouseover).
T_EXEC is the summed recipe and QC report execution time in minutes.
QC CHECK marks the status of the QC report. Not yet executed: ... Executed: DONE and linked to the (first) graphical QC report if existing
SCORE displays scoring information, provided by the QC reports.
Not yet executed: ... Executed: a check box ( =
OK; 
=
NOK but no more than 1 outlier; 
=
NOK, more than 1 outlier). Scores are linked to the score result file (<ab_name>.html). If
an AB has at least one score marked as HC in
the score result page, it gets a similar mark HC on
the AB products page.
CERTIF is filled after the certification process by QC. It is set to OK if the AB has been certified, AUTO if auto-certified (based on scoring), or REJ if rejected. A comment by the QC scientist on the product might show up in the same column.
These reports are customized for each instrument and data type (RAW_TYPE). They combine QC1 parameters and static graphs. They are used in the certification process and stored permanently. Since 2008 they are also archived.
![[ top ]](http://www.eso.org/images/arr1red-up.gif "[top]"){kind=small}
Notes about the QC certification process
In principle certification by the QC scientist is an interactive process. As a compromise between efficiency (mass production) and accuracy, QC is using a semi-automatic scheme. All ABs are reviewed, but not all are inspected. Inspection involves an active decision of the reviewing QC scientist (OK = certified, or REJ = rejected). Reviewing means an automatic script checks for the score results and lets the AB pass if certain conditions are met.
The following ABs are always inspected:
- unsuccessful ABs (pipeline failed)
- ABs without scores (scoring not yet implemented)
- ABs with yellow or red scores
Certain RAW_TYPEs can be auto-certified, still at least one (or more) ABs per such RAW_TYPE are selected at random for active inspection. Typically, those RAW_TYPEs are configured for auto-certification which come in larger numbers and similar properties so that a detailed review of each single of them is not required.
Active inspection is assisted by visual checks, plus QC report inspection, plus trending information, plus scores.
Depending on the inspection level, the certification level is
- OK or REJ if based on active inspection
- AUTO if based on automatic certification.