AB = association block, the QC equivalent of the reduction block, with additional association information. The AB is the fundamental pipeline processing job.
The AB monitor lists all created ABs, their processing status and the quality of their products. It links to the AB content, to processing logs and to detailed QC and scoring information. It is intended to give a complete overview about product quality. These pages are linked to the calChecker and to the Health Check monitor pages and provide detailed "information on demand".
There is one such page for each specified date and instrument. It has two parts: the header and the AB product report.
Header part: collects possible date comments, and statistical information.
| AB product monitor (instrument: GIRAFFE, date:
2011-10-16) This is the product quality monitor, with an overview of the processing status of all processing jobs (ABs) and the quality of the products. [...] For data later than 2011-10-01 only calibrations are processed, no science data. |
|
| monthly DETMON data taken | |
last update: 2011-10-19 15:27:04 (UT); machine: dfo23;
 21
ABs in $DFO_AB_DIR browser_refresh: on (every
300 sec); tool_refresh: on (every 300 sec +
runtime) |
|
|
The date comment field will have an entry like
if there are data in pipeline-unsupported modes. This is to indicate to the Paranal daytime astronomer that such data are not quality-checked by the QC system and need visual inspection on-site.
Links:
| CAL | report | RAW | NLT | refs | [enter date comment] | [recreate ABs] | raw |
| calChecker results | data report | raw display pages | link to nightlog tool | reference frames | option to enter a date comment | option to call recreateABs | link to $DFO_RAW_DIR |
During runtime of a job, the local AB monitor displays additional information:
|
Condor jobs for giraffe@dfo23: |
dagman for giraffe@dfo23: 10/31 17:32:46 Node GIRAF.2011-10-31T15:00:15.542.ab job proc (1180.0) completed successfully. Active jobs: GIRAF.2011-10-31T15:14:02.166_tpl.ab |
AB report: The AB product report has tooltips available for each header field. The page comes in two views. You can select sorting by AB NAME (default) or by SETUP.
These are the possible entries for CALIB data on the AB product monitor:
| BQS | AB NAME | COMPL. | AB LOG | RECIPE | RAW_TYPE | SETUP | AB STATUS | P LOG | T_EXEC [min] |
QC CHECK | SCORE | CERTIF | ||
| just created but not yet executed |  GIRAF.2009-03-24T11:54:06.688.ab |
compl. | OK | giwavecalibration | WAVE | Medusa2_L543.1 | ... | ... | ||||||
| pipeline processed but QC report not yet created | SINFO.2009-04-29T23:18:39.329_tpl.ab |
compl. | -37.4 | sinfo_rec_jitter | PSF | S3_K_0.1 | OK | P_LOG! | 5.7 + 4.3 | ... |  |
|||
| pipeline, QC report, scores done |  GIRAF.2009-03-17T21:15:38.110.ab |
compl. | OK | giwavecalibration | WAVE | Argus_H447.1A | OK | P_LOG! | 1.3 +3.9 | DONE |  HC |
(0/4) | OK | |
| unsuccessful execution | GIRAF.2009-03-24T11:51:48.057_tpl.ab |
compl. | OK | gimasterflat | FFLAT | Medusa2_L543.1 | NOK | P_LOG! | failed, no flux | |||||
|
SCIENCE ABs have only one status (created): |
||||||||||||||
| BQS | AB NAME | COMPL. | AB LOG | RECIPE | RAW_TYPE | SETUP | AB STATUS | P LOG | T_EXEC [min] |
QC CHECK | SCORE | CERTIF | ||
| created | GIRAF.2011-10-11T01:59:13.497.ab |
compl. | OK X H T | fors_img_science | SCI_IMG | 2068_2048_COLL_SR_z_GUNN | no proc. | |||||||
BQS (batch queue system) is a field that displays which ABs (or parts thereof, in case of multi-extension files) are currently processing. This field is only useful within the QC environment and can be ignored on the exported version. See below for possible cases.
AB NAME name of processing job (association block). You can sort by SETUP or by AB NAME. Those ABs which are contained in other ABs in $DFO_AB_DIR are marked by a little blue square. With this marking it is easier to see if the failure, or rejection, of an AB might have consequences for other ABs.
COMPL displays the completeness status of the AB ('compl.' or 'incompl.'). A complete AB contains all associated calibration files as configured. An incomplete AB has some calibration files missing, but - depending on the pipeline recipe - may still be executable.
AB LOG links to the association log under $DFO_AB_DIR. That log lists in a short form all associated calibration files, along with time difference information (in fractions of a day). If a time difference (DELTA_T) warning is found in the log, that leap is displayed in red. If multiple warnings are found, only the absolute largest one is displayed. This is the way how QC implements and monitors the concept of validity of calibrations. The X H T links point to outputs of the Associations Harvester.
RECIPE lists the name of the pipeline recipe associated to the data file under RAW_TYPE.
RAW_TYPE lists the raw data type (which is defined by the QC scientist).
SETUP is the combination of setup keys used to distinguish different data sets. You can sort by SETUP or by AB NAME.
AB STATUS: created but not yet processed (or processing): ... Successfully executed: OK Failed: NOK
P LOG links to the processing (pipeline) log. A '!' marks a WARNING entry (which is visible on mouseover).
T_EXEC is the summed recipe and QC report execution time in minutes.
QC CHECK marks the status of the QC report. Not yet executed: ... Executed: DONE and linked to the (first) graphical QC report if existing
SCORE displays scoring information, provided by the QC reports.
Not yet executed: ... Executed: a check box ( =
OK; 
=
NOK but no more than 1 outlier; 
=
NOK, more than 1 outlier). Scores are linked to the score result file (<ab_name>.html).
If an AB has at least one score marked as HC in
the score result page, it gets a similar mark HC on
the AB products page.
CERTIF is filled after the certification process by QC. It is set to OK if the AB has been certified, AUTO if auto-certified (based on scoring), or REJ if rejected. A comment by the QC scientist on the product might show up in the same column.
Local version only:
For CONDOR, there is an indication if the system is UP, and the current number of active ABs is listed underneath 'load'. There is a link to the CONDOR logs ("last DAG directory"). Error files are linked. The last status messages from the 'dagman' (the queue monitor) are displayed if the queue is active (under processing).
BQS status:
| BQS status (CONDOR only) available only during processing |
|
| blocked (waiting for another AB to finish) | |
| ready for execution, or executing | |
| undefined, or processed | |
| AB status | |
| ... | created |
| OK | successfully executed |
| NOK | failed |
| QC check | |
| ... | pending |
| DONE | done, QC report linked |