make printable	v4.1.3: - adapted to COMPLETE_MODE=YES for PHOENIX; links to cascadeMonitor dropped	see also: - link to HELP page - link to RAW_TYPEs as output of createCalibMap - mucMonitor - RAW screenshots
		databases none dfos tools called by processAB and scoreQC and within the JOBS created by createJobs output monitor page: $DFO_MON_DIR/status_<date>.html, linked to dfoMonitor; after finishing: status page linked to histoMonitor or (for PHOENIX) to phoenixMonitor upload/download export to qcweb: status pages export to w1/2: score_day files (to /qc/<ins>/common) javascript javascripts jquery-latest.js, jquerydataTables.js under ${JSCRIPT_URL}
topics: description | modes | global info | condor section | recursive mode | history mode | Features: pgi dependencies scoresQC reports CalSelector [more...] | Navigation | RAWDISP plots | sorting and filtering | workflows and distribution scheme | usage | configuration

getStatusAB (DFOS and PHOENIX environments)

This is the tool for the AB monitor. For easier overview, its documentation is separated into the pages for the DFOS and PHOENIX environment (this page) and for the OPSHUB environment. This page deals only with the DFOS and PHOENIX environments.

enabled for parallel execution

Note: the tool has restricted display options in PHOENIX environments.

Description

This tool is a DFOS tool with interactive GUI capabilities. It provides a graphical overview of the ABs, their content and their status ('AB monitor'). It is called by the workflow tools createJob, processAB and processQC, as well as by the scoring tool scoreQC (in incremental mode). Its HTML output page is linked to the dfoMonitor. It is also embedded in the histoMonitor/phoenixMonitor where it is executed for the whole set of (historical) ABs. That mode is also available as -H on the command line. The HTML output pages (in edited form) are exported to the DFO_WEB_SERVER qcweb.

The tool has two main modes:

• DATE: all ABs per DATE are scanned
• AB: only one specific AB is scanned (incremental update)

Mode DATE. In mode 'DATE', the tool scans all ABs for that DATE under $DFO_AB_DIR. It reads the process status, the recipe called, the RAW_TYPE, and the match key content. It also checks for available products, association logs, process logs, QC products and scoring results. From the association log, it reads DELTA_T warnings. If found, the absolute largest one is displayed, as indication of a possible problem.

The tool is PHOENIX-aware. As a special case, it supports the DEEP mode of PHOENIX where pseudo-dates are used, to support the cross-OB (and thereby cross-date) organized deep ABs. While the logical organisation of these ABs is by run_ID, that run_ID is encoded as pseudo-date in order to use the tool without major changes. For instance, the run_ID 095.A-0384(A) is encoded as DATE 1095-03-84 where the first digit encodes the last character (actually the run identifier). 1 stands for A, 2 for B etc. The other components are straightforward. The first character A in the run_ID has no particular meaning and is dropped. Run_IDs are unique per period, and so are the pseudo-dates. The PHOENIX mode in general, and the DEEP mode in particular, is recognized automatically from the system configuration (.dfosrc). There is no special call for it.

For DRS_TYPE = CON, it displays the queue status.

Mode AB. In mode 'AB', the same is done but just for one specified AB. This incremental update of a pre-existing table is provided for performance reasons. If called from processAB, it only makes sense for sequential execution (CPL). It should not be chosen for CONDOR since then the blocking status cannot be scanned properly. The tool scoreQC always uses this incremental update mode.

A special option -e can be used together with mode 'AB': then only the content of the last column ('CERTIF') is updated, along with any product comment if found. This is (only) used within certifyProducts in AUTO mode: at this step all other content in the AB monitor is not modified, just the CERTIF flag is entered (AUTO in this case). This takes about 0.4 sec per AB, while the full edit of an AB row in the AB monitor takes about 1 sec. This gain becomes relevant for large amounts of ABs to be auto-certified.

Global information. In addition to the AB specific information, there is also some global information about the number of all/successful/failed ABs and the global scoring result:

Condor section

Condor jobs for giraffe@muc02: 2013-02-15 09:44:17 0 jobs; 0 completed, 0 removed, 0 idle, 0 running, 0 held, 0 suspended Last cascade: /data23/giraffe/condor/CALIB_2013-02-13-1360863054.70915699

Dagman for giraffe@muc02: No job running.

mucMonitor: open not running launch executing by placeholder

The condor section has, in the first field, an overview of the current condor jobs for $DFO_INSTRUMENT, a link to the last executed cascade, and links to (non-zero length) error files in the job directory.

The second field has an overview of the currently active condor jobs for $DFO_INSTRUMENT. Fields #3 and #4 display a toolbox for the mucMonitor. This is offered for DFOS and PHOENIX only.

For DRS_TYPE=CPL and INT the condor table and the dagman table are disabled.

The mucMonitor box checks if that tool is currently running on any of the operational accounts of the $HOSTNAME. Use the 'open' link to open a browser window reserved for the mucMonitor. The possible states for the toolbox are:

mucMonitor: open not running launch executing by placeholder	mucMonitor: open executing by giraffe stop executing by placeholder	mucMonitor: open executing by xshooter   executing by placeholder
mucMonitor not running, option to launch is offered	mucMonitor running on own account --> option to stop is offered	mucMonitor running on other account --> no option to stop

Recursive mode. With CONDOR, the tool executes in recursive mode (option -r). It is then looping in the background (running every 60 seconds).

HISTORY mode. With option -H, the tool scans all ABs under $DFO_LOG_DIR/<date> instead of the $DFO_AB_DIR. This is useful at or after the end of processing for a specific date. This mode is used by histoMonitor. It is also available on the command line and could be useful e.g. for fixing the history of dates where changes were required.

Export links to qcweb for incremental processing. With option -F, all CALIB ABs and their associated information are exported immediately to the logs directories on qcweb. The purpose of this is to provide the same look and feel, and the same depth of information, to the external user (Paranal daytime astronomer) for both already certified and not yet certified dates. All information linked to the AB monitor is exported: ABs, association logs, processing logs, QC reports, score reports. This transfer to the final logs directory is done only on qcweb, not on the operational machine. This mode is potentially dangerous since it gives up the principle of clearly separating uncertified (preliminary) and certified (final) information. To prevent damage by unintentionally creating or executing a set of ABs after they have already been certified, this option is not offered on the command line but only within the more controlled environment of autoDaily. (Technically this is implemented through the environment variable CALL_ENABLED which needs to be set to YES by autoDaily and createJob.)

Features: pgi | dependencies | scores | HC marking | marking of calSelector issue for science ABs | tablesorter | calibmap links | QC reports | date comment | unsupported modes | browser refresh

The AB monitor has a bunch of features which are described in the following.

PGI_GETSTATUSAB. If defined in the configuration file (and provided under $DFO_BIN_DIR), this pgi script is offered for action on top of the HTML page. It can be used for special purposes. For instance, for the PHOENIX projects it can be used for some basic certification.

Dependency marking. Those ABs which are contained in other ABs in $DFO_AB_DIR are marked by a little blue square:

GIRAF.2010-08-27T13:32:41.034_tpl.ab

CAL11

compl.

OK

gimasterflat

FFLAT

Medusa2_H665.0

OK

P_LOG!

2.8+0.7

DONE

HC

(0/6)

OK

With this marking it is easier to see if the failure, or rejection, of an AB might have consequences for other ABs. For more advanced dependency search, see below.

Links to calib maps. The RAW_TYPE is linked to the DFOS_OPS calib map, either to the specific instrument mode if it is configured there (in config.createCalibMap) and is unique, or to the ALL overview page. These links are thought to be useful for better understanding of the calibration cascades.

Scores. There is the column 'SCORE' which displays the scoring result as provided by scoreQC. This comes as a pair of numbers (score/max_score), linked to the score result file and colour-coded (green/yellow/red). More about the scoring tool here.

HC marking. Those ABs which have at least one QC1 parameter scored and marked as HC in the score result page also get a mark HC on the AB products monitor, in the column SCORES. This might be useful for identifying those ABs which created red scores on the HC monitor.

Science ABs: marking of calSelector issues. The SCIENCE ABs are compared to the output of the CALSELECTOR dynamic associations (raw2master, R2M). The output of the comparison comes as XML (X) and text file (T).

If the dynamic association has a 'false' flag in its XML file (for either completeness issues or for Raw2Raw instead of Raw2Master), this is highlighted by the tool with the X (for XML) with a red background. If the result flag is 'true' (Raw2Master and complete) the background is green. More information is available by clicking on the links.

Links to QC reports. The column "QC REPORT" actually has two entries (labelled 'QC' and 'COVER') if there is more than one report associated to the current AB. If a link 'COVER' is shown, it links to a coversheet HTML page for access to all QC reports for a given AB.

Date comment. You can enter a date comment (also entered on the histoMonitor or in createReport) which is displayed on top.

Marking of unsupported modes. If the tool finds the recipe name 'none' in the AB, and it is a 'DPR_CATG = CALIB' AB, the text

"CALIB ABs in unsupported modes - pls. check the raw frame(s)"

is displayed in the date comment field. This is meant to attract the daytime astronomer's attention to the fact that we cannot provide quality information about pipeline products for those data which have no such support. Note that you need to configure the reserved pseudo-recipe name "none" in the OCA configuration.

Browser refresh. The output page usually refreshes itself within 60 seconds. This is useful for incremental processing or during certification when the page is rather dynamic. This behaviour can become annoying if you want to investigate an issue, e.g. during certification. You can then turn this behaviour temporarily off, by clicking

browser_refresh: on (every 60 sec | stop | on); tool_refresh: off

[in this demo you will see the text turning red upon 'stop', and green upon 'on'; on the real AB monitor pages, the browser refresh is controlled as described].

Navigation. There is top/bottom navigation on both the local and the exported versions. Forward/backward navigation is also available, but only on the exported versions and on the local versions linked to the histoMonitor/phoenixMonitor. The local versions under $DFO_MON_DIR are preliminary so that forward/backwards navigation is not relevant.

DFOS:
The exported page on http://qcweb.hq.eso.org has links to the corresponding calChecker result page CAL, the raw data report report, the RAW screenshots RAW, the night log tool NLT, and the reference frame web page REF):

The local page has additional links for entering a date comment, for calling recreateAB etc.:

The RAW link to the screenshots (see below) is offered for dates younger than 7 days. The tool checks for outdatedness upon creation of the page: if the date is older than 7 dates, the link is not offered. Even if the link is created, it eventually becomes outdated. The helper tool cleanupRawdisp edits those pages (once a day) and makes the link invisible. This is necessary because the RAWDISP plots are stored only for 7 days. Without the management of the RAW link, they would eventually get broken.

The raw link (sorry for the duplication with RAW ...) is temporary only and is useful for easy checks of quality problems during certification. If you configure your browser to open e.g. rtd when it is exposed to a fits file, you can visually check the raw file with just two mouse clicks.

The score_investigate link connects to the output of the 'scoreQC -I' call, $TMP_DIR/score-investigate.html (see more here).

Furthermore, the local page has several links to manage comments: one to enter a date comment, and three for removing ("hiding") comments using the qc1_hide interface. These are: date comments, raw file comments, and product comments. One could also manage all three types of comments using the qc1_hide interface directly (DFO database part), the links are included here for convenience.

PHOENIX:

Only the links to the data report and to the night log are enabled.

RAWDISP plots (DFOS only)

The screenshots for raw files (RAWDISP plots) are created by the TQS tool qc_rawdisp.py. The getStatusAB tool creates the HTML wrapper pages and manages their distribution (from the local $DFO_MON_DIR to the exported contant on the DFO_WEB_SERVER). These pages are created per date, for CALIB data, and updated as part of the autoDaily workflow. There is a dedicated documentation page for their content, linked to each RAWDISP page (http://www.eso.org/qc/ALL/raw_help.html).

Behind the RAW link you will always find the exported RAWDISP plots. The REF link connects to the reference frames pages, the current collection of all RAWDISP plots. The 'products' link on those pages brings you back to the public version of the AB monitor.

If you have the optional text file info.rawdisp under $DFO_CONFIG_DIR, the tool will add its content on the information part of the RAWDISP pages.

Table sorting. This feature allows comfortable client-side sorting of an HTML table. It is provided by the java script 'dataTables' and is provided by the jQuery library (http://tablesorter.com/docs/). Both are installed under http://www.eso.org/observing/dfo/quality/ALL/jscript (find more technical hints there). Click on any of the main table columns and get the whole table sorted instantaneously. You can sort in asc or desc directions. For a few columns the sorting is disabled, for obvious reasons. Possible applications are quick overviews of all ABs per setting, or per recipe, per RAW_TYPE etc.

Filtering and searching. You can do two kinds of search. For the normal search, enter a string or a combination of strings. For the dependency search, use the column INDEX. Its values come in two flavours: SCI and CAL. For the dependency search, enter e.g. SCI01? and see all ABs displayed which contributed to SCI01, plus SCI01 itself. The question mark stands for "left dependencies", i.e. all ABs which are organized left of the science AB in the cascade (parent ABs) (see here). Search for parent ABs also works for CAL ABs: e.g., find a standard star AB and enter its INDEX with '?' to see all its parent ABs. Child ABs (depending ones, "right" dependencies) are displayed if you enter an exclamation mark (e.g. CAL10!).

INDEX searches work on the full data set displayed on the page, but not beyond.

For these kind of searches it might be desirable to turn off the automatic browser refresh.

The main table of the AB monitor has the following columns (all have tooltips on hovering; most of them are sortable, marked by ):

BQS

AB NAME

INDEX

COMPL.

AB LOG

RECIPE

RAW_TYPE

SETUP

AB STATUS

P LOG

T_EXEC

QC REPORT

SCORE

CERTIF

created by

vultur_exec_cascade (CONDOR script)

createAB

AB key COMPLETENESS

createAB, CalSelector

AB key RECIPE

AB key RAW_TYPE

createJob

processAB; AB key PROCESS_STATUS

processAB, esorex

processAB, esorex; AB keys TEXEC and T_QCEXEC

processQC

scoreQC

certifyProducts

linking to

$DFO_AB_DIR

association log; SCIENCE: xml, txt files, all under $DFO_AB_DIR;

calibration cascade on www.eso.org/qc

processing log, under $DFS_LOG

graphical files in certification area

$DFO_AB_DIR/<ab>.html and <ab>.tlog

AB prod_comment_<date>.txt

If called in the PHOENIX environment, COMPLETE_MODE=YES, the tool displays HISTORICAL SCORES and HISTORICAL CERTIFICATION notes for the CALIB ABs. The last two columns have corresponding labels.

dfos workflow. The tool is automatically called from processAB, processQC, certifyProducts, scoreQC and histoMonitor. It can also be called from the command line.

Output:

- status_<date>.html under $DFO_MON_DIR

Distribution scheme of status pages

The AB status output pages are distributed to the local host and to the $DFO_WEB_SERVER according to the following scheme:

This scheme ensures that locally you always see the most up-to-date status page (containing CAL or SCI ABs, whatever is just being processed), while the exported version always shows the CAL ABs and only those, unless a date is finished and histoMonitor has created the complete overview, in which case the complete status page is displayed, with both CAL and SCI ABs. In that way, the qcweb versions cannot be easily and unintentionally overwritten if you e.g. decide to reprocess a single historical AB.

How to use

Type getStatusAB -h for on-line help, getStatusAB -v for the version number, and

getStatusAB -d 2024-09-12

to obtain the AB list for date 2024-09-12. This is the standard mode for getStatusAB.

getStatusAB -a GIRAF.2024-08-31T10:00:34.345.ab

updates the page for 2024-08-30 with the entries for that specific AB only. [With option -n, there is no check done whether this AB is currently executing, for performance.]

There are further options, called internally by other tools:

getStatusAB -a GIRAF.2024-08-31T10:00:34.345.ab -e

does the same update for the CERTIF column only. This is not for interactive use but only called within certifyProducts for AUTOmatic certification.

getStatusAB -d 2024-09-12 -r

runs the AB monitor in recursive mode (not for interactive use, and for CONDOR only).

getStatusAB -d 2024-09-12 -H

will create the AB monitor page for all (CAL+SCI) ABs, based on all ABs found under $DFO_LOG_DIR/$DATE. This is the final AB monitor page linked to the nautilus (histoMonitor).

There is also the hidden mode

getStatusAB -d 2024-09-12 -F

which can be used only within the autoDaily environment. It is used for incremental CAL processing. It transfers the result HTML files, plus also all linked information to qcweb (to its logs/<date> directory) where it is immediately visible for calChecker users. This information is still preliminary (since not yet certified) and will be eventually overwritten by the certified files.

Configuration file

config.getStatusAB defines:

You can have the optional text file info.rawdisp under $DFO_CONFIG_DIR. The tool will add its content on the information part of the RAWDISP pages, for instrument-specific information.

In PHOENIX environments, the tool evaluates the optional content of AB_FILTER_COMMENT if configured in config.phoenix. This is a free text intended to help with the search option (e.g. for GIRAF_STACK: how to filter the relevant ABs out of the thousands of ABs displayed).

Operational aspects

• The tool is enabled for parallel processing, i.e. can be called for several ABs or DATEs at once.
• Option -F cannot be called on the command-line, only within autoDaily / createJob.
• Option -H can be called anytime after finishing a date, either by histoMonitor or on the command line.