Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
 

v4.1:
- display of multiple OB grades

v4.1.1:
- bug fix with INFO_FILE (here)

[ used databases ] databases opc70, obs_metadata; ngas for availability check; DFO_db for comments
[ used dfos tools ] dfos tools fitsreport, qcdate, getObInfo
[ output used by ] output used by histoMonitor, dfoMonitor, calChecker , ngasMonitor
[ output used by ] upload/download export to qcweb: data reports (html and txt)
javascript javascripts jquery-latest.js, jquery.datatables.js under ${JSCRIPT_URL}
topics: description | modes: DATE | ALL | QUERY| export | help page | config: tool | config: fitsreport | workflow | public output example

createReport

enabled for parallel execution (multiple instances at a time do not interfer with each other)

Since the tool is called in automatic mode (by autoDaily and calChecker), its performance has been optimized with v4.0. Also, there is a safety mechanism to prevent automatic execution if NRAW > 2000 (hard-coded).

[ top ] Description

This tool creates reports for QC. It has two main modes:

Find its place in the daily workflow here.

Output

The tool creates the following output:

[ top ] DATE mode:

science: 180.B-0806(B)
A 287269
CARINA_CORE_5 2 1 GIRAF.2008-01-01T07:26:48.497.fits FLAMES_GIRAF_OBS001_0009.fits SCIENCE OBJECT,OzPoz MOS MED Medusa1 L881.7 POS_1_67 1.323 3600 (0/4)  
calib: 60.A-9022(B)
X 200170106
HD49798 1 1 GIRAF.2008-01-01T09:09:56.964.fits FLAMES_GIRAF_OBS001_0010.fits CALIB STD,SimCal IFU IFU IFU1 L682.2 POS_1_67 1.776 16.0 (0/4)  
HC calib: 60.A-9022(B) 200117229 *** 1 1 GIRAF.2008-01-01T14:20:42.358.fits FLAMES_GIRAF_WAVE001_0003.fits CALIB LAMP,WAVE MOS MED Medusa2 H525.8B POS_1_67 *** 710. (0/4)  
hidden: H!  60.A-9022(B) 200117229 *** 1 1 GIRAF.2008-01-16T16:19:50.640.fits FLAMES_GIRAF_WAVE016_0001.fits CALIB LAMP,WAVE IFU IFU IFU1 H525.8B POS_1_67 *** 710.   very low flux, hidden

All non-SCIENCE, non-CALIB or non-TECHNICAL files display grey.
All hidden files display with grey font colour and are marked by H!. They should have a file comment about the hiding reason.
Note: in exceptional cases (e.g. instrument is not yet operational), you could configure HIDE_FLAG = DISABLED; then no check for the hide flag is done, and of course no marking. A clear message underneath the title reminds you about this non-default situation. The advantage is a gain in performance (since the coding is done in a way that assumes that there is no or just a few files being hidden).

[ top ] ALL mode:

[ top ] QUERY mode:

[ top ] Export

The created reports are exported to the QC web server (http://qcweb.hq.eso.org), with the action buttons removed. They accumulate in the directory /qc/<instr>/REPORT which is created and managed by the tool webDircheck. Find an example page here.

[ top ] Public HELP page

The reports are published on qcweb and linked to the calChecker result pages. Therefore they can be read by Paranal Science operations staff. There is a public help page linked under HELP with some hints about functionality and purpose of the pages.

How to use

Type createReport -h | -v for on-line help and version;

createReport -d 2005-12-30

to produce a standard raw file report for date 2005-12-30,

createReport -d 2015-12-30 -F

if NRAW is higher than 2000 and the tool refuses automatic report creation,

createReport -A

for re-creating all data reports,

createReport -Q

to call the query mode.

By default the tool runs with the temporary workspace $TMP_DIR/$$ which exists during runtime only. This protects a given instance (e.g. launched on the command line) against another one (launched from a cronjob tool). You can override this workspace, for debugging, by assigning a workspace with parameter -T:

createReport -d 2010-12-30 -T $TMP_DIR/test

Note:

[ top ] Configuration files

- config.createReport

Section 1: mode DATE (create report for daily workflow)
INFO_FILE createReport_raw.inf name of optional info file which is pasted into the raw file report (to reside under $DFO_CONFIG_DIR)
HIDE_FLAG ENABLED ENABLED | DISABLED: you can disable the hide flag (e.g. if the inastrument is not yet operational) (default: ENABLED)
 
Section 2: special HC marker
The files found by these criteria are marked as HealthCheck files in the HTML report (dark green instead of light green)
You can configure multiple criteria, all starting with "CHECK_HC". All files meeting these criteria will display dark green in the HTML report . This feature is optional and thought to support recognizing e.g. HC files.
CHECK_HC &&grep CALIB | grep "SIMLAMP,TAL"&&  
Section 3: HTML table enhancements (new with version v3.7)
SORT_COLUMN 6


6 specify default sorting column for enhanced HTML output table; counter starts with 0 (optional, default: 1)
NOSORT_COLUMN 1,3,5 specify column indexes (starting left with 0!) that you do not want sortable; first one (AV) is always not sortable (optional, default: none); separate multiple indexes by comma

[ top ] - fitsreport_raw.cfg

This file has the structure of fitsreport config files. Check out here for details.

Operational aspects

[ top ] Workflow description: create a raw file report for QC purpose

1. call fitsreport, using fitsreport_raw.cfg, output in text file list1, scanning $DFO_HDR_DIR/<date>/*hdr

1.1 check if NHDR is higher than 2000; if so, stop and send email (unless called with flag -F)
1.2 check for file availability
1.3 check for hide flag
1.4 call getObInfo to get OB grades and comments
1.5 check for score information
1.6 check for raw comments

2. add a text header

4. add some statistical information

5. add a final part (created by ...)

6. write all this into the final report: $DFO_LST_DIR/list_${DFO_INSTRUMENT}_<date>.txt

Workflow for mode ALL

1. go to $DFO_HDR_DIR, find first and last date

2. ask for confirmation, then start the date loop

Workflow for mode QUERY

1. Enter a date string or a part thereof for search (this will limit your time range)

2. Enter a string for pattern search (any content of a row in the report; a setup string, a DPR key, any combination)

3. The tool creates an HTML output page with all files matching the search pattern and the date string.


Last update: April 26, 2021 by rhanusch