Common DFOS tools:
Documentation
|
dfos
= Data Flow Operations System, the common tool set for DFO |
createReport
|
enabled for parallel execution (multiple instances
at a time do not interfer with each other) |
Since the tool is called in automatic mode (by autoDaily and calChecker), its performance
has been optimized with v4.0. Also, there is a safety mechanism to prevent automatic execution
if NRAW > 2000 (hard-coded).
Description
This tool creates reports for QC. It has two main modes:
- create
the daily data report for raw data (date mode)
- create all historical data reports and create reports per file properties (query mode)
Find its place in the daily workflow here.
Output
The tool creates the following output:
- in date and all modes: HTML report list_<ins>_<date>_data.html
plus txt version under $DFO_LST_DIR/REPORT, plus exported versions on qcweb
- in query mode: HTML report $DFO_LST_DIR/REPORT/extract.html and file list
DATE
mode:
- the output of the tool comes as set of HTML files, featuring
color coding of types, comfortable navigation, display of OB grades and scores
- you can enter comments:
- per raw file, using the action button enter
comments: [ file ] ;
these comments are written into the DFO database table raw_comments
(same action offered in certification workflow) ; already entered
comments display in the last column
- per date, using the action button enter
comments: [ date ] ;
these comments are written into the DFO database table daily_comments
(same ation as on histoMonitor); already entered comments display in the header
part of the HTML pages
- you can launch a query for file properties,
based on your local set of daily data reports
- you can re-create any historical report provided you have headers
- navigation:
- use the navigation to
open or close the initial nightlog section, to see the txt file or to see the full
nightlog.
- use the horizontal navigation
2007-12:
,
or the arrows to move forth and back in time; you can also jump to
the first or last report
- use the links bottom or top for
vertical navigation within any given result page, and for
incremental navigation (to the next/previous OB)
Note: this is a new feature with v3.5, you need to update earlier reports if you want
this feature for all reports.
- the report is auto-updated by finishNight after finishing the workflow,
it then registers the last DFO flag and also links to the scores
- if there is a comment for the whole date (entered here or in the nautilus tool),
it displays in the title section (provided you have updated the report after entering
the comment)
- file types come color-coded, with the following cases:
science: |
180.B-0806(B) |
| CARINA_CORE_5 |
2 |
1 |
GIRAF.2008-01-01T07:26:48.497.fits |
FLAMES_GIRAF_OBS001_0009.fits |
SCIENCE |
OBJECT,OzPoz |
MOS |
MED |
Medusa1 |
L881.7 |
POS_1_67 |
1.323 |
3600 |
|
(0/4) |
|
|
calib: |
60.A-9022(B) |
| HD49798 |
1 |
1 |
GIRAF.2008-01-01T09:09:56.964.fits |
FLAMES_GIRAF_OBS001_0010.fits |
CALIB |
STD,SimCal |
IFU |
IFU |
IFU1 |
L682.2 |
POS_1_67 |
1.776 |
16.0 |
|
(0/4) |
|
HC calib: |
60.A-9022(B) |
200117229 |
*** |
1 |
1 |
GIRAF.2008-01-01T14:20:42.358.fits |
FLAMES_GIRAF_WAVE001_0003.fits |
CALIB |
LAMP,WAVE |
MOS |
MED |
Medusa2 |
H525.8B |
POS_1_67 |
*** |
710. |
|
(0/4) |
|
hidden: |
H! 60.A-9022(B) |
200117229 |
*** |
1 |
1 |
GIRAF.2008-01-16T16:19:50.640.fits |
FLAMES_GIRAF_WAVE016_0001.fits |
CALIB |
LAMP,WAVE |
IFU |
IFU |
IFU1 |
H525.8B |
POS_1_67 |
*** |
710. |
|
very low flux, hidden |
|
All non-SCIENCE, non-CALIB or non-TECHNICAL files display grey.
All hidden files display with grey font colour and are marked by H!.
They should have a file comment about the hiding reason.
Note: in exceptional cases (e.g. instrument is not yet operational), you could configure
HIDE_FLAG = DISABLED; then no check for the hide flag is done, and of course no marking.
A clear message underneath the title reminds you about this non-default situation. The advantage
is a gain in performance (since the coding is done in a way that assumes that there is no
or just a few files being hidden).
- the reports start with the availability flag (indicating if
the file is already in NGAS)
- reports include the score flag and score result, linked
to the score HTML file. The score information is included once the corresponding files
(AB monitor, scores HTML and tlog) are moved to $DFO_LOG_DIR (i.e. after moveProducts
-m CALIB). The automatic recreation of the data report is provided by histoMonitor.
- the OB grades for science data come color-coded, and display the OB comment
if marked with a '!' and if the mouse is moved across it:
In the
second row, there is the rare case of a multiple execution of an OB. In that case, all grades are displayed, and the comments give further information related to the multiple grades.
- the output tables created by v3.7+ are sortable and can be filtered; this enhancement
is provided by the javascript datatables under $JSCRIPT_URL.
- the tool decides, based on the configuration in getObInfo, if nightlog
information should be retrieved from the associations..night_logs database (the "old format") or linked to the nightlog tool (NLT, the "new format"). Roughly speaking,
the switch between old and new scheme was on 2012-04-01 for UT2 instruments, on 2012-10-02
for the other UTs and VLTI. For the survey telescopes, there is only the new scheme,
as well as for all future instruments. [The old PANL2 database has been switched off in 2017.]
ALL
mode:
- for convenience, the all mode (-A) is offered. It takes the
first and the current last date directory in $DFO_HDR_DIR. It then starts creating the
data report for the first one, increments this by +1 and so on until it reaches the
last one.
- with this strategy, it creates a continuous set of reports, even for those dates without
any headers (and presumably without any data).
- This is useful for the nautilus and for the navigation
bar in the HTML reports. It may also be useful in case of changes in the fitsreport configuration
file: you may then want to re-create all reports to make full advantage of the query function.
- Of course this mode may take a while (hours ...) to execute.
QUERY
mode:
- Use query: [launch] to
start a query. You can also start with createReport
-Q on the command line.
- This works very much like a database query, except that it is based on
your local set of data reports. Technically, it uses 'egrep'-based pattern matching on
the set of HTML data reports which you pre-select with the 'month' (time-selection) query.
Each file in the result data set then has exactly one line with all the entries which
you have chosen in previous createReport runs, plus HTML tags of course.
- You will be prompted for a "month"; this is a simple-minded date selection
mechanism and could actually be:
- a real month (e.g. 2008-05)
- a year (e.g. 2008)
- the full range (e.g. 200)
- Then, you will be prompted for a selection string. Here you can enter
any string which can be used by egrep. The logical OR is implemented as e.g. "U_BESS|V_BESS".
The logical AND operator is implemented as '&'.
Other operators (like NOT) are not supported.
- An example: you want to find all LAMP,WAVE files in the L682.2 setting taken in 2008-01.
You enter that month, and 'LAMP,WAVE&L682.2' on the second prompt. The tool will return
a list of all LAMP,WAVE files in that setting and date range, formatted like in the standard
reports, including all comments, OB grades etc.
- The tool has a memory which makes repeated (iterated) queries comfortable:
it offers your previous selection (if any) as default choice.
- The result list is offered under the query: [read] link.
There is also a text file with the names of all selected files, useful e.g. for data
downloads or other workflow steps.
- This is (hopefully) a very powerful mechanism for investigation of non-standard
situations; calibration completeness; listings per OB or run_ID; re-processing etc.
Export
The created reports are exported to the QC web server
(http://qcweb.hq.eso.org), with the action buttons removed. They accumulate in the directory
/qc/<instr>/REPORT
which is created and managed by the tool webDircheck.
Find an example page here.
Public
HELP page
The reports are published on qcweb and linked to the calChecker result
pages. Therefore they can be read by Paranal Science operations staff. There is a public help
page linked under HELP with some hints about functionality and purpose of the pages.
How to use
Type
createReport -h | -v for on-line help and version;
createReport -d 2005-12-30
to produce a standard raw file report for date 2005-12-30,
createReport -d 2015-12-30 -F
if NRAW is higher than 2000 and the tool refuses automatic report creation,
createReport -A
for re-creating all data reports,
createReport -Q
to call the query mode.
By default the tool runs with the temporary workspace $TMP_DIR/$$ which exists
during runtime only. This protects a given instance (e.g. launched on the command line)
against another one (launched from a cronjob tool). You can override this workspace, for
debugging, by assigning a workspace with parameter -T:
createReport -d 2010-12-30 -T $TMP_DIR/test
Note:
- There is at least one config file to support the calls of
fitsreport. For the DATE mode there is fitsreport_raw.cfg.
- There is also the optional report_raw.inf file (a text file to
be pasted into your report).
Configuration
files
- config.createReport
Section 1: mode DATE (create report for daily workflow) |
INFO_FILE |
createReport_raw.inf |
name of optional info file which is pasted into the raw
file report (to reside under $DFO_CONFIG_DIR) |
HIDE_FLAG |
ENABLED |
ENABLED | DISABLED: you can disable the hide flag (e.g. if the inastrument is not yet
operational) (default: ENABLED) |
|
Section 2: special HC marker
The files found by these criteria
are marked as HealthCheck files in the HTML report (dark green instead of light green)
You can configure multiple criteria, all starting with "CHECK_HC". All files meeting
these criteria will display dark green in the HTML report . This feature is optional and thought
to support recognizing e.g. HC files. |
CHECK_HC |
&&grep CALIB | grep "SIMLAMP,TAL"&& |
|
Section 3: HTML table enhancements (new with version
v3.7)
|
SORT_COLUMN 6
|
6 |
specify default sorting column for enhanced HTML output
table; counter starts with 0 (optional, default: 1) |
NOSORT_COLUMN |
1,3,5 |
specify column indexes (starting left with 0!) that
you do not want sortable; first one (AV) is always not sortable (optional, default:
none); separate multiple indexes by comma |
-
fitsreport_raw.cfg
This file has the structure of fitsreport config files. Check
out here for details.
Operational aspects
- createReport is called by the workflow tools calChecker and autoDaily. It provides up-to-date data reports based on the headers.
- With v4.0, its performance is much faster than before. Nevertheless it
has a safety mechanism, it is not called in automatic mode if the number of headers exceeds
2000. Instead the user is warned via email and can call the tool on the command line,
with flag -F.
- Version 4.1.1 fixes a bug that prevented the tool (for a long time!) from displaying the configured INFO_FILE. Now that this is fixed, please note: the entire content is displayed as one HTML line; pls. use '|' or similar as separator.
Workflow
description: create a raw file report for QC purpose
1. call fitsreport, using fitsreport_raw.cfg, output in text file
list1, scanning $DFO_HDR_DIR/<date>/*hdr
1.1 check if NHDR is higher than 2000; if so, stop and send email (unless
called with flag -F)
1.2 check for file availability
1.3 check for hide flag
1.4 call getObInfo to get OB grades and comments
1.5 check for score information
1.6 check for raw comments
2. add a text header
4. add some statistical information
5. add a final part (created by ...)
6. write all this into the final report: $DFO_LST_DIR/list_${DFO_INSTRUMENT}_<date>.txt
Workflow for mode ALL
1. go to $DFO_HDR_DIR, find first and last date
2. ask for confirmation, then start the date loop
Workflow for mode QUERY
1. Enter a date string or a part thereof for search (this will limit your time range)
2. Enter a string for pattern search (any content of a row in the report; a setup string, a DPR key, any combination)
3. The tool creates an HTML output page with all files matching the search pattern and the date string.
Last update: April 26, 2021 by rhanusch
|