Common Trending and QC tools:
Documentation

tqs = Trending and Quality Control System
make printable new: see also:
 

qc_rawdisp.py v1.2.1:
- option -A <DIR> for non-standard AB directory
qc_rawdisp.py v1.2:

- create log files in <DFO_MON_DIR>/RAWDISP/

help for raw file monitor

processQC
getStatusAB

cleanupRawdisp
rawdisp2reference

topics: components | installation | qc_rawdisp.py | configuration | operational hints

RAWDISP plots: overview of raw data

Introduction

RAWDISP plots are a set of screenshot-like overviews and detailed plots of raw calibration data for each instrument. They are organized in web pages per RAW_TYPE that allow a quick overview of the calibration quality for the particular instrument. They are created within the daily workflow and are kept for seven days within the system (both on a Garching and on a Paranal web server).

[ top ] Components and data structure

The following tools are part of the RAWDISP system.

RAWDISP plots are created seamlessly via the hourly execution of autoDaily within the automated QC workflow.

Plots and web pages exist at following places:

For logging the availability of raw fits files, log files are written into <DFO_MON_DIR>/RAWDISP/<DATE>. Outdated directories are deleted by cleanupRawdisp.

[ top ] Installation and set-up

The tools are installed in the normal way using dfosExplorer or dfosInstall. There is one new configuration file, config.rawdisp, that goes into DFO_CONFIG_DIR. The tools can be run using the default configuration, but some editing will be necessary in general to:

The maintenance script rawdispCleanup should be automatically called once a day with a crontab like:

10 00 * * * source $HOME/.bashrc;  $HOME/bin/cleanupRawdisp . 

[ top ] qc_rawdisp.py

The Python script qc_rawdisp.py creates for a given AB for each file listed in the RAWFILE section (up to 6 files or a configurable number of files) a thumbnail image and a detailed plot from each 2D image that the raw files contain. In case of data cubes, the first layer is plotted. The detailed plot has an image of the data, two histograms showing the full range of pixel values and an interesting area, and cuts through the raw image. It resides in DFO_BIN_DIR and is operationally called from processQC but can also be called from the command line.

Example command-line usage:

	qc_rawdisp.py -a <AB>

This creates plots for the raw files listed in <AB>. The AB must be in DFO_AB_DIR, plots are written into <DFS_PRODUCT>/RAWDISP/<DATE>.

	qc_rawdisp.py -n -a <AB> | qc_rawdisp.py --no_overwrite -a <AB>

Plots that already exist in the local RAWDISP directory are not created again. The --no_overwrite option is normally only used within processQC for performance reasons.

	qc_rawdisp.py -A <DIR> -a <AB> | qc_rawdisp.py --ab_dir=<DIR> -a <AB>

This uses <DIR> as the location where ABs are expected instead of <DFO_AB_DIR>. This is needed when plots are re-created (by rawdisp2reference) after moveProducts has been executed. Non-existing raw files are downloaded; log files in <DFO_MON_DIR> are not updated.

	qc_rawdisp.py -h | qc_rawdisp.py --help
qc_rawdisp.py -v | qc_rawdisp.py --version

for getting a (short) tool help and the version number.

A brief description of the output can be found here.

[ top ] Configuration

All necessary configuration is contained in config.rawdisp. Optional instrument-specific information can be put into info.rawdisp which then appears in the header of each overview web page. The content of config.rawdisp is as follows:

Section 1: global
MEF_NCOL 3

MEF only: number of detectors in x direction of HTML overview pages
e.g. 3 (KMOS),
4 (VIRCAM), 2 (HAWK_I)
Default: 1

MEF_NROW 1 MEF only: number of detectors in y direction of HTML overview pages
e.g. 1 (KMOS),
4 (VIRCAM), 2 (HAWK_I)
Default: 1
SUPPRESS_RAWTYPE DARK_SPECIAL A list of RAW_TYPEs for which no plots shall be created, one line per RAW_TYPE.
Default: plots are created for all CALIB RAW_TYPEs .
REMOVE_RAW_MATCH TPL.ID Any raw match key that is present in the AB and that shall not be printed in the plots and in the overview HTML pages. One line per removed key. This can be useful to avoid too long output.
Default: all RAW_MATCH_KEYs as present in the ABs are printed.
THUMBNAIL_SIZE 10 Size of thumbnail imges in per cent of original raw image. Default: 5 (%). Should be changed only in case of small detectors.
Section 2: per RAWTYPE
This section basically controls the appearance of the plots. Default values normally give already good results but can be altered per RAW_TYPE and (if needed) also per DO_CATG. There is one line per RAW_TYPE/DO_CATG with several columns.
column name example values description
RAWTYPE ANY, BIAS, FLAT_LAMP RAW_TYPE as in AB. 'ANY' can be used to change default values for all raw types of an instrument, with the exception of those types that are configured individually.
DO_CATG ANY, LAMP_OFF, LAMP_ON as in RAWFILE section of AB. 'ANY' refers to all DO_CATGs of the raw type in the previous column. Useful for example to have different settings for lamp-on and lamp-off frames that are processed together and have therefore the same raw type.
MAXRAW

DEFAULT
2
2,
1,3,7

Maximum number of raw files per AB that are plotted. 'DEFAULT' refers to the default setting which is 6. This is useful to control the width of the HTML overview pages (all thumbnails per AB are plotted in one row) and/or to avoid performance issues in case of huge raw files. In case of MAXRAW=1, only the first input raw frame of the AB is plotted; for MAXRAW=2, the first and the last raw frames are plotted. Otherwise, if MAXRAW is lower than the actual number of raw frames in the AB then the first, the last, and a random selection of the remaining frames (MAXRAW - 2) are plotted.
Alternatively, a comma-separated list of exposure numbers can be given, e.g.
'2,' for plotting only the second exposure or '1,3,7' for plotting exposures 1, 3, and 7
.

IMAGE_CUTS

DEFAULT
SIG=3
MAD=3
VAL=0,20000

MINMAX

Cut values for colours in the 2D plots.
SIG=<n>: cuts = median +/- <n> * sigma
MAD=<n>: cuts = median +/- <n> * MAD (Median Average Deviation)
VAL=<low>,<high>: cuts = <low>, <high>
MINMAX: cuts = min(image), max(image)

Default: cuts = median +/- 1 * sigma.

HISTO_CUTS DEFAULT
SIG=1
MAD= 1
VAL=0,1000
MINMAX

Range of the bins in the detailed histogram.
SIG=<n>: binrange = median +/- <n> * sigma
MAD=<n>: binrange = median +/- <n> * MAD
VAL=<low>,<high>: binrange = <low>, <high>
MINMAX: binrange = min(image), max(image)
Default: binrange = median +/- 6 * sigma.

HISTO_FULL DEFAULT
SIG=3
MAD= 3
VAL=0,65535
MINMAX

Range of the bins in the full histogram.
SIG=<n>: binrange = median +/- <n> * sigma
MAD=<n>: binrange = median +/- <n> * MAD
VAL=<low>,<high>: binrange = <low>, <high>
MINMAX: binrange = min(image), max(image)
Default: binrange = -5000, 70000.

LINES_XY DEFAULT,DEFAULT
DEFAULT,256
512,1024
223,NONE
Positions for cuts in x and y direction through image. Format: <x_position>,<y_position>. Please note that <x_position> refers to the pixel position of the cut in y direction. <x_position> and <y_position> can be 'DEFAULT', 'NONE', or an integer number. 'NONE' means that the specific cut is not plotted; default values are the centres of the x and y axes.
EXT_LIST

DEFAULT
1,2,3
-1

List of extensions to be used for plotting.
Default: all images in the primary header unit and in extensions are plotted.
0: refers to primary header unit.
-1: refers to the last extension (VISIR).

MEF_NCOL_NROW 2,1 Optional last column to overwrite MEF_NCOL/MEF_NROW definition from Sect. 1.
Format: <ncol>,<nrow>, e.g. 2,1 for UVES raw files from the RED arm.
Section 3: configuration for rawdisp2reference
Optional section to control rawdisp2reference.
NO_REFERENCE STD A list of RAW_TYPEs which have RAWDISP plots but no references because raw files look significantly different from one measurement to the other (e.g. standard stars, science-dependent MOS masks, etc.).

[ top ] Operational hints


Last update: April 26, 2021 by rhanusch