Common Trending and QC tools:
Documentation

tqs = Trending and Quality Control System

make printable	new:	see also:
	qc_rawdisp.py v1.2.1: - option -A <DIR> for non-standard AB directory qc_rawdisp.py v1.2: - create log files in <DFO_MON_DIR>/RAWDISP/	help for raw file monitor processQC getStatusAB cleanupRawdisp rawdisp2reference
topics: components \| installation \| qc_rawdisp.py \| configuration \| operational hints

RAWDISP plots: overview of raw data

Introduction

RAWDISP plots are a set of screenshot-like overviews and detailed plots of raw calibration data for each instrument. They are organized in web pages per RAW_TYPE that allow a quick overview of the calibration quality for the particular instrument. They are created within the daily workflow and are kept for seven days within the system (both on a Garching and on a Paranal web server).

Components and data structure

The following tools are part of the RAWDISP system.

qc_rawdisp.py: This is the core of the RAWDISP system. The Python script creates the thumbnail overviews and detailed plots from raw fits files. It is called per AB from processQC.
processQC calls qc_rawdisp.py before the QC scripts are executed. It only executes qc_rawdisp.py for CALIB ABs when the date in the ABs is younger then the current date minus seven days. A second call with processQC does not overwrite already existing RAWDISP plots.
rawdisp2reference copies and renames existing RAWDISP plots so that these can be used as reference for comparison with current plots. If needed, the tool also manages re-creation of plots that have already been deleted from disk.
getStatusAB creates the overview HTML web pages per RAW_TYPE and copies plots and web pages to the DFO web server if it is called within a processing job (created from createJob) or within certifyProducts.
cleanupRawdisp removes outdated (older than seven days) information on the web servers. It is called once a day via cron.
config.rawdisp: The configuration file, read by qc_rawdisp.py and getStatusAB.

RAWDISP plots are created seamlessly via the hourly execution of autoDaily within the automated QC workflow.

Plots and web pages exist at following places:

Locally on each QC machine in <DFS_PRODUCT>/RAWDISP/<DATE>. This directory is filled via calls of processQC during execution of autoDaily, web pages are added by getStatusAB. It is deleted with finishNight.
On the Garching web server at /observing/dfo/quality/<INSTRUMENT>/reports/RAWDISP/<DATE>. getStatusAB creates the web pages locally and copies these and the plots to the server. Outdated directories are deleted by cleanupRawdisp.
On the Paranal server also at /observing/dfo/quality/<INSTRUMENT>/reports/RAWDISP/<DATE>. An automated process on the server is active that copies every 15 minutes new files to the Paranal site. Outdated direcotries are deleted by cleanupRawdisp.
Reference plots can be found in <DFS_PRODUCT>/RAWDISP/REFERENCE locally and in /observing/dfo/quality/<INSTRUMENT>/reports/RAWDISP/REFERENCE on the servers.

For logging the availability of raw fits files, log files are written into <DFO_MON_DIR>/RAWDISP/<DATE>. Outdated directories are deleted by cleanupRawdisp.

Installation and set-up

The tools are installed in the normal way using dfosExplorer or dfosInstall. There is one new configuration file, config.rawdisp, that goes into DFO_CONFIG_DIR. The tools can be run using the default configuration, but some editing will be necessary in general to:

fine tune plot appearance
have the proper ordering of detectors in the overview web pages for MEF instruments
control the number of created plots in order to avoid poor performance for huge raw files.

The maintenance script rawdispCleanup should be automatically called once a day with a crontab like:

10 00 * * * source $HOME/.bashrc;  $HOME/bin/cleanupRawdisp .

qc_rawdisp.py

The Python script qc_rawdisp.py creates for a given AB for each file listed in the RAWFILE section (up to 6 files or a configurable number of files) a thumbnail image and a detailed plot from each 2D image that the raw files contain. In case of data cubes, the first layer is plotted. The detailed plot has an image of the data, two histograms showing the full range of pixel values and an interesting area, and cuts through the raw image. It resides in DFO_BIN_DIR and is operationally called from processQC but can also be called from the command line.

Example command-line usage:

	qc_rawdisp.py -a <AB>

This creates plots for the raw files listed in <AB>. The AB must be in DFO_AB_DIR, plots are written into <DFS_PRODUCT>/RAWDISP/<DATE>.

	qc_rawdisp.py -n -a <AB> | qc_rawdisp.py --no_overwrite -a <AB>

Plots that already exist in the local RAWDISP directory are not created again. The --no_overwrite option is normally only used within processQC for performance reasons.

	qc_rawdisp.py -A <DIR> -a <AB> | qc_rawdisp.py --ab_dir=<DIR> -a <AB>

This uses <DIR> as the location where ABs are expected instead of <DFO_AB_DIR>. This is needed when plots are re-created (by rawdisp2reference) after moveProducts has been executed. Non-existing raw files are downloaded; log files in <DFO_MON_DIR> are not updated.

	qc_rawdisp.py -h | qc_rawdisp.py --help
	qc_rawdisp.py -v | qc_rawdisp.py --version

for getting a (short) tool help and the version number.

A brief description of the output can be found here.

Configuration

All necessary configuration is contained in config.rawdisp. Optional instrument-specific information can be put into info.rawdisp which then appears in the header of each overview web page. The content of config.rawdisp is as follows:

Section 1: global
MEF_NCOL	3	MEF only: number of detectors in x direction of HTML overview pages e.g. 3 (KMOS), 4 (VIRCAM), 2 (HAWK_I) Default: 1
MEF_NROW	1	MEF only: number of detectors in y direction of HTML overview pages e.g. 1 (KMOS), 4 (VIRCAM), 2 (HAWK_I) Default: 1
SUPPRESS_RAWTYPE	DARK_SPECIAL	A list of RAW_TYPEs for which no plots shall be created, one line per RAW_TYPE. Default: plots are created for all CALIB RAW_TYPEs .
REMOVE_RAW_MATCH	TPL.ID	Any raw match key that is present in the AB and that shall not be printed in the plots and in the overview HTML pages. One line per removed key. This can be useful to avoid too long output. Default: all RAW_MATCH_KEYs as present in the ABs are printed.
THUMBNAIL_SIZE	10	Size of thumbnail imges in per cent of original raw image. Default: 5 (%). Should be changed only in case of small detectors.
Section 2: per RAWTYPE This section basically controls the appearance of the plots. Default values normally give already good results but can be altered per RAW_TYPE and (if needed) also per DO_CATG. There is one line per RAW_TYPE/DO_CATG with several columns.
column name	example values	description
RAWTYPE	ANY, BIAS, FLAT_LAMP	RAW_TYPE as in AB. 'ANY' can be used to change default values for all raw types of an instrument, with the exception of those types that are configured individually.
DO_CATG	ANY, LAMP_OFF, LAMP_ON	as in RAWFILE section of AB. 'ANY' refers to all DO_CATGs of the raw type in the previous column. Useful for example to have different settings for lamp-on and lamp-off frames that are processed together and have therefore the same raw type.
MAXRAW	DEFAULT 2 2, 1,3,7	Maximum number of raw files per AB that are plotted. 'DEFAULT' refers to the default setting which is 6. This is useful to control the width of the HTML overview pages (all thumbnails per AB are plotted in one row) and/or to avoid performance issues in case of huge raw files. In case of MAXRAW=1, only the first input raw frame of the AB is plotted; for MAXRAW=2, the first and the last raw frames are plotted. Otherwise, if MAXRAW is lower than the actual number of raw frames in the AB then the first, the last, and a random selection of the remaining frames (MAXRAW - 2) are plotted. Alternatively, a comma-separated list of exposure numbers can be given, e.g. '2,' for plotting only the second exposure or '1,3,7' for plotting exposures 1, 3, and 7 .
IMAGE_CUTS	DEFAULT SIG=3 MAD=3 VAL=0,20000 MINMAX	Cut values for colours in the 2D plots. SIG=<n>: cuts = median +/- <n> * sigma MAD=<n>: cuts = median +/- <n> * MAD (Median Average Deviation) VAL=<low>,<high>: cuts = <low>, <high> MINMAX: cuts = min(image), max(image) Default: cuts = median +/- 1 * sigma.
HISTO_CUTS	DEFAULT SIG=1 MAD= 1 VAL=0,1000 MINMAX	Range of the bins in the detailed histogram. SIG=<n>: binrange = median +/- <n> * sigma MAD=<n>: binrange = median +/- <n> * MAD VAL=<low>,<high>: binrange = <low>, <high> MINMAX: binrange = min(image), max(image) Default: binrange = median +/- 6 * sigma.
HISTO_FULL	DEFAULT SIG=3 MAD= 3 VAL=0,65535 MINMAX	Range of the bins in the full histogram. SIG=<n>: binrange = median +/- <n> * sigma MAD=<n>: binrange = median +/- <n> * MAD VAL=<low>,<high>: binrange = <low>, <high> MINMAX: binrange = min(image), max(image) Default: binrange = -5000, 70000.
LINES_XY	DEFAULT,DEFAULT DEFAULT,256 512,1024 223,NONE	Positions for cuts in x and y direction through image. Format: <x_position>,<y_position>. Please note that <x_position> refers to the pixel position of the cut in y direction. <x_position> and <y_position> can be 'DEFAULT', 'NONE', or an integer number. 'NONE' means that the specific cut is not plotted; default values are the centres of the x and y axes.
EXT_LIST	DEFAULT 1,2,3 -1	List of extensions to be used for plotting. Default: all images in the primary header unit and in extensions are plotted. 0: refers to primary header unit. -1: refers to the last extension (VISIR).
MEF_NCOL_NROW	2,1	Optional last column to overwrite MEF_NCOL/MEF_NROW definition from Sect. 1. Format: <ncol>,<nrow>, e.g. 2,1 for UVES raw files from the RED arm.
Section 3: configuration for rawdisp2reference Optional section to control rawdisp2reference.
NO_REFERENCE	STD	A list of RAW_TYPEs which have RAWDISP plots but no references because raw files look significantly different from one measurement to the other (e.g. standard stars, science-dependent MOS masks, etc.).

Operational hints

After installation of qc_rawdisp.py it is a good idea to try the script on some ABs that are currently present in the DFO_AB_DIR and to verify the output.
DISPLAY variable: qc_rawdisp.py requires that the DISPLAY environment variable is set. While this is normally the case on the command line, DISPLAY is not set if processQC is called within CONDOR unless it is explicitly set e.g. in .qcrc.
For MEF instruments with lots of extensions (MUSE, OMEGACAM, VIRCAM), it will be necessary to set MAXRAW to 1 or 2 in order to avoid long execution times of processQC during automated processing.
It may be necessary for MEF instruments to experiment with MAXRAW and THUMBNAIL_SIZE so that all thumbnails of an AB fit into one row of the overview HTML page.
Cuts through the raw image can be omitted if they would give misleading information. The appearance of a cut in dispersion direction for IFU flats/arcs may, for example, depend strongly on the IFU position and could change from one day to the other which would give the wrong impression that something significantly changed.
The y axis of the cut plots has the same dimension as the x axis of the detailed histrogram and can, therefore, be controlled via the HISTO_CUTS configurations.
The SUPPRESS_RAWTYPE option shall be used only in exceptional cases.

Last update: April 26, 2021 by rhanusch

Common Trending and QC tools: Documentation

RAWDISP plots: overview of raw data

Common Trending and QC tools:
Documentation