make printable	new:	see also:
	v1.3: -optional non-alphabetical sorting of set-ups in HTML overview pages, incremental update of outdated plots	- qc_rawdisp.py
	v1.4: -added html navigation, output promoted to become reference frames site v1.4.2: - added definition of RAW_TYPEs as read from the OCA rules v1.4.3: - possibility to select individual plots per AB

rawdisp2reference

Description

This tool manages the selection and publication of RAWDISP plots as references. Chosen plots are copied to $DFS_PRODUCT/RAWDISP/REFERENCE/, renamed, and published on the web server at www.eso.org/qc/<INSTRUMENT>/reports/RAWDISP/REFERENCE. Thereafter, the reference plots are found by getStatusAB when creating the RAWDISP web pages and are linked to the current plots for comparison. For usage in overview pages, also the thumbnail images are renamed, saved, and published.

The tool can be called from the command line for a specified date after moveProducts. It then expects ABs in $DFO_LOG_DIR/<DATE>. It only offers certified ABs and ABs from pipeline-unsupported modes for selecting reference plots. The RAWDISP plots are expected to be on the operational machine in $DFS_PRODUCT/RAWDSIP/<DATE>/. If this directory was already removed by finishNight, the user is asked whether the needed plots shall be re-created.

The tool is also called automatically at the end of certifyProducts while ABs are still in $DFO_AB_DIR. It then checks whether RAWDISP plots from the just certified ABs are available for settings which are currently without references. It also checks whether new plots are available for references that are older than half a year. If there are RAWDISP plots that fulfil these conditions then the user is asked for confirmation to add these plots to the already existing references.

The naming scheme for the reference plots is

<RAWTYPE>_<DO_CLASS>_<SETUP>_<EXT>.gif

<RAWTYPE>, <DO_CLASS>, and <SETUP> are taken from the RAW_TYPE, RAWFILE, and RAW_MATCH_KEY sections of the relevant AB. Header key names in the RAW_MATCH_KEYs are removed, any existing '/' in the set up is replaced by '_'. <EXT> is the fits extension number (0 for the primary header unit) in the raw file. Newly created reference plots replace automatically any existing references with the same name.

Output

The renamed reference plots exist locally at

$DFS_PRODUCT/RAWDISP/REFERENCE/

and on the web server at

www.eso.org/qc/<INSTRUMENT>/reports/RAWDISP/REFERENCE/ .

Thumbnail images are locally at

$DFS_PRODUCT/RAWDISP/THUMBS/

and at

www.eso.org/qc/<INSTRUMENT>/reports/RAWDISP/THUMBS/ .

A listing of all reference plots exists as $DFO_MON_DIR/RAWDISP/list_RAWDISP_references. It matches the plots with their original names, raw types, DO classes, instrument set-ups, and DFO dates.

Every time new reference plots have been published, a set of overview HTML pages of all plots is created. The entry page can be found at $DFS_PRODUCT/RAWDISP/REFERENCE/rawfiles_START_REFERENCE.html and at www.eso.org/qc/<INSTRUMENT>/reports/RAWDISP/REFERENCE/rawfiles_START_REFERENCE.html and is linked behind 'REF'.

How to use

Type rawdisp2reference -h for on-line help, rawdisp2reference -v for the version number.

The standard usage is

rawdisp2reference -d <DATE>

for selecting plots from <DATE> as references. It may eventually be useful to re-create the RAWDISP plots even in the case that the local directory still exists. This can be enforced with the -F option:

rawdisp2reference -F -d <DATE>

To update the overview HTML pages without adding new reference plots use the option -H:

rawdisp2reference -H

The -I option (incremental update) is normally only used within certifyProducts:

rawdisp2reference -I -d <DATE>

Configuration file

The configuration file config.rawdisp (same as for qc_rawdisp.py) must exist in $DFO_CONFIG_DIR. It can contain optionally some keywords for fine-tuning the behaviour of rawdisp2reference.

Any number of

NO_REFERENCE <RAW_TYPE>

lines mark those raw types which RAWDISP plots are never considered as reference plots.

Raw types that shall not be considered when rawdisp2reference is called from certifyProducts can be specified with

NO_INCREMENTAL <RAW_TYPE>

This is useful for raw types that are eligible for reference plots but have an infinite number of set-ups (like infrared darks). It is not necessary to repeat the NO_REFERENCE raw types in this section as these are automatically added.

The number of set-ups per table row in the HTML overview pages can be controlled via

HTML_NSET <N>

The default value is 5 which is typically a good value for SEF instruments. For MEF, this key should be set to 1.

The set-up strings in the HTML overview pages can be controlled via

LONG_SETUP YES | NO

The default value is "NO" which means that only the values of the header keys are displayed. With "YES" also the keyword names are included which can, however, result in rather long strings. Note that on "mouse over" the keyword names are always included.

Set-up strings are sorted in alphabetical order in the HTML overview pages. An alternative sorting can be configured per raw type:

SORTING <RAW_TYPE> <ORDER>

with <ORDER> being a space-separated list of character strings which must be part of the <SETUP> values (as in the file names). The UNIX shell command 'grep' is then used to select the reference plots. For example

SORTING FLAT U_ B_ V_ R_ I_

would sort flats in the order UBVRI. (The underscore is part of the set-up string and needed for identifying the set-ups uniquely.) And

SORTING FFLAT Medusa IFU Argus

would show first all Medusa set-ups, then IFU, and then Argus.

Installation

The tool comes as part of the normal dfos installation.

Operational aspects

• Use the optional file info.rawdisp in $DFO_CONFIG_DIR for useful specific information. Feel free to use links etc. Please do not use fancy colors, sizes, fonts or blinking.
• Reference plots are only useful for raw types and set-ups that are more or less regularly measured. Health check set-ups and the most frequently used instrument settings should have references, a set-up that was used only once by a user should not.
• Reference plots should be restricted to raw types that are comparable from one measurement to the other. Raw types that produce highly individual raw files like standard star observations or calibrations using science-dependent MOS masks should be de-configured via the NO_REFERENCE mechanism in config.rawdisp.
• For raw types with (potentially) infinite number set-ups (e.g. infrared darks), it is useful to choose only the most frequently used settings as reference. In order to avoid unwanted messages about newly available set-ups at the end of certifyProducts, those raw types can be de-configured with the NO_INCREMENTAL key in config.rawdisp.
• In cases where several plots of identical raw type are available per AB, rawdisp2reference chooses the first one. New with v1.4.3: The tool dialogue offers the possibility to select a different plot.
• The RAWDISP files are stored under $DFS_PRODUCT/RAWDISP where the normal operational backup is not implemented. To have an operational backup of the RAWDISP files, make sure that your config.dfosBackup contains these two lines:
  
  

  ITEM	$DFS_PRODUCT/RAWDISP	REFERENCE	RAWDISP	# RAWDISP reference plots
  ITEM	$DFS_PRODUCT/RAWDISP	THUMBS	RAWDISP	# RAWDISP thumbnail images

  
  

Removal of existing reference plots

In order to remove existing reference plots, the following steps are needed:

• Delete the entry in the list $DFO_MON_DIR/RAWDISP/list_RAWDISP_references
• Remove the reference file in the directory $DFS_PRODUCT/RAWDISP/REFERENCE/ and the thumb nail in $DFS_PRODUCT/RAWDISP/THUMBS/
• [optional] To have a clean system: ssh to qc@qcweb and remove the file in ~/qc/<INSTRUMENT>/reports/RAWDISP/REFERENCE/ and the thumb nail in ~/qc/<INSTRUMENT>/reports/RAWDISP/THUMBS/
• [optional] ssh to qc_garching@odyssey5.pl.eso.org and delete the file in ~/dfo/quality/<INSTRUMENT>/reports/RAWDISP/REFERENCE and the thumb in ~/dfo/quality/<INSTRUMENT>/reports/RAWDISP/THUMBS/

Overview HTML pages

The exported overview HTML pages form the reference frames web site for a given instrument. The page rawfiles_START_ALL.html is linked (via 'REF') for each instrument to the CAL, HC, and QC web sites. The reference pages are updated automatically by the rawdisp2reference tool.