Common Trending and QC tools:
Documentation

make printable	new:	see also:
	v1.1: - option -a[ll] to refresh all reports - improved design - improved structure of horizontal navbar (option BREAK) - a number of config keys has become obsolete: QC_SERVER etc. v1.1.1: - PROC_SYNTAX, PROC_EXT obsolete - QC_ADDRESS: to be included in .dfosrc!	databases none dfos tools none output visible under http://www.eso.org/~qc/docu/QC1/qc1docu.html upload/download wget for tqs/config.qcDocu_instr

qcDocu

Description

This tool helps the QC scientist to maintain the documentation of the QC procedures. These procedures are designed to extract quality information from pipeline products. They produce graphical reports, and QC1 parameters (in addition to those extracted by the pipeline). They also call the tool scoreQC which creates scores. That information is then reviewed in the certifyProducts step of the daily workflow and helps to certify, or reject, the pipeline products. The reports are stored under $DFO_PLT_DIR by moveProducts. The QC1 parameters are ingested by the QC procedures, using the DFS tool qc1Ingest.

The set of QC procedures is conceptually quite similar to the pipeline recipes: for each raw data type, there is one dedicated procedure. Their call is controlled by processQC and is scheduled right after the pipeline processing.

The qcDocu tool is designed for documentation of the QC procedures. It creates, per QC procedure, one HTML page which is transferred to the QC documentation web site, http://www.eso.org/~qc/docu/QC1/qc1docu.html. These pages are cross-linked to, and therefore externally visible from, the exported score HTML pages.

The documentation describes the procedures, the algorithm, the graphical output, and the QC1 parameters. The content of the documentation is written into the header of the QC procedures. The qcDocu tool then extracts and reformats this information.

All QC procedures are expected under $DFO_PROC_DIR. Each procedure to be documented has a set of documentation header keywords. In addition to this content, there should be example plots to represent the graphical QC reports. These example plots go to the configuration directory dedicated for qcDocu ($DFO_CONFIG_DIR/qcDocu).

In general, the documentation should be complete enough to permit a QC colleague to understand the QC reporting procedure and the criteria for acceptance or rejection. Also, this documentation might be helpful for the SciOps astronomer, in particular daytime astronomer.

Output

• HTML file with name <qc_proc>.html; delivered via scp to http://www.eso.org/~qc/docu/QC1/qc1docu.html
• the example graphical reports also also scp'ed to that web site

How to use

Type qcDocu -h for on-line help, qcDocu -v for the version number,

qcDocu -a

to create or refresh the complete set of documentation for your instrument,

and

qcDocu -p <qc_proc>

for creating the documentation page for the procedure <qc_prog>. The tool is called interactively. Note that usually you will call option -a which guarantees that all navigation is up-to-date. Option -p is only useful if you work on a specific documentation page and it is not a new one.

Installation

Use dfosExplorer or dfosInstall.

Configuration

The tool needs a configuration file and a set of reference images (screenshots or copies of your operational QC reports). All these go to $DFO_CONFIG_DIR/qcDocu (!).

There is also a general configuration file with the information about the supported instruments. This is maintained on, and downloaded from, ${INTQC_URL}/tqs/config.qcDocu_instr. You don't need to edit it. It is loaded into your $DFO_CONFIG_DIR/qcDocu anytime the tool is called.

The tool configuration file config.qcDocu defines:

Documentation content.

The documentation content is stored in the header of each QC procedure. The information is stored in the following way:

• comment marker (! for MIDAS procedure; # for shell script)
• docu key name
• content

In principle the content could be stored at any place in the procedure but for obvious reasons you should place it into the header. The only strict syntax requirement is to start with the marker, followed by a blank space, followed by white space and a valid docu key. Anything after that is considered documentation text. Technically, it is interpreted as HTML code. This means: plain text is fine; any < or > is interpreted as part of HTML tags; you may enter HTML tags explicitly ( like <b> or <br> ).

The following docu keys exist:

In the following an example header is displayed. It is the one which has been transformed by the tool to the web page http://www.eso.org/~qc/docu/QC1/GIRAFFE/bias.html. This example header is distributed as GIRAFFE_bias_filled_template.prg. There is also one called GIRAFFE_bias_empty_template.prg.

Reference plots.

Along with your documentation, you should provide a complete set of reference QC plots. These are then hyperlinked by the tool to the documentation web pages. Do the following, per procedure to be documented:

• pick an example for each plot which is generated by the procedure; you can do this from the current data flow, or by selecting examples from $DFO_PLT_DIR.
• if they do not come in a browser-compatible format, transform them (using e.g. xv) to either gif, png or jpg; don't use ps or pdf
• usually they will carry technical names using the PIPEFILE root name; rename them to something simpler like bias1. gif or flat4.png
• move them to $DFO_CONFIG_DIR/qcDocu/ where they will be found by qcDocu and scp'ed to the QC1 docu web site (provided their name is configured under DOCU_QC1PLOTS).

Operational hints.

• The QC documentation is thought as reference both for yourself and for others. Keep it up-to-date and informative.
• For each QC procedure, the corresponding web page will be created by qcDocu -p <proc_name>. Repeated calls will update the page.
• Don't forget to configure the names of all procedures in the tool configuration. Otherwise they won't appear on the horizontal navigation bar.
• The vertical navigation bar is created by the tool by reading the config.qcDocu_instr file, in a completely automatic way.
• All URLs will be added automatically, provided they are configured in config.qcDocu.
• Note the non-standard config path ($DFO_CONFIG_DIR/qcDocu/) which is required to collect the reference QC plots.