make printable	new:	see also:
	v4.6: - BADQUAL dialogue - forced refresh button - check for other instance	- output of this tool is operationally relevant for Paranal QC - tellTracker - the tool uses OCA syntax - define your CALCHECKER calib map using createCalibMap
	v4.6.1: - optional PGI_FILTER v4.6.2: - enabled for P100 prog_Ids	databases observations..calibration_queues for B/EOCAL signal dfos tools called createAB; getObInfo; output linked to tellTracker output HTML output under $DFO_MON_DIR/CALCHECK upload/download upload: HTML files exported to ESO web server and QC web server javascript jquery-latest.js, jquery.datatables.js under ${JSCRIPT_URL}
topics: description | CAL4CAL and tellTracker | strategy | incremental execution | memory | full | repair | cal_scores | certification | b/eocal calibration configuration: OCA | long-term calibrations | nighttime || interfaces: monitor (main page) | details workflows: BADQUAL | analysis | forced refresh || technical: mirror web site | news section || tool configuration | quick links | operations | workflow
You need to initialize the scp connection to the Paranal mirror server odyssey5.pl.eso.org: - call 'calChecker -s' - follow the instructions - make sure to not have 'BatchMode=yes' in your ~/.ssh/config file If all went fine, you don't need this anymore. Find information about the replication script here.

calChecker

You may also want to read the information linked to the public result page (here) and the general calChecker home page.

Description & concepts

This tool monitors the Paranal calibration plan. It detects missing calibrations and provides feedback to QC Garching as well as to SciOps Paranal.

It scans the header files of the last 7 nights. It reads all headers of CALIB files in the defined period, classifies them and creates headers of virtual calibration products. These headers define the pool of available calibration files.

Then, in a second step, it reads the headers of all SCIENCE files and creates association blocks. The SCIENCE ABs (actually the ALOG files which are derived from them) are then scanned for completeness and time matches, in the usual createAB way.

The third step involves an evaluation of the created SCIENCE ABs. These ABs are not evaluated one by one, but per "setup box" (a single cell in the main interface). A setup box is defined by a datatype, setup and date. Each setup box is analyzed as:

• OK if all types of calibrations are available, within the configured time ranges
• NOK if all types are available but at least one CALIB file type is outdated
• MISS if at least one CALIB file type is missing
• LOST for a MISS or NOK case if that calibration cannot be taken anymore (special case e.g. for STD_IMA if that night is over).

After creation, ABs are evaluated here only in terms of setups. If e.g. a sequence of 50 science files has been measured, all in the same setup, then all science ABs will be OK, in terms of association, if one is OK.

Usually, NOK and MISS setups can be properly calibrated within an acceptable time window if PSO is informed properly and gets a chance to acquire the missing calibrations. To detect those cases is the main purpose of this tool.

LOST cases cannot be recovered, but that information can be used by PSO to review the OB grades. When the current night is over (UT > 10:00), or for older dates than $TODAY, a MISS calibration configured in the RAWFILE_NIGHT section causes a LOST case.

In case of hybrid cases, a MISS+NOK case is flagged as MISS. A MISS+LOST is flagged as MISS, a NOK+LOST case is flagged as NOK.

There is also the option to provide a "preliminary comment" which is just a comment without analysis. This could be useful in case of

• LOST: with the proper analysis being 'OK' but the action to re-grade the OB has not been taken yet; then you/PSO can provide a preliminary comment "OB to be graded C, USD informed"; the final analysis to OK is waiting until the ticket is closed;
• mixed cases LOST and MISS: a STD has been omitted, no problem; a TWIFLAT is missing, must be taken; then you/PSO can provide a preliminary comment "missing STD is not an issue; TWILIGHT flat pending"; the final analysis to OK is waiting until TWIFLATs are taken.
  

CAL4CAL mode. This is a mode which calls a separate set of configuration files and checks calibrations depending on other calibrations (as opposed to calibrations depending on science data as in the classical case). For example, a standard star calibration (also called the triggering calibration) needs a set of DARK and FLAT calibrations (dependent calibrations). These dependent calibrations are not necessarily identical to the ones needed for science, and there might also be nights when no science in these setups have been taken but only the triggering calibration (like the standard star). In those cases one might want to ensure that the triggering calibration can safely be calibrated, like the science data in the classical case.

The configuration, workflow and results are very similar to the science case. The CAL4CAL mode is called within the normal calChecker call. It must be enabled in the configuration, by default it is disabled, to account for those instruments having no need for CAL4CAL checks. If enabled, the results are displayed on a second page being linked to the first one. Find more about CAL4CAL configuration here. The CAL4CAL result page has the same functionalities (action buttons) as the main result page (e.g. ANALYSIS, BADQUAL).

The CAL4CAL mode has no LOST cases, but the preliminary comments can be used as well.

tellTracker. For some instruments, a link to a third output page is offered, the tellTracker. The link is enabled by setting the optional configuration key CHECK_TELLTRACK to YES. tellTracker is called stand-alone, with higher cadence than calChecker and follows time-critical night calibrations like telluric standard stars. If there is an issue with the current last night, a flag ! is displayed. Check out here for more.

Links to RAW file screenshots and calibration product quality. The tool provides easy access to RAW file screenshots (as created by the python tool qc_rawdisp.py) and to the exported AB product monitor pages. However, these pages are not created or maintained by calChecker itself, but by autoDaily and getStatusAB in incremental processing mode.

Strategy. The strategy of the tool is to follow the behaviour of the calOBbuilder tool used on Paranal to define the daytime calibration queue. It goes beyond that in the sense that also nighttime calibrations (standard stars) and twilight calibrations (sky flats) can be included in the configuration. These types of calibrations are often subject to be overlooked by daytime staff on Paranal. Also, it can monitor long-term calibrations which are not triggered by science exposures at all. It also checks CAL4CAL data (calibrations for calibrations).

In short:

Included:

all daytime calibrations defined in the calibration plan as driven by the science observations (e.g. dome flats)

all twilight calibrations defined as science-driven (e.g. twilight flats)

all nighttime calibrations defined as science-driven (standard stars; configured if they qualify for LOST case)

long-term calibrations (maintenance, technical, health checks)

all calibrations triggered by other calibrations (daytime, twilight, nighttime)

Excluded:

optional attached calibrations (they cannot be detected as missing or ok!)

short-term calibrations (like telluric standards to be taken within time and airmass constraints; these are monitored with tellTracker).

Incremental execution

Scanning 7 nights twice (CAL4CAL and SCIENCE) costs performance. Although the calChecker results are analyzed and displayed per setup, ABs need to be created in the first place. Hence the tool performance scales with the number of ABs, actually with the number of raw files (headers). There are cases when a full calChecker execution would take more than half an hour which is fatal for automatic cronjob execution.

For all dates scanned so far for which the calChecker result boxes are green (OK), a repeated scan is normally obsolete (unless the BADQUAL mechanism is used, see below). Therefore an incremental scanning strategy is usually fully sufficient. The following rules are implemented:

• For the calibration data, only the last (current) date is scanned and used for AB creation. All previous dates have all virtual calibrations already in the VCAL memory. This saves 6/7 of the execution time for CALIB AB creation (assuming that all dates have equal size).
• For the science data, always the last date is scanned, plus all previous dates which have a MISS or NOK result. The other dates are skipped (assuming that an OK result will be OK again when new calibration data are added). They are not skipped if a LOST case exists for that night, because that requires an analysis to become OK.The incremental execution saves between 6/7 (all dates are OK) and 0/7 (all dates are NOK) of the execution time for SCIENCE AB creation (assuming equal size).
• In incremental mode, you can still enforce the re-evaluation of a certain date with the repair mode.
• If OLD issues exist (older than the canonical 7 days), they are left on the result table, and the corresponding dates are included in the SCIENCE scans, until they are resolved by new calibrations, or they ultimately expire (as configured in N_VAL_MEM).
  

The following table displays a set of typical OK/NOK cases and explains their origins.

DATE*:     [?] [color if science data acquired]		2010-09-14 SM 71 report | NLT	2010-09-15 SM 320 report | NLT	2010-09-16 SM 15 report | NLT	2010-09-17 SM 97 report | NLT	2010-09-18 SM 5 report | NLT	2010-09-19 report | NLT	2010-09-20 SM 13 report | NLT
r... Raw CAL displays:     [?]		raw	raw	raw	raw	raw	raw	raw
P... Product CAL quality:     [?]		products	products	products	products	! products	products	products
Data types (Mode):	Setup:
SCIENCE_IMG_BROAD	ND_10_H				nok	ok
	ND_10_J	ok	LOST		ok	ok
	ND_114_Y	ok		ok		ok		nok [daycals not yet started]
	ND_5_Ks			nok				nok [daycals not yet started]
SCIENCE_IMG_NARROW	ND_10_CH4				miss	ok analyzed: [1]
	ND_20_H2	ok
	ND_20_NB2090	ok
checked in incremental mode?	CALIB	NO why? 1*	NO 1	NO 1	NO 1	NO 1	NO 1	YES 1
	SCIENCE	NO why? 2*	NO 7	YES 3	YES 3	NO 4	NO 5	YES 6
*Why? 1 - CALIBs scanned only for current (last) date (other dates are already complete in VCALs) 2 - SCIENCE not scanned if all setups OK for that date (nothing to improve) 3 - SCIENCE scanned again if a setup returns NOK or MISS for that date 4 - same as 2, original NOK/MISS result overridden by OK analysis (nothing to improve) 5 - no scan if no SCIENCE data 6 - SCIENCE always scanned for current (last) date 7 - SCIENCE not scanned since the calibration situation cannot change anymore, but the result table is scanned to catch if the analysis turns into 'OK'

ANALYSIS results entered through the ANALYSIS workflow (see below) are taken into account:

• If a MISS or NOK has been forced into an OK (by the QC scientist or daytime astronomer), it will be treated as OK in incremental mode (case 4).

BADQUAL results entered through the BADQUAL workflow (see below) are taken into account:

• If an OK is forced into a MISS or NOK by the BADQUAL workflow, the corresponding night is forced to be analyzed again in the next call. This is included in the BADQUAL mechanism. If you override this mechanism, it is your responsibility to ensure consistency of the calChecker results.

Long-term calibrations are always scanned completely.

Memory about open issues. If an open issue exists (NOK, MISS; not: LOST!) that is older than 7 days, it is kept in the calChecker result table for as long as that date falls within the total time range defined by N_VCAL_MEM. The output design is slightly different from the main date range: the colors are pale and the font is smaller, this to indicate that the memory time range is not displayed completely but only for the dates having an issue. Issues in the memory range can be analyzed. As soon as all calibration issues for a date in the memory are solved, that date disappears from the output. This handling applies to both 'science' and 'cal4cal' parts of the calChecker, independently. Issues in the memory range contribute to the calibration scores (see below) just like the normal ones.

Full execution. A full execution is necessary when the OCA configuration has been modified. Also it might happen that the tool has been screwed up, or the results are suspicious.

A full execution is possible with the option -F. In mode -F, the full calibration memory is reconstructed, then all CAL4CAL and SCIENCE dates are scanned. This mode is enabled to kill all other running instances.

Operationally, it is required to call 'calChecker -F' once a day in the cronjob file (this is also checked by the dfoMonitor). This provides you with a complete reconstruction of all calChecker information (if corrupted), and a clean restart (if hanging). The suggested pattern is:

# calChecker: every half hour; plus once a day FULL:

10,40 * * * * . .bashrc; export TZ=Europe/Berlin; $HOME/bin/calChecker > /tmp/giraffe/CALCHECK/cron.log

41 18 * * * . .bashrc; export TZ=Europe/Berlin; $HOME/bin/calChecker -F > /tmp/giraffe/CALCHECK/cron.log

Important: 10,40 is your choice; the time of the FULL mode should always be slightly later than a normal call (41 in this case), to ensure that the FULL mode correctly kills any previous instance. If you do it the other way round (like 39), the incremental instance will crash into the FULL instance one minute later, with unwanted results.

Call the FULL refresh after the end of daytime calibrations, before the new night starts. A reasonable choice seems 18 UT, as in the example.

Repair a date in incremental mode. It might happen that a certain date has inconsistent entries. For instance, headers got lost. To satisfy the need to reconstruct the results for this particular date, without running the full refresh, there is the 'repair' mode of calChecker. (In practise, running the full mode may easily cost half an hour or more and bear the risk that a cronjob-triggered instance starts and gets corrupted.) Calling 'calChecker -d 2018-09-15' for the above scenario will add the specified date to the list of dates to be checked. Technically, this date is removed from the list of OK dates. The repair mode is always applied to the calibrations plus the science data, and both for the regular and the CAL4CAL mode. It can only be executed interactively.

Important: if you change your OCA configuration, the above assumption (an OK result will be OK again when new calibration data are added) is not valid anymore. Then you should always enforce a full execution (option -F).

The execution mode (incremental or full) is indicated at the very bottom right of the main result page:

increm

full

Calibration scores. Inspired by the HC monitor, the calChecker monitor also has scores. They are not related to product quality but to completeness. They are called 'cal_scores'. They are intended to make it easier to see an issue on a 'hidden' calChecker page (e.g. on cal4cal while you scan 'science'). The following values exist:

	all calibrations existing and within validity time range
	issue with outdated or missing calibrations but daytime calibs not yet finished
	issue with outdated or missing calibrations, daytime queue finished

The cal_scores are determined for the SCIENCE and the CAL4CAL mode independently. They are exported to the URL www.eso.org/qc/<instr>/reports/CAL from where they are embedded in the overview page. (For the cal_scores, a LOST issue is handled the same way as a MISS issue.)

Certification. The links labelled "Products quality" are marked products if the CALIBs for that date have already been certified by QCG (to indicate to PSO that the product quality has been checked and issues been raised). Those nights which are not yet certified are marked products . An exclamation mark ! marks nights with data coming in unsupported modes, to highlight the necessity of visual checks of raw data. See the above example output page.

Begin/end of calibration signal (B/EOCAL). The begin (BOCAL) and end (EOCAL) of calibration signals are read from the database. These signals are inserted (with timestamp, per instrument) upon start and end of the daytime calibration queue. This information is used to improve the feedback for the last date. There are three values: pending; ongoing; finished. Only upon finishing the daytime calibration queue, a NOK/MISS flag becomes relevant. These flags can also be used to detect a broken/aborted daytime queue.

There is the special case of UVES and FLAMES/UVES: the BEOCAL signals come twice, once for UVES and once for FLAMES. calChecker evaluates them following these rules:

• BOCAL: UVES signal taken (the FLAMES signal comes almost simultaneously, if it comes, and is obsolete)
• if a FLAMES BOCAL was fired, the overall EOCAL is determined by the FLAMES EOCAL (the UVES EOCAL is ignored)
• if no FLAMES BOCAL was fired, the overall EOCAL is determined by the UVES EOCAL.

This special rule set is needed to avoid false "calibrations missing" signals for FLAMES modes.

Calibration configuration

OCA configuration. The tool uses createAB, ABbuilder and a dedicated OCA configuration. The OCA configuration files used for the calChecker are a subset of the operational files <instr>_organisation.h/association.h/classification.h. They go to $DFO_CONFIG_DIR/CALCHECK/OCA. They are simplified copies, simplified in the following sense:

• calChecker creates ABs for association only: no need for information related to processing or packing.
  This means you can remove DO.CLASS, recipe and parameter information, PACK.TYPEs, RASSOC and MASSOC configuration. (They do not confuse the tool if existing.)
• calChecker is driven by the SCIENCE cascade; you need to include here only data types (RAW.TYPEs) which produce calibrations that are needed for science reduction (and for the triggering calibrations in CAL4CAL, see below).
• calChecker effectively checks, and associates, raw data with raw data; if a (science or calibration) raw file type depends on another raw file type, this dependency must be expressed by the name of a virtual product. For calChecker, you need one virtual product only.
• Any association with optional file types should be removed, to avoid false alerts.
• General mcalibs should be removed (by definition they cannot have parent raw data which need to be generated day by day or frequently).
• There is no need to include Health Check calibrations in the calChecker rules (they don't associate to science data and won't be discovered anyway). If needed include them in the long-term calibrations.
• NOTE: do not use the string 'NR' in your RAW_TYPE definitions. This is a reserved string for calChecker. Using it for RAW_TYPEs would lead to output corruption.
• Make sure to create up-to-date calib maps (using createCalibMap) since the science RAW_TYPEs are linked to those maps.

For CAL4CAL, you need a second set of OCA configuration files. They go to $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA. See more here.

A completeness check for SCIENCE files is added (provided by createAB). A comparison is made between an archive query, having all science files, and a list extracted from all created science ABs. Any difference, except for the current date, is reported in an email, as to protect against undetected OCA configuration mistakes. In exceptional cases this test can be turned off: if raw files are artifically split into raw files per detector (applies for VIMOS and FORS2 only: pseudo-MEF), the calChecker OCA configuration will legally apply to one of the possible detectors only. Then a completeness check would constantly create false alerts and can be turned off with SUPPRESS_ABCHECK=YES.

Calibration cascade for normal (science) mode of calChecker. Completeness of calibrations for the triggering science data types and setups is checked.

N_VCAL_LIST: By agreement with PSO and QC, the operational use of this OCA parameter is 7 days. It cannot be configured but is hard-coded. Smaller or larger ranges can be used for testing (call option -D <n>). They cause a warning and an email. Much smaller ranges may result in inconsistencies (headers and nightlogs are not delivered instantaneously; the daytime calibrations take some time; the calibration plan may foresee validity ranges larger than 24 hours). Larger ranges cost performance and don't make sense operationally, but they might be useful for testing, e.g. in order to catch more instrument modes and check proper configuration. For a complete refresh of all 7 days, call option -F; this option will also refresh the entire calibration memory which is either configured as N_VCAL_MEM or set by default to 2*7 days.

In addition to the user-defined OCA files, the tool creates its own config.createAB on the fly and copies all other standard configuration files (like macro.h) from the operational OCA directory.

Stripping down the configuration to the "essential cascade" pays off in terms of performance: in FULL mode the tool needs to create ABs for 2*N nights where N is the configured N_VCAL_LIST and the factor 2 comes from running it in modes CALIB and SCIENCE.

OCA CAL4CAL configuration: there is a dedicated set of CAL4CAL configuration files. They come in OCA style and syntax and reside under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA. Let's assume an example with STD as the triggering calibration, and DARK and FLAT as depending calibrations.

1. OCA files. Best strategy is to start with the OCA set already operational for calChecker, copy the three OCA files from $DFO_CONFIG_DIR/CALCHECK/OCA to $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA and then trim and edit them. Use the same setup key names, rule names, macros etc. as in the classical calChecker OCA rules.

All rules as listed above for the SCIENCE checks apply here, plus the following:

• You need to include in the three OCA files only the data types (RAW.TYPEs) for the triggering calibrations for CAL4CAL (the STD data types in our example). In most cases the depending calibrations (DARKs, FLATs) are already defined in the OCA files for the main mode. Only if this is not the case, make sure to add them there. They will be read only there.
• Mark the triggering calibrations as
  CATG = "SCIENCE";
• There is no need for adding definitions of depending calibrations here: the tool never reads them.
• All other files in $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA will be copied over from $DFO_CONFIG_DIR/CALCHECK/OCA by calChecker.

2. config.calChecker: edit the one under $DFO_CONFIG_DIR/CALCHECK to include the key CHECK_C4C (see here). Then the CAL4CAL mode is enabled. The file with the same name under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL is an auto-copy and needs no editing.

3. info.calChecker: provide a file with this name and the proper content as an edited copy under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL.

All other files are simple copies of the corresponding files under $DFO_CONFIG_DIR/CALCHECK/OCA and are auto-copied by calChecker.

Calibration cascade for CAL4CAL mode of calChecker. Completeness of CAL4CAL calibrations (the ones for the triggering calibration data types and setups) is checked.

Long-term calibrations. The tool can monitor these calibrations which can be Health Check calibrations or maintenance/monitoring calibrations. Their defining property is not necessarily the longer timescale but the fact that they are not triggered by science observations. For performance reasons, these types of calibrations are searched for not by OCA technology, but by pattern matching in the daily raw file reports. If e.g. the last set of detector monitoring flats is to be monitored, the tool checks for the pattern "LAMP,FLAT,DETCHECK" and maybe also for a certain template name, a minimum number of exposures, a specific exposure time etc. Of course all strings of the pattern must be included in the report configuration (see the createReport documentation) and must be unique.

They are monitored on the calChecker in the following way:

• they have a configured validity (e.g. 30 days)
• the last occurrence of each data type is scanned in the raw data reports; the tool iterates backwards in time, to a maximum of one year
• the current age of the last instance is evaluated and flagged as
  • OK/green: if it is younger than 70% of the validity
  • soft REMINDER/yellow: younger than validity but within 30%; recommended action: refresh on next occasion
  • REMINDER/yellow: outdated but by not more than 30%; recommended action: refresh on next occasion
  • REMINDER/red: outdated by more than 30%, refresh as soon as possible

• the tag defining the type of calibration (e.g. DETECTOR_MON for the detector monitoring sequence), and a description of what that means; that description is meant to be specific enough to identify clearly the particular data set. It may contain the template name, a set of DPR keys etc.
• the validity and age in days
• a slightly more extensive version of the recommended action

The long-term calibrations have their own configuration file, see here.

There is the link how to execute to a page containing information for the Paranal astronomers how to execute the corresponding templates. The content of that page is provided by the instrument scientists, on a PSO wiki page.

Configuration of nighttime calibrations for the 'LOST' case. Certain data types cannot be taken after a night is over, e.g. photometric standard stars. As a rule of thumb, this applies to those calibrations with a validity of 0.5 or less ("same night"). They need to be configured (section 2, RAWFILE_NIGHT) since not all standard stars fall into that regime (e.g. the ones for spectroscopic modes are typically optional or have a longer validity).

CALCHECKER monitor

Main output page. The tool produces an HTML page, the calChecker monitor. This monitor (see here an exported example) has a result table displaying, per setup and date, the association flags (OK/NOK/MISS/LOST). The standard time depth is 7 days.

The tool also creates a simplified, user-friendly version of the association rules. This is accessible under "ASSOC-RULES". For investigation, it also links to an HTML page with details (file by file).

On top of that page, there is update information and a link to the Data Transfer Monitor.

The science RAW_TYPEs are linked to the calib maps.

For CAL4CAL enabled installations, two main output pages exist: the normal page for the SCIENCE data, and the CAL4CAL page for the CAL4CAL data. Both pages have the same format. They are linked and you can switch forth and back:

science  .   cal4cal

DATE*:   [?] [color if science data acquired] 2011-01-03 SM 54 report | NLT 2011-01-04 SM 128 report | NLT 2011-01-05 SM 224 report | NLT 2011-01-06 SM 250 report | NLT 2011-01-07 SM 162 report | NLT 2011-01-08 VM 46 report | NLT 2011-01-09 VM 48 report | NLT

On installations with CAL4CAL disabled, there is a blind link only:

science  .  no cal4cal

DATE*:   [?] [color if science data acquired]

The tellTracker-enabled instruments have a third link:

science	.	cal4cal	.	tellTracker

Reports. The main page has links to data reports ("report") and nightlogs ("NLT"), to the RAW file displays ("raw"), and to the AB monitor ("products"). This link implements the feedback about calibration data quality. See more here. The product quality pages (AB monitor) are always linked to the ABs, to the logs and to score information.

The CAL4CAL output table looks the same, it lists the triggering calibration data types instead of the science data types.

Detailed overview (similar for normal and CAL4CAL).

The evaluation labels (ok/nok/miss) link to a result table (sorted per date) with details useful for understanding and analysing a problem. This page lists all files affected, with details about run_id, OB id, comments and grades. It is a sortable table, per default sorted by DATA_TYPE, SETUP and RAW FILE:

Calibration report for all science files, for date 2008-06-06
This is the detailed calChecker report about the calibrations for all science OBs from the indicated date. Note: all science data with PROG_ID starting with 60. or 060. or 0060. are ignored.
DATE	PROG_ID	MODE	OBS_ID	GRADE	OB Comment	(first) RAW FILE	DATA TYPE	SETUP		CALIBRATIONS
2008-06-06	381.D-0329(B)	SM	307629	C	01:49: stopped after 2400 secs fue to a robot problem. OB must be repeated	GIRAF.2008-06-07T01:50:28.491.fits	SCIENCE_MED	Medusa1_H875.7		BIAS 0.44	FLAT 0.47	WAVE -0.83
2008-06-06	081.D-0286(A)	SM	306951	A	05:09: Guide probe seeing always under constraints	GIRAF.2008-06-07T05:13:25.230.fits	SCIENCE_MED	Medusa1_H875.7		BIAS 0.24	FLAT 0.27	WAVE -0.91
2008-06-06	081.D-0253(A)	SM	310094	A		GIRAF.2008-06-07T06:40:57.467.fits	SCIENCE_MED	Medusa1_H651.5B		BIAS 0.10	FLAT 0.12	WAVE 0.15
2008-06-06	081.D-0253(A)	SM	310095	A		GIRAF.2008-06-07T08:01:59.264.fits	SCIENCE_MED	Medusa1_H651.5B		BIAS 0.054	FLAT 0.04	WAVE 0.04

Special workflows

Action buttons for calChecker interface.

The calChecker monitor has a button and link panel. The leftmost button 'HC' links to the HC monitor. There are three actions buttons that can be used externally. They link to special workflows that are supported by php scripts.

'Mark BAD QUALITY' mechanisms. The calChecker tool checks for the formal availability of calibrations but not for their quality. As an example, flats may be saturated, or an arclamp exposure may have had the lamp turned off. It might be necessary to remove those calibration files from the pool visible to calChecker in order to flag a quality issue along with the need to take better data. This is supported in two way: with the BADQUAL button (for SciOps), and with the local BADQUAL file (for QC).

BADQUAL button. This button opens a php-based dialogue that is primarily intended for SciOps on Paranal. This dialogue offers all calibrations of TODAY (only!), for being marked as having BAD quality. The interface asks for the email address (for notification about the successful handling), and then flags the marked AB as 'BAD'. You can mark multiple ABs in one go. You can call the interface multiple times.

With the marking as 'BAD', these ABs are not yet properly considered on the calChecker interface. This happens during the next calChecker execution (either scheduled by the cronjob, or triggered by the 'forced execution' button). Then these ABs are marked as 'HIDDEN', they are not visible for calChecker anymore and will likely set a NOK/MISS issue and eventually be taken again, under good conditions.

Important to know:

• The data offered to hide are organized in ABs, since hiding some individual files and leave others unhidden makes no sense for data processing.
• The offered ABs are taken from the pool of operational (DFOS_OPS) ABs since in general it is not easily possible to recognize the calibrations needed for calChecker from the ones taken for other purposes. Hence they reflect the state-of-affairs as seen from the autoDaily perspective. It might happen that very recent data become visible later.
• The data offered to hide are from TODAY only. This avoids confusion by having too many calibrations to display, and is based on the assumption that SciOps is not interested in flagging data from yesterday or even older.
• For older data discovered to have quality issues, the second BADQUAL workflow exists which is available locally and is intended for the QC scientist only (who will never work on data from TODAY). Both dialogues are therefore complimentary: the button is for SciOps, while the local 'BADQUAL' link is intended for QC.

The dialogue is created by the php script calBadQual.php that catches the input and stores it in a text file on the web server. Upon its next execution, calChecker finds that file and stores the new entries in the local CAL_BADQUAL file, just like the ones edited by the local BADQUAL dialogue (see next paragraph). The calBadQual.php script is generated and uploaded by calChecker.

	BADQUAL dialogue, after having marked a calibration file as BAD.
	Same, after calChecker has entered ("registered") this file in the CAL_BADQUAL file. The flag is now "HIDDEN".

BADQUAL file. For calibration data older than a day, it may become obvious during certification that certain calibrations have a quality problem. If rejected by QC, they might also need to be taken away from calChecker.

calChecker has an interactive mechanism to "hide" those files. There is the local file $DFO_MON_DIR/CALCHECK/CAL_BADQUAL which is filled with the applicable DATE and the ARCFILE name(s) of those raw calibrations which should be hidden for calChecker. Any file listed here will be moved away from the header directory before createAB is called, and later moved back again (to ensure completeness). Thereby it won't be exposed to the tool. calChecker will then either find another valid calibration (this time a true 'OK', hopefully), or raise a NOK or MISS alert (which then should be analysed in the standard way).

The steps of the BADQUAL workflow can be invoked by clicking the action button [edit BADQUAL] on the local calChecker interface:

- enter the DATE and the ARCFILE name(s) in $DFO_MON_DIR/CALCHECK/CAL_BADQUAL (don't forget to add *all* raw files of a template files here)
- remove the corresponding VCAL files from $DFO_CAL_DIR/CALCHECK/VCAL (done automatically if you use the [edit BADQUAL] button).

[edit BADQUAL] workflow steps: step1 | step2 | step3 | step4

The VCAL step is important since entering ARCFILE names in BADQUAL will only prevent the creation of new "bad" VCALs, while the VCAL step will delete the existing "bad" VCALs.

These steps are also documented in the CAL_BADQUAL file itself. After some time you can safely remove outdated entries from this file. calChecker will clean entries older than 2 years.

You need to re-run the calChecker tool to make any changes visible.
- If only the current date is affected, the incremental call is sufficient. It is offered at the end of this dialogue.
- If a SCIENCE date in the past is affected, call 'calChecker -d <date>'.
- If more than one SCIENCE date is affected, or you don't know for sure, call 'calChecker -F' (full mode).

'Analyze ISSUES' workflow and interface (same for SCIENCE and CAL4CAL). Another important feature of calChecker is the analysis workflow. Whenever a new NOK, MISS or LOST case is detected, the tool displays this as a color-coded box, with the information "not yet analyzed". For analysis, the calChecker tool offers a web-based workflow which can be used by both the QC scientist and the SciOps staff. [Note that NOK and MISS entries are re-analyzed from scratch by the tool in incremental mode every time. Hence, turning a NOK/MISS case into OK in the analysis workflow not only helps with understanding the calibration situation but also improves tool performance.]

There are two possible situations for the NOK/MISS cases:

• for the current date (last science is no older than 12 hours): daytime calibrations are not yet finished, usually there is no need for any action.

  miss [daycals still ongoing]

  (applies both to MISS and NOK)
• for all other dates, or when the last science is older than 12 hours: it is useful to analyze the situation, in order to make the alert significant.
  

  miss [not yet analyzed]

  (applies both to MISS and NOK)

LOST cases are initially MISS cases which, after the night is over, turn into LOST. An analysis of a LOST case always makes sense: although the calibration issue cannot be resolved anymore, it is still useful as a reminder, or acknowledgement, about the corresponding OB grades. Possible analysis results could be "OB raded C" or "OB did not require STD". In any case, the flag should become 'OK' then since there is no calibration issue.

Click the 'analyze ISSUES' button to open the analysis interface.

Steps for analysis:

• step 1: analyze the situation. Obviously this cannot be done by the tool. Very much like in the certification procedure for pipeline products, the whole set of circumstances needs to be evaluated.
  For example:
  • A calibration is formally detected as outdated, but based on Health Check Monitor information you know that it can still be considered valid. Analysis result would be 'OK'.
  • You have configured a minimum number of 5 twilight flats, and only 4 have been taken. The tool claims MISS, but after careful analysis you decide this is OK in this particular case.
  • An OB has been graded 'C' by the night astronomer. Then there might be no need for taking daytime calibrations. Analysis result would be 'OK'.
  • OBs have been graded 'A', daytime calibration queue has been broken. If the calibrations are not taken soon, the OB has to be graded 'C' and be repeated. Analysis result would be 'MISS'.
  • A STD_IMA is missing for an OB. No matter what the OB grade is in the end, the missing calibration cannot be recovered. If the RAW_TYPE 'STD_IMA' is configured properly, the tool will mark this case as 'LOST'. While the OB grade is being reviewed, you can enter a preliminary comment like e.g. "OB to be graded C, USD informed", while not providing an analysis. The comment can be modified anytime. The final analysis can be entered anytime. The point is that the preliminary comment can be useful as a reminder for PSO to review the OB grade, while the issue is still flagged 'LOST'.
    
• step 2: enter the solution (or a preliminary comment only). Use the button 'analyze ISSUES'  and, after authentication, start the dialog (check out below for a shortcut in case of "many" similar entries). The calChecker analysis editor opens in a new window:
  
  

  ANALYSIS dialogue: interface offering NOK/MISS/LOST cases for analysis.

  
  
  It is a created by a php script (calAnalysis.php) that catches your input and stores it in a text file on the web server. Upon its next execution, calChecker finds that file, interprets the analysis result and modifies the NOK/MISS/LOST boxes accordingly, with the analysis results displayed in the ANALYSIS NOTES section. Find more about this php script and about php in general here.
• The editor is password protected. There are two possible log-ins. Ask your colleagues or send a mail.
  
• Enter the analysis cases. You can analyze multiple identical cases in one go (identical meaning: same flag, same note). The already solved cases appear on top of the page, the remaining ones underneath. If there are preliminary comments, their cases appear twice (they are shown because you have entered the comment, and they are offered for analysis).
  
  
• The php dialog is available for the main and for the CAL4CAL tabs separately. CAL4CAL has no LOST cases.
• The php dialog can be used both by the QC scientist and by the Paranal astronomers. Thereby it is guaranteed that every single night of the year calibration issues can be analyzed.
  
• The solution is a free text being a comment, or a recommendation (max. 80 characters). Examples:
  • solution: OK; comment: "science OB was graded C, no calibrations required"
  • solution: MISS; comment: "twilight flats urgently needed"
  • solution: NOK; comment: "STD was not measured in the same night; nothing can be done do repair; OB to be graded C"
  • preliminary comment: no solution (yet), but comment "STD was not measured; OB to be graded C, action pending, see PPRS001234"

  As far as possible, use the standard comments offered (how to modify ...). They are also displayed if you click on "ANALYSIS template" on the local calChecker interface.

• After confirming the analysis result, you may want to decide what Paranal should do to remedy the situation. Use the button [send mail to PSO] .
  
• The solution text is displayed after the next execution of the tool:

miss analyzed: [1]

with [1] referring to the section

ANALYSIS NOTES:
Index	mode	setup	date	flag	analysis
[1]	SCIENCE_MED	Medusa1_L682.2	2008-02-20	OK	[this is just a test]

which is displayed only in case of such notes in the current time range. A resolution NOK or MISS would display yellow or red, resp.

• overriding the automatic evaluation. The next time calChecker is called, your analysis will override the automatic evaluation. Such an overridden analysis tag is marked by a slightly paler color and has always a reference to the table with the ANALYSIS NOTES. There are two important rules about overriding:
  
  • You cannot override a green/OK evaluation by a yellow/NOK or red/MISS analysis: in those cases you must use the BADQUAL mechanism to hide the corresponding raw files, and then analyse NOK or MISS.
  • You cannot override a yellow/NOK evaluation by a red/MISS analysis. The only way of doing this is again the BADQUAL mechanism.
  • You cannot override a LOST evaluation by any other analysis than OK.

  These rules can be summarized as "you cannot downgrade an automatic evaluation". There is a fundamental logical reason for this: if you have initially a red/MISS situation (no matter if automatic or by analysis), and then Paranal reacts by taking the missing calibrations (which is the ultimate purpose of calChecker), then the tool must be able to detect this improvement, and the only way of doing so is to then ignore a MISS or NOK analysis in the CAL_ANALYSIS file. The tool detects the mismatch in CAL_ANALYSIS and comments out that line. If the automatic result is then green/OK, nothing else remains to be done. If the automatic result is yellow/NOK (which might still be better than red/MISS), the local interface again blinks and asks for an analysis.

  In essence the analysis can improve or confirm an automatic flag, but not downgrade. For downgrading you need to use the BADQUAL mechanism.
  
• The whole set of ANALYSIS_NOTES is kept in the local disk file $DFO_MON_DIR/CALCHECK/CAL_ANALYZED, linked to "ANALYSIS file" in the tool bar of your local calChecker monitor page. You could also edit this text file directly using the action button [edit ANALYSIS] and see the analysis notes being displayed.
  
• You can use the action button [send mail to PSO] to send a standardized email to the configured $PSO_EMAIL (the account of the shift leader who is responsible for dispatching the note).

Sketch of the analysis workflow

Refresh button

The 'Refresh' button is designed to force the calChecker to execute. The intention is to offer the SciOps user a possibility to renew the calChecker interface if needed, e.g. after the BADQUAL dialogue (see above).

Technically, the tool is running in the forced mode ('calChecker -I' where 'I' stands for interface [the F is already used for FULL mode]). It is implemented in the same way as the autoDaily forced mode:

• the php tool calForcedRefresh.php creates a trigger file on the web server, CAL_ENFORCE;
• the trigger file is catched by the 'dfosCron -t autoDaily' cronjob, patrolling every minute;
• the dfosCron tool starts a session of 'calChecker -I';
• if that session discovers that another instance of calChecker is already running: it terminates immediately, sending an email notification;
• if it is running, and another instance of calChecker discovers it, then the first instance continues while the second one terminates.

The tool calForcedRefresh.php is created and maintained by calChecker itself.

Sketch of the forced mode of calChecker: The user creates a request for running calChecker, using the php interface and creating a file CAL_ENFORCE. A monitoring cronjob on the muc machine discovers this "trigger file" and then launches 'calChecker -I'. Processes on the web server and on the muc machine are strictly separated. It is not possible to launch a muc process from the web server directly. Execution of the processing job on the muc server is controlled by the cronjob. The trigger file must have correct name and a defined content, otherwise it is ignored.

Technical aspects

Workspace. In order to have a clean separation of calChecker and routine DFO operations, the tool has its own workspace:

• $DFO_CONFIG_DIR/CALCHECK
• $DFO_AB_DIR/CALCHECK
• $DFO_CAL_DIR/CALCHECK
• $DFO_MON_DIR/CALCHECK
• $TMP_DIR/CALCHECK

The CAL4CAL mode splits them all by CAL4CAL:

• $DFO_CONFIG_DIR/CALCHECK/CAL4CAL
• $DFO_AB_DIR/CALCHECK/CAL4CAL
• $DFO_CAL_DIR/CALCHECK/CAL4CAL
• $DFO_MON_DIR/CALCHECK/CAL4CAL
• $TMP_DIR/CALCHECK/CAL4CAL

You do not need to care about this, the tool does it for you. Just don't be surprised to see new subdirectories.

For the incremental mode, the tool uses $TMP_DIR/CALCHECK/PREVIOUS as a subdirectory to keep the memory about previous runs (for CAL4CAL, this becomes $TMP_DIR/CALCHECK/CAL4CAL/PREVIOUS).

Analysis and php scripts:

• Find more about usage of php within dfos here.
• The open analysis cases (the ones with not yet analyzed MISS and NOK flags) are extracted from CAL_ANALYZED into the file calEditor.dat and then uploaded, to the same URL. You can check it under the above URL.
• The standard comments are downloaded by calChecker each time from the URL http://www.eso.org/qc/ALL/CC_STDCOMMENTS (see below).
• calEditor.dat contains the subset of records in CAL_ANALYZED which are not yet analyzed and offers them to the user for analysis.

You need the mandatory config key CALEDIT_PWD. Fill it with the name of the file under $DFO_BIN_DIR hosting the password (not with the password itself!). Without this key, the tool is unable to download the edited content of calEditor.dat from the web (since its URL is password protected, wget needs to log-in like any other user).

BADQUAL and php scripts:

• The list of files offered on the php interface contains calibration ABs from TODAY. Its content is managed by autoDaily (!): any new AB from the incremental calls is added to that list. New ABs can be expected only after autoDaily has found them. See why under 'operational aspects'.
  
• Any new calibrations entered as BAD on the interface are added to the $DFO_MON_DIR/CALCHECK/CAL_BADQUAL file.
• They get the flag HIDDEN once calChecker has seen them.
• In principle they can be overridden by QC. See how under 'operational aspects'.
  

Standard analysis comments: They are stored in http://www.eso.org/qc/ALL/CC_STDCOMMENTS. If you want to edit them:

• get permission from the Group Head
• login to stargate1, cd qc/ALL; change permission, edit the file, change back to read permission
• upon the next execution, calChecker will offer the edited standard comments on the analysis interface.

Export to mirror web site:

• The main pages of the SCIENCE and the CAL4CAL mode are exported to the mirror web site on Paranal, defined by the environment variable $MIRROR_WEB_SERVER. This is currently qc_garching@odyssey5.pl.eso.org. This is implemented to improve the tool performance and browser experience on Paranal.
• To initialize, call once
  calChecker -s
  and follow the workflow, to give your credentials to the odyssey5 server.
• If needed, find the replication script as follows: 'ssh $MIRROR_WEB_SERVER; cd bin'. It is called 'quality_sync.sh' and is maintained by Paranal IT.

News (similar for normal and CAL4CAL). There is the option to post notes, reminders or comments of temporary character into a text file and include them in the monitor (POSTIT function). Just click on the [edit NEWS] button and edit the file CAL_POSTIT. It is downloaded from the web server (URL is http://www.eso.org/qc/<instr>/reports/CAL/CAL_POSTIT; e.g. CRIRES: http://www.eso.org/qc/GIRAFFE/reports/CAL/CAL_POSTIT). The text is linked via 'include virtual' to the calChecker monitor.

Note that this function is intended for news and/or important information for Paranal staff; it should refer to current content only, and should be removed once it is outdated.

There is also a General news section. This is filled from the text file under http://www.eso.org/qc/ALL/CC_POSTIT. Usually it is empty, but it could be used to communicate things like "Saturday night 22-24:00 UT maintenance of Garching web server, no updates for calChecker possible". It is accessible to all QC scientists on the stargate1 server, cd qc/ALL.

Exported monitor. The tool exports the HTML output (and the association text file) to the external Health Check site (http://www.eso.org/qc/<instr>/reports/CAL/calChecker_<instr>.html). The exported version is visible to PSO and is supposed to be checked by both PSO and QC, like the Health Check monitor pages.

Information file

There is an optional (however strongly recommended) information file displayed in the HTML output. This could e.g. be used to explain the setups. This file goes to $DFO_CONFIG_DIR/CALCHECK. A similar one (same name) should be provided for CAL4CAL enabled installations.

Output

• $DFO_MON_DIR/CALCHECK/calChecker_<instr>.html, calResults_<instr>.html, and rawtypes_<instr>.html
• exported versions
• the calResults pages come in three flavours: per date (linked to the result boxed); one with all results sorted by setups (linked to 'DETAILS'); one with all results in a user-sortable and searchable format (linked to the setup page).

How to use

Type calChecker -h | -v for on-line help and version number. Type

calChecker

to create the calChecker output pages (per default in incremental mode),

calChecker -n (no scp: useful for testing)

calChecker -N (no report creation: useful for testing)

calChecker -D <N_VCAL_LIST> (non-standard number of days, for testing only)

calChecker -F(ull): full execution (otherwise incremental execution), including reconstruction of the calibration memory.

calChecker -C(AL4CAL): run CAL4CAL mode only

calChecker -s: initialize scp to Paranal mirror site odyssey5.pl.eso.org (ssh key transfer)

Configuration files

The tool has

• its own config file config.calChecker
• a config file for long-term calibrations called long_<instr>_classification.h
• a set of OCA configuration files

1. The tool configuration config.calChecker goes to $DFO_CONFIG_DIR/CALCHECK and defines:

2. The long-term calibrations config file goes to $DFO_CONFIG_DIR/CALCHECK/OCA. It has the following entries:

3. The OCA configuration files reside under $DFO_CONFIG_DIR/CALCHECK/OCA. You take care of the usual three files, <instr>_association.h / _classification.h / _organisation.h . All others are either created by the tool on the fly, or copied from your operational $DFO_CONFIG_DIR/OCA. As mentioned above, while editing these three OCA files please keep in mind that they create ABs which are read for association information only. They are never used for processing, nor for packing, and they are not stored permanently anywhere.

Quick links

The quick links on top of the vertical navigation bar link to the most relevant sites for Paranal QC. "HC", "refs" and "QC" go to general pages from where you click to the selected instrument. The "HC" link in the navigation pane of "analyze ISSUES" directs to the first HC report of the respective instrument, for easy cross-navigation.

Operational aspects

• The tool can be called from the command line, by using the 'refresh' button on the GUI, or by a cronjob.
• Operationally, you must have a cronjob to execute this tool. Current operational agreement with PSO is N30D30: every 30 minutes during day and night, 24/7.
• In addition, call 'calChecker -F' once a day in the cronjob file. This provides you with a complete reconstruction of all calChecker information (if corrupted), and a clean restart (if hanging). Check out here for more.
• The tool can be refreshed anytime with the 'refresh' button. In case of conflicts, the already running instance is given precedence, while the second one is terminated.
• The list of ABs offered for the BADQUAL dialogue is managed by autoDaily and not by calChecker. Here are the reasons why:
  - the amount of calChecker ABs is a subset of the DFOS_OPS ABs;
  - we (and SCIOPS) cannot easily say which of the DFOS_OPS ABs is relevant for CALCHECKER and which not;
  - therefore we must offer the full set of ABs for the BADQUAL workflow, and this is generated by autoDaily.
• The BADQUAL dialogue is designed around the use case of flagging calibration issues for TODAY's data; hence it might become necessary during certification (i.e. one or more days later) that the BADQUAL flag is not confirmed by QC and needs to be overridden. In that case:
  - edit the $DFO_MON_DIR/CAL_BADQUAL file (via the dialog on the local calChecker interface) and comment out the file(s) affected by the AB you want to unhide;
  - call calChecker again, or wait for the cronjob execution;
  - you will get an email confirmation that the file was unhidden and is evaluated again by calChecker.
• If you change your OCA configuration, the fundamental assumption for incremental mode "an OK result will be OK again when new calibration data are added" is not valid anymore. Then you should always enforce a full execution (calChecker -F).
  
• You include the output page in the standard DFOS monitor navigation bar (recommended: on top), by adding it to the navigation bar configuration option on the dfoMonitor .
  
• The calChecker calib memory is incrementally built. While this is good for performance, it may result in (partial or complete) loss of content in case of unusual events (e.g. NGAS problems). In that case, the calib memory can be rebuilt by running calChecker once with N_VCAL_LIST set to the same value as N_VCAL_MEM.
• You can (and should) monitor execution times under http://www.eso.org/qc/WISQ/exectime.html.
  
• Use the button [send mail to PSO] to send a standardized email to <telescope>@eso.org (the account of the shift leader who is responsible for dispatching the note).
• All science files with runID starting with 60. or 060. or 0060., or with runID = "maintenance", are suppressed. For special cases (instrument under commissioning) you can waive that restriction by setting the optional key ENABLE_60 to YES. For complex cases you can create a short shell script that contains an additional filter condition and call it as $PGI_FILTER.

Workflow

Details:

1. prepare configuration (create or copy required config files); check for older instances
2. update headers for current date (HdrDownloader), update data report and nightlog
3. create CALIB ABs and populate $DFO_CAL_DIR/CALCHECK/VCAL
   INCREM=YES: only for TODAY
   INCREM=NO: for all dates defined by $N_VCAL_DATE
   
   
4. if CHECK_C4C=YES: call CAL4CAL mode
   4.1 create triggering CALIB ABs (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
   4.2 get OB grades and comments
   4.3 create result pages (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
   4.4 check for technical/maintenance calibrations
   4.5 check for beocal signal
   4.6 check completeness and create main page
   4.7 scp CAL4CAL result pages to external web sites and to Paranal mirror web site
   
   if CHECK_C4C=NO: skip this step
   
   
5. create ABs for SCIENCE (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
6. get OB grades and comments
7. create result pages (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
8. check for technical/maintenance calibrations
9. check for beocal signal
10. check completeness and create main page
11. scp result pages to external web site and to Paranal mirror web site