Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
 

v4.6:
- BADQUAL dialogue
- forced refresh button
- check for other instance

- output of this tool is operationally relevant for Paranal QC
- tellTracker
- the tool uses OCA syntax
- define your CALCHECKER calib map using createCalibMap
 

v4.6.1:
- optional PGI_FILTER

v4.6.2:
- enabled for P100 prog_Ids

[ used databases ] databases observations..calibration_queues for B/EOCAL signal
[ used dfos tools ] dfos tools called createAB; getObInfo; output linked to tellTracker
[ output used by ] output HTML output under $DFO_MON_DIR/CALCHECK
[ output used by ] upload/download upload: HTML files exported to ESO web server and QC web server
javascript jquery-latest.js, jquery.datatables.js under ${JSCRIPT_URL}
topics: description | CAL4CAL and tellTracker | strategy | incremental execution | memory | full | repair | cal_scores | certification | b/eocal
calibration configuration: OCA | long-term calibrations | nighttime || interfaces: monitor (main page) | details
workflows: BADQUAL | analysis | forced refresh || technical: mirror web site | news section || tool configuration | quick links | operations | workflow
You need to initialize the scp connection to the Paranal mirror server odyssey5.pl.eso.org:
- call 'calChecker -s'
- follow the instructions
- make sure to not have 'BatchMode=yes' in your ~/.ssh/config file
If all went fine, you don't need this anymore. Find information about the replication script here.

calChecker

You may also want to read the information linked to the public result page (here) and the general calChecker home page.

[ top ] Description & concepts

This tool monitors the Paranal calibration plan. It detects missing calibrations and provides feedback to QC Garching as well as to SciOps Paranal.

It scans the header files of the last 7 nights. It reads all headers of CALIB files in the defined period, classifies them and creates headers of virtual calibration products. These headers define the pool of available calibration files.

Then, in a second step, it reads the headers of all SCIENCE files and creates association blocks. The SCIENCE ABs (actually the ALOG files which are derived from them) are then scanned for completeness and time matches, in the usual createAB way.

The third step involves an evaluation of the created SCIENCE ABs. These ABs are not evaluated one by one, but per "setup box" (a single cell in the main interface). A setup box is defined by a datatype, setup and date. Each setup box is analyzed as:

After creation, ABs are evaluated here only in terms of setups. If e.g. a sequence of 50 science files has been measured, all in the same setup, then all science ABs will be OK, in terms of association, if one is OK.

Usually, NOK and MISS setups can be properly calibrated within an acceptable time window if PSO is informed properly and gets a chance to acquire the missing calibrations. To detect those cases is the main purpose of this tool.

LOST cases cannot be recovered, but that information can be used by PSO to review the OB grades. When the current night is over (UT > 10:00), or for older dates than $TODAY, a MISS calibration configured in the RAWFILE_NIGHT section causes a LOST case.

In case of hybrid cases, a MISS+NOK case is flagged as MISS. A MISS+LOST is flagged as MISS, a NOK+LOST case is flagged as NOK.

[ top ] There is also the option to provide a "preliminary comment" which is just a comment without analysis. This could be useful in case of

[ top ] CAL4CAL mode. This is a mode which calls a separate set of configuration files and checks calibrations depending on other calibrations (as opposed to calibrations depending on science data as in the classical case). For example, a standard star calibration (also called the triggering calibration) needs a set of DARK and FLAT calibrations (dependent calibrations). These dependent calibrations are not necessarily identical to the ones needed for science, and there might also be nights when no science in these setups have been taken but only the triggering calibration (like the standard star). In those cases one might want to ensure that the triggering calibration can safely be calibrated, like the science data in the classical case.

The configuration, workflow and results are very similar to the science case. The CAL4CAL mode is called within the normal calChecker call. It must be enabled in the configuration, by default it is disabled, to account for those instruments having no need for CAL4CAL checks. If enabled, the results are displayed on a second page being linked to the first one. Find more about CAL4CAL configuration here. The CAL4CAL result page has the same functionalities (action buttons) as the main result page (e.g. ANALYSIS, BADQUAL).

The CAL4CAL mode has no LOST cases, but the preliminary comments can be used as well.

tellTracker. For some instruments, a link to a third output page is offered, the tellTracker. The link is enabled by setting the optional configuration key CHECK_TELLTRACK to YES. tellTracker is called stand-alone, with higher cadence than calChecker and follows time-critical night calibrations like telluric standard stars. If there is an issue with the current last night, a flag ! is displayed. Check out here for more.

Links to RAW file screenshots and calibration product quality. The tool provides easy access to RAW file screenshots (as created by the python tool qc_rawdisp.py) and to the exported AB product monitor pages. However, these pages are not created or maintained by calChecker itself, but by autoDaily and getStatusAB in incremental processing mode.

[ top ] Strategy. The strategy of the tool is to follow the behaviour of the calOBbuilder tool used on Paranal to define the daytime calibration queue. It goes beyond that in the sense that also nighttime calibrations (standard stars) and twilight calibrations (sky flats) can be included in the configuration. These types of calibrations are often subject to be overlooked by daytime staff on Paranal. Also, it can monitor long-term calibrations which are not triggered by science exposures at all. It also checks CAL4CAL data (calibrations for calibrations).

In short:
Included:
all daytime calibrations defined in the calibration plan as driven by the science observations (e.g. dome flats)
all twilight calibrations defined as science-driven (e.g. twilight flats)
all nighttime calibrations defined as science-driven (standard stars; configured if they qualify for LOST case)
long-term calibrations (maintenance, technical, health checks)
all calibrations triggered by other calibrations (daytime, twilight, nighttime)
Excluded:
optional attached calibrations (they cannot be detected as missing or ok!)
short-term calibrations (like telluric standards to be taken within time and airmass constraints; these are monitored with tellTracker).


[ top ] Incremental execution

Scanning 7 nights twice (CAL4CAL and SCIENCE) costs performance. Although the calChecker results are analyzed and displayed per setup, ABs need to be created in the first place. Hence the tool performance scales with the number of ABs, actually with the number of raw files (headers). There are cases when a full calChecker execution would take more than half an hour which is fatal for automatic cronjob execution.

For all dates scanned so far for which the calChecker result boxes are green (OK), a repeated scan is normally obsolete (unless the BADQUAL mechanism is used, see below). Therefore an incremental scanning strategy is usually fully sufficient. The following rules are implemented:

The following table displays a set of typical OK/NOK cases and explains their origins.
DATE*:     [?]
[color if science data acquired]
2010-09-14
SM
71
report | NLT
2010-09-15
SM
320
report | NLT
2010-09-16
SM
15
report | NLT
2010-09-17
SM
97
report | NLT
2010-09-18
SM
5
report | NLT
2010-09-19

report | NLT
2010-09-20
SM 13
report | NLT
r...
Raw CAL displays:     [?]
raw raw raw raw raw raw raw
P...
Product CAL quality:     [?]
products products products products ! products products products
Data types (Mode): Setup:  
SCIENCE_IMG_BROAD ND_10_H  
 
 
nok
ok
 
 
ND_10_J ok
LOST
 
ok
ok
 
 
ND_114_Y ok
 
ok
 
ok
 
nok
[daycals not yet started]  
ND_5_Ks  
 
nok
 
 
 
nok
[daycals not yet started]  
SCIENCE_IMG_NARROW ND_10_CH4  
 
 
miss
ok
analyzed: [1]
 
 
ND_20_H2 ok
 
 
 
 
 
 
ND_20_NB2090 ok
 
 
 
 
 
 
 
checked in incremental mode? CALIB NO
why? 1*
NO
1
NO
1
NO
1
NO
1
NO
1
YES
1
  SCIENCE NO
why? 2*
NO
7
YES
3
YES
3
NO
4
NO
5
YES
6
*Why?
1 - CALIBs scanned only for current (last) date (other dates are already complete in VCALs)
2 - SCIENCE not scanned if all setups OK for that date (nothing to improve)
3 - SCIENCE scanned again if a setup returns NOK or MISS for that date
4 - same as 2, original NOK/MISS result overridden by OK analysis (nothing to improve)
5 - no scan if no SCIENCE data
6 - SCIENCE always scanned for current (last) date
7 - SCIENCE not scanned since the calibration situation cannot change anymore, but the result table is scanned to catch if the analysis turns into 'OK'

ANALYSIS results entered through the ANALYSIS workflow (see below) are taken into account:

BADQUAL results entered through the BADQUAL workflow (see below) are taken into account:

Long-term calibrations are always scanned completely.

[ top ] Memory about open issues. If an open issue exists (NOK, MISS; not: LOST!) that is older than 7 days, it is kept in the calChecker result table for as long as that date falls within the total time range defined by N_VCAL_MEM. The output design is slightly different from the main date range: the colors are pale and the font is smaller, this to indicate that the memory time range is not displayed completely but only for the dates having an issue. Issues in the memory range can be analyzed. As soon as all calibration issues for a date in the memory are solved, that date disappears from the output. This handling applies to both 'science' and 'cal4cal' parts of the calChecker, independently. Issues in the memory range contribute to the calibration scores (see below) just like the normal ones.

[ top ] Full execution. A full execution is necessary when the OCA configuration has been modified. Also it might happen that the tool has been screwed up, or the results are suspicious.

A full execution is possible with the option -F. In mode -F, the full calibration memory is reconstructed, then all CAL4CAL and SCIENCE dates are scanned. This mode is enabled to kill all other running instances.

Operationally, it is required to call 'calChecker -F' once a day in the cronjob file (this is also checked by the dfoMonitor). This provides you with a complete reconstruction of all calChecker information (if corrupted), and a clean restart (if hanging). The suggested pattern is:
# calChecker: every half hour; plus once a day FULL:
10,40 * * * * . .bashrc; export TZ=Europe/Berlin; $HOME/bin/calChecker > /tmp/giraffe/CALCHECK/cron.log
41 18 * * * . .bashrc; export TZ=Europe/Berlin; $HOME/bin/calChecker -F > /tmp/giraffe/CALCHECK/cron.log

Important: 10,40 is your choice; the time of the FULL mode should always be slightly later than a normal call (41 in this case), to ensure that the FULL mode correctly kills any previous instance. If you do it the other way round (like 39), the incremental instance will crash into the FULL instance one minute later, with unwanted results.

Call the FULL refresh after the end of daytime calibrations, before the new night starts. A reasonable choice seems 18 UT, as in the example.

[ top ] Repair a date in incremental mode. It might happen that a certain date has inconsistent entries. For instance, headers got lost. To satisfy the need to reconstruct the results for this particular date, without running the full refresh, there is the 'repair' mode of calChecker. (In practise, running the full mode may easily cost half an hour or more and bear the risk that a cronjob-triggered instance starts and gets corrupted.) Calling 'calChecker -d 2018-09-15' for the above scenario will add the specified date to the list of dates to be checked. Technically, this date is removed from the list of OK dates. The repair mode is always applied to the calibrations plus the science data, and both for the regular and the CAL4CAL mode. It can only be executed interactively.

Important: if you change your OCA configuration, the above assumption (an OK result will be OK again when new calibration data are added) is not valid anymore. Then you should always enforce a full execution (option -F).
The execution mode (incremental or full) is indicated at the very bottom right of the main result page: increm  full

[ top ] Calibration scores. Inspired by the HC monitor, the calChecker monitor also has scores. They are not related to product quality but to completeness. They are called 'cal_scores'. They are intended to make it easier to see an issue on a 'hidden' calChecker page (e.g. on cal4cal while you scan 'science'). The following values exist:

all calibrations existing and within validity time range
issue with outdated or missing calibrations but daytime calibs not yet finished
issue with outdated or missing calibrations, daytime queue finished

The cal_scores are determined for the SCIENCE and the CAL4CAL mode independently. They are exported to the URL www.eso.org/qc/<instr>/reports/CAL from where they are embedded in the overview page. (For the cal_scores, a LOST issue is handled the same way as a MISS issue.)

[ top ] Certification. The links labelled "Products quality" are marked products if the CALIBs for that date have already been certified by QCG (to indicate to PSO that the product quality has been checked and issues been raised). Those nights which are not yet certified are marked products . An exclamation mark ! marks nights with data coming in unsupported modes, to highlight the necessity of visual checks of raw data. See the above example output page.

[ top ] Begin/end of calibration signal (B/EOCAL). The begin (BOCAL) and end (EOCAL) of calibration signals are read from the database. These signals are inserted (with timestamp, per instrument) upon start and end of the daytime calibration queue. This information is used to improve the feedback for the last date. There are three values: pending; ongoing; finished. Only upon finishing the daytime calibration queue, a NOK/MISS flag becomes relevant. These flags can also be used to detect a broken/aborted daytime queue.

There is the special case of UVES and FLAMES/UVES: the BEOCAL signals come twice, once for UVES and once for FLAMES. calChecker evaluates them following these rules:

This special rule set is needed to avoid false "calibrations missing" signals for FLAMES modes.


Calibration configuration

[ top ] OCA configuration. The tool uses createAB, ABbuilder and a dedicated OCA configuration. The OCA configuration files used for the calChecker are a subset of the operational files <instr>_organisation.h/association.h/classification.h. They go to $DFO_CONFIG_DIR/CALCHECK/OCA. They are simplified copies, simplified in the following sense:

For CAL4CAL, you need a second set of OCA configuration files. They go to $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA. See more here.

A completeness check for SCIENCE files is added (provided by createAB). A comparison is made between an archive query, having all science files, and a list extracted from all created science ABs. Any difference, except for the current date, is reported in an email, as to protect against undetected OCA configuration mistakes. In exceptional cases this test can be turned off: if raw files are artifically split into raw files per detector (applies for VIMOS and FORS2 only: pseudo-MEF), the calChecker OCA configuration will legally apply to one of the possible detectors only. Then a completeness check would constantly create false alerts and can be turned off with SUPPRESS_ABCHECK=YES.

Calibration cascade for normal (science) mode of calChecker. Completeness of calibrations for the triggering science data types and setups is checked.

N_VCAL_LIST: By agreement with PSO and QC, the operational use of this OCA parameter is 7 days. It cannot be configured but is hard-coded. Smaller or larger ranges can be used for testing (call option -D <n>). They cause a warning and an email. Much smaller ranges may result in inconsistencies (headers and nightlogs are not delivered instantaneously; the daytime calibrations take some time; the calibration plan may foresee validity ranges larger than 24 hours). Larger ranges cost performance and don't make sense operationally, but they might be useful for testing, e.g. in order to catch more instrument modes and check proper configuration. For a complete refresh of all 7 days, call option -F; this option will also refresh the entire calibration memory which is either configured as N_VCAL_MEM or set by default to 2*7 days.

In addition to the user-defined OCA files, the tool creates its own config.createAB on the fly and copies all other standard configuration files (like macro.h) from the operational OCA directory.

Stripping down the configuration to the "essential cascade" pays off in terms of performance: in FULL mode the tool needs to create ABs for 2*N nights where N is the configured N_VCAL_LIST and the factor 2 comes from running it in modes CALIB and SCIENCE.

[ top ] OCA CAL4CAL configuration: there is a dedicated set of CAL4CAL configuration files. They come in OCA style and syntax and reside under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA. Let's assume an example with STD as the triggering calibration, and DARK and FLAT as depending calibrations.

1. OCA files. Best strategy is to start with the OCA set already operational for calChecker, copy the three OCA files from $DFO_CONFIG_DIR/CALCHECK/OCA to $DFO_CONFIG_DIR/CALCHECK/CAL4CAL/OCA and then trim and edit them. Use the same setup key names, rule names, macros etc. as in the classical calChecker OCA rules.

All rules as listed above for the SCIENCE checks apply here, plus the following:

2. config.calChecker: edit the one under $DFO_CONFIG_DIR/CALCHECK to include the key CHECK_C4C (see here). Then the CAL4CAL mode is enabled. The file with the same name under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL is an auto-copy and needs no editing.

3. info.calChecker: provide a file with this name and the proper content as an edited copy under $DFO_CONFIG_DIR/CALCHECK/CAL4CAL.

All other files are simple copies of the corresponding files under $DFO_CONFIG_DIR/CALCHECK/OCA and are auto-copied by calChecker.

Calibration cascade for CAL4CAL mode of calChecker. Completeness of CAL4CAL calibrations (the ones for the triggering calibration data types and setups) is checked.

[ top ] Long-term calibrations. The tool can monitor these calibrations which can be Health Check calibrations or maintenance/monitoring calibrations. Their defining property is not necessarily the longer timescale but the fact that they are not triggered by science observations. For performance reasons, these types of calibrations are searched for not by OCA technology, but by pattern matching in the daily raw file reports. If e.g. the last set of detector monitoring flats is to be monitored, the tool checks for the pattern "LAMP,FLAT,DETCHECK" and maybe also for a certain template name, a minimum number of exposures, a specific exposure time etc. Of course all strings of the pattern must be included in the report configuration (see the createReport documentation) and must be unique.

They are monitored on the calChecker in the following way:

All cases (if any) which flag yellow or red are displayed in the summary table on the main calChecker page. The complete overview is available behind the link "complete overview here". As an example, you may want to inspect the GIRAFFE case. It lists:

The long-term calibrations have their own configuration file, see here.

There is the link how to execute to a page containing information for the Paranal astronomers how to execute the corresponding templates. The content of that page is provided by the instrument scientists, on a PSO wiki page.

[ top ] Configuration of nighttime calibrations for the 'LOST' case. Certain data types cannot be taken after a night is over, e.g. photometric standard stars. As a rule of thumb, this applies to those calibrations with a validity of 0.5 or less ("same night"). They need to be configured (section 2, RAWFILE_NIGHT) since not all standard stars fall into that regime (e.g. the ones for spectroscopic modes are typically optional or have a longer validity).


CALCHECKER monitor

[ top ] Main output page. The tool produces an HTML page, the calChecker monitor. This monitor (see here an exported example) has a result table displaying, per setup and date, the association flags (OK/NOK/MISS/LOST). The standard time depth is 7 days.

The tool also creates a simplified, user-friendly version of the association rules. This is accessible under "ASSOC-RULES". For investigation, it also links to an HTML page with details (file by file).

On top of that page, there is update information and a link to the Data Transfer Monitor.

The science RAW_TYPEs are linked to the calib maps.

For CAL4CAL enabled installations, two main output pages exist: the normal page for the SCIENCE data, and the CAL4CAL page for the CAL4CAL data. Both pages have the same format. They are linked and you can switch forth and back:

 science  .   cal4cal   
DATE*:   [?]
[color if science data acquired]
2011-01-03
SM
54
report | NLT
2011-01-04
SM
128
report | NLT
2011-01-05
SM
224
report | NLT
2011-01-06
SM
250
report | NLT
2011-01-07
SM
162
report | NLT
2011-01-08
VM
46
report | NLT
2011-01-09
VM
48
report | NLT

On installations with CAL4CAL disabled, there is a blind link only:

  science  .  no cal4cal   
DATE*:   [?]
[color if science data acquired]

The tellTracker-enabled instruments have a third link:

 science  .  cal4cal  .  tellTracker 
  

Reports. The main page has links to data reports ("report") and nightlogs ("NLT"), to the RAW file displays ("raw"), and to the AB monitor ("products"). This link implements the feedback about calibration data quality. See more here. The product quality pages (AB monitor) are always linked to the ABs, to the logs and to score information.

The CAL4CAL output table looks the same, it lists the triggering calibration data types instead of the science data types.

[ top ] Detailed overview (similar for normal and CAL4CAL).

The evaluation labels (ok/nok/miss) link to a result table (sorted per date) with details useful for understanding and analysing a problem. This page lists all files affected, with details about run_id, OB id, comments and grades. It is a sortable table, per default sorted by DATA_TYPE, SETUP and RAW FILE:
Calibration report for all science files, for date 2008-06-06
This is the detailed calChecker report about the calibrations for all science OBs from the indicated date. Note: all science data with PROG_ID starting with 60. or 060. or 0060. are ignored.
DATE PROG_ID MODE OBS_ID GRADE OB Comment (first) RAW FILE DATA TYPE SETUP   CALIBRATIONS
2008-06-06 381.D-0329(B) SM 307629 C 01:49: stopped after 2400 secs fue to a robot problem. OB must be repeated GIRAF.2008-06-07T01:50:28.491.fits SCIENCE_MED Medusa1_H875.7 BIAS 0.44 FLAT 0.47 WAVE -0.83
2008-06-06 081.D-0286(A) SM 306951 A 05:09: Guide probe seeing always under constraints GIRAF.2008-06-07T05:13:25.230.fits SCIENCE_MED Medusa1_H875.7 BIAS 0.24 FLAT 0.27 WAVE -0.91
2008-06-06 081.D-0253(A) SM 310094 A GIRAF.2008-06-07T06:40:57.467.fits SCIENCE_MED Medusa1_H651.5B BIAS 0.10 FLAT 0.12 WAVE 0.15
2008-06-06 081.D-0253(A) SM 310095 A GIRAF.2008-06-07T08:01:59.264.fits SCIENCE_MED Medusa1_H651.5B BIAS 0.054 FLAT 0.04 WAVE 0.04


Special workflows
Action buttons for calChecker interface.

The calChecker monitor has a button and link panel. The leftmost button 'HC' links to the HC monitor. There are three actions buttons that can be used externally. They link to special workflows that are supported by php scripts.

[ top ] 'Mark BAD QUALITY' mechanisms. The calChecker tool checks for the formal availability of calibrations but not for their quality. As an example, flats may be saturated, or an arclamp exposure may have had the lamp turned off. It might be necessary to remove those calibration files from the pool visible to calChecker in order to flag a quality issue along with the need to take better data. This is supported in two way: with the BADQUAL button (for SciOps), and with the local BADQUAL file (for QC).

BADQUAL button. This button opens a php-based dialogue that is primarily intended for SciOps on Paranal. This dialogue offers all calibrations of TODAY (only!), for being marked as having BAD quality. The interface asks for the email address (for notification about the successful handling), and then flags the marked AB as 'BAD'. You can mark multiple ABs in one go. You can call the interface multiple times.

With the marking as 'BAD', these ABs are not yet properly considered on the calChecker interface. This happens during the next calChecker execution (either scheduled by the cronjob, or triggered by the 'forced execution' button). Then these ABs are marked as 'HIDDEN', they are not visible for calChecker anymore and will likely set a NOK/MISS issue and eventually be taken again, under good conditions.

Important to know:

The dialogue is created by the php script calBadQual.php that catches the input and stores it in a text file on the web server. Upon its next execution, calChecker finds that file and stores the new entries in the local CAL_BADQUAL file, just like the ones edited by the local BADQUAL dialogue (see next paragraph). The calBadQual.php script is generated and uploaded by calChecker.

BADQUAL dialogue, after having marked a calibration file as BAD.
Same, after calChecker has entered ("registered") this file in the CAL_BADQUAL file. The flag is now "HIDDEN".

BADQUAL file. For calibration data older than a day, it may become obvious during certification that certain calibrations have a quality problem. If rejected by QC, they might also need to be taken away from calChecker.

calChecker has an interactive mechanism to "hide" those files. There is the local file $DFO_MON_DIR/CALCHECK/CAL_BADQUAL which is filled with the applicable DATE and the ARCFILE name(s) of those raw calibrations which should be hidden for calChecker. Any file listed here will be moved away from the header directory before createAB is called, and later moved back again (to ensure completeness). Thereby it won't be exposed to the tool. calChecker will then either find another valid calibration (this time a true 'OK', hopefully), or raise a NOK or MISS alert (which then should be analysed in the standard way).

The steps of the BADQUAL workflow can be invoked by clicking the action button [edit BADQUAL] on the local calChecker interface:

- enter the DATE and the ARCFILE name(s) in $DFO_MON_DIR/CALCHECK/CAL_BADQUAL (don't forget to add *all* raw files of a template files here)
- remove the corresponding VCAL files from $DFO_CAL_DIR/CALCHECK/VCAL (done automatically if you use the [edit BADQUAL] button).

[edit BADQUAL] workflow steps: step1 | step2 | step3 | step4

The VCAL step is important since entering ARCFILE names in BADQUAL will only prevent the creation of new "bad" VCALs, while the VCAL step will delete the existing "bad" VCALs.

These steps are also documented in the CAL_BADQUAL file itself. After some time you can safely remove outdated entries from this file. calChecker will clean entries older than 2 years.

You need to re-run the calChecker tool to make any changes visible.
- If only the current date is affected, the incremental call is sufficient. It is offered at the end of this dialogue.
- If a SCIENCE date in the past is affected, call 'calChecker -d <date>'.
- If more than one SCIENCE date is affected, or you don't know for sure, call 'calChecker -F' (full mode).


[ top ] 'Analyze ISSUES' workflow and interface (same for SCIENCE and CAL4CAL). Another important feature of calChecker is the analysis workflow. Whenever a new NOK, MISS or LOST case is detected, the tool displays this as a color-coded box, with the information "not yet analyzed". For analysis, the calChecker tool offers a web-based workflow which can be used by both the QC scientist and the SciOps staff. [Note that NOK and MISS entries are re-analyzed from scratch by the tool in incremental mode every time. Hence, turning a NOK/MISS case into OK in the analysis workflow not only helps with understanding the calibration situation but also improves tool performance.]

There are two possible situations for the NOK/MISS cases:

LOST cases are initially MISS cases which, after the night is over, turn into LOST. An analysis of a LOST case always makes sense: although the calibration issue cannot be resolved anymore, it is still useful as a reminder, or acknowledgement, about the corresponding OB grades. Possible analysis results could be "OB raded C" or "OB did not require STD". In any case, the flag should become 'OK' then since there is no calibration issue.

Click the 'analyze ISSUES' button to open the analysis interface.

Steps for analysis:

with [1] referring to the section

which is displayed only in case of such notes in the current time range. A resolution NOK or MISS would display yellow or red, resp.

 

Sketch of the analysis workflow

[ top ] Refresh button

The 'Refresh' button is designed to force the calChecker to execute. The intention is to offer the SciOps user a possibility to renew the calChecker interface if needed, e.g. after the BADQUAL dialogue (see above).

Technically, the tool is running in the forced mode ('calChecker -I' where 'I' stands for interface [the F is already used for FULL mode]). It is implemented in the same way as the autoDaily forced mode:

The tool calForcedRefresh.php is created and maintained by calChecker itself.

Sketch of the forced mode of calChecker: The user creates a request for running calChecker, using the php interface and creating a file CAL_ENFORCE. A monitoring cronjob on the muc machine discovers this "trigger file" and then launches 'calChecker -I'. Processes on the web server and on the muc machine are strictly separated. It is not possible to launch a muc process from the web server directly. Execution of the processing job on the muc server is controlled by the cronjob. The trigger file must have correct name and a defined content, otherwise it is ignored.

 


Technical aspects

[ top ] Workspace. In order to have a clean separation of calChecker and routine DFO operations, the tool has its own workspace:

The CAL4CAL mode splits them all by CAL4CAL:

You do not need to care about this, the tool does it for you. Just don't be surprised to see new subdirectories.

For the incremental mode, the tool uses $TMP_DIR/CALCHECK/PREVIOUS as a subdirectory to keep the memory about previous runs (for CAL4CAL, this becomes $TMP_DIR/CALCHECK/CAL4CAL/PREVIOUS).

[ top ] Analysis and php scripts:

You need the mandatory config key CALEDIT_PWD. Fill it with the name of the file under $DFO_BIN_DIR hosting the password (not with the password itself!). Without this key, the tool is unable to download the edited content of calEditor.dat from the web (since its URL is password protected, wget needs to log-in like any other user).

BADQUAL and php scripts:

[ top ] Standard analysis comments: They are stored in http://www.eso.org/qc/ALL/CC_STDCOMMENTS. If you want to edit them:

[ top ] Export to mirror web site:


[ top ] News (similar for normal and CAL4CAL). There is the option to post notes, reminders or comments of temporary character into a text file and include them in the monitor (POSTIT function). Just click on the [edit NEWS] button and edit the file CAL_POSTIT. It is downloaded from the web server (URL is http://www.eso.org/qc/<instr>/reports/CAL/CAL_POSTIT; e.g. CRIRES: http://www.eso.org/qc/GIRAFFE/reports/CAL/CAL_POSTIT). The text is linked via 'include virtual' to the calChecker monitor.

Note that this function is intended for news and/or important information for Paranal staff; it should refer to current content only, and should be removed once it is outdated.

There is also a General news section. This is filled from the text file under http://www.eso.org/qc/ALL/CC_POSTIT. Usually it is empty, but it could be used to communicate things like "Saturday night 22-24:00 UT maintenance of Garching web server, no updates for calChecker possible". It is accessible to all QC scientists on the stargate1 server, cd qc/ALL.

Exported monitor. The tool exports the HTML output (and the association text file) to the external Health Check site (http://www.eso.org/qc/<instr>/reports/CAL/calChecker_<instr>.html). The exported version is visible to PSO and is supposed to be checked by both PSO and QC, like the Health Check monitor pages.

Information file

There is an optional (however strongly recommended) information file displayed in the HTML output. This could e.g. be used to explain the setups. This file goes to $DFO_CONFIG_DIR/CALCHECK. A similar one (same name) should be provided for CAL4CAL enabled installations.


Output

How to use

Type calChecker -h | -v for on-line help and version number. Type

calChecker

to create the calChecker output pages (per default in incremental mode),

calChecker -n (no scp: useful for testing)

calChecker -N (no report creation: useful for testing)

calChecker -D <N_VCAL_LIST> (non-standard number of days, for testing only)

calChecker -F(ull): full execution (otherwise incremental execution), including reconstruction of the calibration memory.

calChecker -C(AL4CAL): run CAL4CAL mode only

calChecker -s: initialize scp to Paranal mirror site odyssey5.pl.eso.org (ssh key transfer)

[ top ] Configuration files

The tool has

1. The tool configuration config.calChecker goes to $DFO_CONFIG_DIR/CALCHECK and defines:

Section 1: general parameters
N_VCAL_MEM e.g. 20 optional;
number of nights to be kept in virtual calib memory ($DFO_CAL_DIR/VCAL)
should be >= 7; default: 2*N_VCAL_LIST = 14
INFO_NAME
info.calChecker name of info file, the content of which is added on the HTML output page
XTERM_GEOM
120x25+900+00 size of xterm windows being launched by browser
PSO_EMAIL <telescope>@eso.org functional email for PSO contact (e.g. kueyen@eso.org or vlti@eso.org)

CHECK_C4C

YES | NO YES: check completeness of calibs for calibs (default: NO)

CHECK_TELLTRACK

YES | NO YES: link to tellTracker enabled (default: NO)
CALEDIT_PWD .qbttxe7 name of file in $DFO_BIN_DIR hosting the password for calEditor.dat (forgotten?)
SUPPRESS_ABCHECK NO YES|NO: if YES, completeness check is skipped in createAB (optional; default: NO)
YES makes sense for FORS2 and VIMOS only
ENABLE_60 NO YES|NO: always NO for operational instruments; YES: accept SCIENCE files with 60. run_ID, for instr. under commissioning
PGI_FILTER pgi_calChecker optional pgi in $DFO_BIN_DIR; no arguments; called in sect. 5.4.2 as additional filter against pseudo-SCIENCE files; make sure it exists, and it is executable; must contain executable command string like `dfits *hdr | fitsort -d dpr.type | grep -v "dummy_science" | awk '{print $2}' | wc -l`

Section 2: Expired calibrations for LOST
List here those raw file(s) which cannot be calibrated after the night has expired (e.g. standard stars). If missing, these cases generate a LOST flag rather than a calibration issue.

RAWFILE_NIGHT STD_IMA there can be multiple entries

Section 3. Make output nicer (optional section)
3.1 Simplify setup values in the HTML display


This is an option to simplify setup strings, basically to increase readability
e.g.: SPECTRUM,NODDING_SW_LR_1.6500000_slit_0.8 can be simplified to SPECTRUM_NOD_SW_LR_1.65_slit1, without loosing information.
You can have multiple entries; they are all piped and applied together to each setup string. Make sure however that the result is still unique!

MODIF_SETUP

&&sed "s/1.6500000/1.65/"&& example to shorten a setup string; the last two characters line must be &&

3.2 Define explicit sorting for mode (RAW_TYPE) values

By default, the tool will sort RAW_TYPEs like in <instr>_classification.h; here you could define a better sorting, e.g. "by importance" or "by increasing complexity"

You need to list all possible values, non-listed ones will not display!

SORTING
SCIENCE_MED  
SORTING SCIENCE_IFU  
SORTING SCIENCE_ARG etc.

2. The long-term calibrations config file goes to $DFO_CONFIG_DIR/CALCHECK/OCA. It has the following entries:

Classification of technical calibrations
name DETECTOR_MON free tag to define type of calibration (single word, must be unique!)
VAL
30 validity in days, follows the definition in the calibration plan
rule
CALIB&LAMP,FLAT,DETCHECK&64

definition of the matching rule; this entry is evaluated by egrep, use '&' for AND and "...|..." for OR ):
e.g.: I_spec&FLAT means 'find entries with "I_spec" AND "FLAT"'
ARC&"543.3|881.7" means 'find entries with "ARC" AND ("543.3" OR "881.7") '

The content of the pattern is defined by the available entries of your raw data reports.

description &&detector monitoring template&& free text to clearly describe (to PSO) what these data are; to be enclosed by &&...&&

name = HRULER (no further fields): get a horizontal ruler in the result table, for better overview

3. The OCA configuration files reside under $DFO_CONFIG_DIR/CALCHECK/OCA. You take care of the usual three files, <instr>_association.h / _classification.h / _organisation.h . All others are either created by the tool on the fly, or copied from your operational $DFO_CONFIG_DIR/OCA. As mentioned above, while editing these three OCA files please keep in mind that they create ABs which are read for association information only. They are never used for processing, nor for packing, and they are not stored permanently anywhere.

[ top ] Quick links

The quick links on top of the vertical navigation bar link to the most relevant sites for Paranal QC. "HC", "refs" and "QC" go to general pages from where you click to the selected instrument. The "HC" link in the navigation pane of "analyze ISSUES" directs to the first HC report of the respective instrument, for easy cross-navigation.

[ top ] Operational aspects

[ top ] Workflow

Classical (CHECK_C4C=NO):
1. Fill VCAL pool with all calibs required
2. Create SCIENCE ABs
3. Check completeness of SCIENCE ABs

CAL4CAL (CHECK_C4C=YES):
1. Fill VCAL pool with all calibs required    
  2. Create CAL4CAL ABs [calChecker -C]
  3. Check completeness of CAL4CAL ABs
4. Create SCIENCE ABs    
5. Check completeness of SCIENCE ABs    

Details:

  1. prepare configuration (create or copy required config files); check for older instances
  2. update headers for current date (HdrDownloader), update data report and nightlog
  3. create CALIB ABs and populate $DFO_CAL_DIR/CALCHECK/VCAL
    INCREM=YES: only for TODAY
    INCREM=NO: for all dates defined by $N_VCAL_DATE

  4. if CHECK_C4C=YES: call CAL4CAL mode
    4.1 create triggering CALIB ABs (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
    4.2 get OB grades and comments
    4.3 create result pages (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
    4.4 check for technical/maintenance calibrations
    4.5 check for beocal signal
    4.6 check completeness and create main page
    4.7 scp CAL4CAL result pages to external web sites and to Paranal mirror web site

    if CHECK_C4C=NO: skip this step

  5. create ABs for SCIENCE (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
  6. get OB grades and comments
  7. create result pages (INCREM=YES: for NOK/MISS dates and TODAY; NO: all dates)
  8. check for technical/maintenance calibrations
  9. check for beocal signal
  10. check completeness and create main page
  11. scp result pages to external web site and to Paranal mirror web site

Last update: April 26, 2021 by rhanusch