make printable	new:	see also: trendPlotter creating the HealthCheck reports; scoreQC feeding the scores into qc1_score database. All configuration for scoreHC is read from both trendPlotter and scoreQC (with the exception of the TEL and INS information for the all_score_overview pages stored in config.scoreHC).
		documentation about the php comment interface databases qc1..qc1_score; qc1..<qc_table>; _qc1..prod_comments dfos tools qcdate; scoreQC ; embedded in trendPlotter; see also webNavBar output used by trendPlotter upload/download score_overview.html and score files to web server
topics: description | score calculation & output | info on demand | comments | scoreManager | consistency checks I | checks II | checks III and IV | quick edits | action buttons | multiple datasets | config | operational aspects

scoreHC

enabled for parallel execution

embedded in trendPlotter: trendPlotter scoreHC

Description

This tool connects scores, as created by scoreQC, and the Health Check (HC) plots, as created by trendPlotter. It re-arranges the scores, originally derived per QC1 parameter and AB, into a scheme which focuses on instrument properties. The tool is called by trendPlotter, but can also be called from the command line.

The goal of trendPlotter+scoreHC is to come from QC1 parameters of individual frames to a statement about the current instrument status, and raise alerts in due time. Technically, this is achieved by representing each HC plot by its score report counterpart, also known as QUICK-LOOK version. The creation and delivery of that version is provided by scoreHC.

The tool is called for a specified report: scoreHC -r <report_name> (just like trendPlotter). It collects all QC1 parameters used in that report over the last 7 days, and reads their scores from the qc1_score database table. This is a two-step reduction of complexity: an instrument property is represented by a set of QC1 parameters (step 1), and these values are then scored and replaced by numbers 0 or 1 (step 2).

Resulting scores from the last 7 days before the last date indicated in the report are finally aggregated (older scores are ignored since they are considered to be irrelevant for the current status):

• all scores per plot are evaluated: the last one is used for higher-level scores, the others are just displayed
• higher-level scores: each last score gives the plot score
• all plot scores are aggregated to the report score
• all report scores are aggregated to the group score (representing one aspect/component of the instrument)
• all group scores result finally in the instrument score

Aggregation means a value of 0 or 1 where 0 can only be achieved if all component scores are 0. Thereby a single last plot score will turn all affected higher-level scores to 1, including the instrument score. This is an efficient hierarchical alert system.

All aggregated scores (derived from scores of QC1 parameters) are also called "Health Check scores" or HC scores (thus the name of the tool). HC scores come either as 0 (displayed green) or 1 (red). Contrary to QC1 scores, there are no yellow scores.

SCORE_DEPTH is hard-coded to 7 days (to align with calChecker). SCORE_DEPTH is counted backwards from the date with the last available score (as indicated in the score report). Thereby it is usually guaranteed that at least one score is visible and evaluated, even if it is already weeks old. This is because that score is considered as the latest available information, which might be outdated but will be replaced only by a new measurement.

Score calculation and output. The tool reads the trendPlotter configuration of the HC report, to derive the names of the QC1 parameters. Then, it reads the corresponding scores from qc1_score, and arranges the results in a graphical way using HTML format:

FF level (scores, last 7 days up to 2010-01-31*) 1: Med1 3| 2: Med2 2| 4: IFU1 4| 5: IFU2 2| 6: Argus 3|

QUICK-LOOK version

The format of this table is arranged exactly the same way as the original plot:

[click to enlarge]

HEALTH CHECK (HC) plot

Thereby, the HC report can be represented by its QUICK-LOOK version for quick reference. The resulting HC report then comes in two flavours, one of which is provided by scoreHC (just scores and HTML) , the other by trendPlotter (graphical information).

Information on demand. Whenever useful, you can get more score information on demand:

• The little score symbols are hyper-linked to the corresponding score report (try in the example score report).
• On moving the mouse over the score symbols you can see the corresponding date to which the score applies.

Comment section. Just below the score report, there is the News section, replicated from the HC report. It displays three kinds of news:

• 'General news' (of general interest for the HC monitor)
• 'Instrument news' (applicable to the instrument as a whole) and
• 'Report news' (news for this report).

The 'Instrument news' and the 'Report news' are managed locally with the scoreManager. With v2.0, the 'Report news' are managed with the web-based php script hcComments.php, see below.

There is also an extraction of the product comments. This is intended to align with the calChecker ANALYSIS section. For each score with a product comment (entered upon certification), there is a row with the corresponding DATE (linked to the AB product monitor), the name of the AB (linked to the score report), the score symbol, and the product quality comment. Note that the comment does not necessarily fit to the particular report displayed here, it is usually entered for the whole set of products and QC1 parameters.

General news: 2010-02-01: DTS works again. GIRAFFE news: 2010-02-02: FFLAMP flux is now ok. Report news: 2010-01-23: IFU1 flats saturated	News section
FF level (scores, last 7 days up to 2010-01-31*) 1: Med1 3| 2: Med2 2| 4: IFU1 4| 5: IFU2 2| 6: Argus 3|	score report
Quality comments: Plot Date* AB name score product quality comment Plot 4 2010-01-22 GIRAF.2010-01-23T06:50:12.384_tpl.ab saturated Plot 4 2010-01-23 GIRAF.2010-01-24T01:37:34.812_tpl.ab saturated Plot 4 2010-01-23 GIRAF.2010-01-24T06:02:36.832_tpl.ab ok Plot 5 2010-01-23 GIRAF.2010-01-24T07:45:59.520_tpl.ab ok "Date" links to complete AB product page; "AB" links to score report	Product quality comments

SCORE OVERVIEW AND MANAGER. In addition to the QUICK-LOOK score output, scoreHC also creates two overview pages:

Overview. The file score_overview.html provides an overview of all current scores per instrument. It is exported to the HC site and can be found under the URL http://www.eso.org/qc/<INSTRUMENT>/common/score_overview.html. Find the SINFONI example here.

It comes in the following versions:

• the main overview page
• the one with report comments for red score reports
• the one with all report comments
• the one with a query form to manage the comments.

The red score comments are helpful, both for Paranal and QC, to better and faster understand issues related to red scores. The all score comments are primarily useful for the QC scientist to monitor outdated report comments. The query form can be used by Paranal and by QC to create, update or delete comments.

The comment interface has a mandatory DATE field to be filled. This field is used by 'trendPlotter' to check if the comment is still valid. After 14 days, the comment is considered outdated, and the tool starts to send an email to the QC scientist with this information. It is up to the QC scientist to decide what to do: most likely delete, or update. The tool does not delete outdated comments by itself. Report comments with historical interest should go to the "HISTORY" part of the tutorial pages. General comments should go to documentation section ('This plot').

Score manager. There is also a local scoreManager file ($DFO_TREND_DIR/reports/scoreManager/scoreManager.html). This is an interactive GUI to manage scores. Here you can obtain an overview of all AB which contributed to the current instrument score. It helps you understanding how the HC scores are linked to the QC1 scores.

The local score manager has, apart from the main page, one detailed page per report, which is described below.

stat_grp file. The main scoreManager.html file has a useful feature that is not available on the exported score_overview.html file: it has a column named stat_grp that links to the technical trendPlotter result tables under $DFO_TREND_DIR/reports/stat_files. These files show you the names and group scores for each HC group and thereby help to better understand, and possibly debug, the configuration of a report group.

HC group	stat_grp*
detector	DETECTOR

The 'DETECTOR' link displays the stat_grp_DETECTOR file (the report group file for the DETECTOR group):

Find a complete example here.

Consistency checks I. The scoreManager assists you in checking for hidden 'zombie' group scores. This is a notorious problem if at some point your HC report system has been modified (groups have been re-arranged; group names or report names have been changed; tests have been executed). Then it might happen easily that a score file of an inactive report, or group, survives. If that inactive file would store a red score, this alert would propagate and could not be fixed by any action on active reports.

Therefore the scoreManager scans your system for inactive group scores. It checks for two aspects: inactive group files, and invalid content of (active or inactive) group files. In either case, the found candidates are displayed and marked in red (example). Their analysis and removal is left to you. If the system is consistent, all is marked green, like in the example here.

A brute-force approach, as devised by Mark Neeser:

1. delete all stat_* files in your $DFO_TREND_DIR/reports/statFiles/   directory
2. re-create the contents of your statFiles directory by running scoreHC -r <report_name> for all the reports listed in your JOBS_TREND file.

This should eliminate all of the zombies created in past scoring experiments, but make sure to reconstruct the contents (step 2) as soon as possible after the deletion, otherwise your HC scoring system is out of order!

Consistency checks II. The tool does a second kind of consistency check. It checks, for a given QC1 parameter and data type, if there is an inconsistency between thresholds configured in config.scoreQC, and in a related HC report (config.tp_<report>). An example: the struct_row parameter for data type BIAS is scored with fixed thresholds 0.07 and 0.10, while there is a BIAS HC report for the same parameter with thresholds 0.05 and 0.12. Then, a value of struct_row=0.11 would produce a red score without graphically indicating an outlier in the HC report. This would cause confusion. Instead the user should link the scoreQC configuration to the trendPlotter configuration (dynamic configuration). This configuration problem is hard to discover. Therefore scoreHC checks for those cases. It sends an email reminder to the QC scientist when a potential inconsistency is found, along with some proposal how to improve.

There might be cases where this non-standard situation nevertheless makes sense, then the user can keep the configuration and avoid future alerts by marking the corresponding line in config.scoreQC by #STATIC (at the end of the line). Known such exceptions are:

• The HC plot displays multiple data sets (distinguished by setup), but scoreQC has configured "ANY". Then you should take care to configure the "most appropriate" value in scoreQC and flag it (#STATIC).
• The HC plot displays a subsample or a supersample of the data set scored in scoreQC.

Consistency checks III and IV.

Because the HC_PLOT section of config.scoreQC is important for the incremental update of HC reports within autoDaily, the configuration is checked in two directions:

• All HC reports configured in the HC_PLOT section of config.scoreQC should have their config file in $DFO_CONFIG_DIR/trendPlotter/config.tp_<...>. This check protects against obsolescence of entries in the HC_PLOT section.
• All config.tp_<...> files should have a report configured in the HC_PLOT section. This check protects against forgotten HC plots in the scoreQC configuration, which can happen easily with new reports. The consequence would be that the corresponding HC report does not get updated by autoDaily (only by the bulk refresh once a day but even then only if included in the jobs file.).

The results of both checks appear as section 3 and 4 on the scoreManager. Inconsistencies in part 4 are also mailed once a day to $OP_ADDRESS, because of their fundamental importance.

Consistency checks V.

There is the operational rule that all HC plots (including the FULL and KPI plots) should be updated at least once per day. This bulk refresh is done by a cronjob task using the job file JOBS_TREND. For full efficiency this file needs to list all existing HC plots. The scoreHC tool checks for config files in $DFO_CONFIG_DIR/trendPlotter that do not have an entry in JOBS_TREND. If found, these cases are listed in section 5, for further action.

Detail pages per report.

Quick updates. The detail report pages have on top a function to quickly update the NEWS file. The 'edit' button calls the web-based form hcComments.php (the same which is linked to the score overview page). All edits are done on the web. This local page cannot be refreshed by the php form (since it cannot start local processes), and it cannot include the new content of the news (since this works only on the web server). Hence only you can refresh the local content, and this is the purpose of the 'refresh' button. If you forget to refresh that page, the next automatic trendPlotter process will do.

GIRAFFE news:
Report news:			If you have edited the report news, they are visible on the Web but not locally. Don't forget to refresh the page.
HC report	edit config_tp_BIAS file	view config.tp_BIAS file

Each detail report page is structured like the following example (find the full page here). The content is extracted from the AB status pages:

1. Score results [this repeats the score report]

BIAS parameters (scores, last 7 days up to 2008-12-12) 1: BIAS_level 6| 2: RON_raw 6| 3: RON_master 6| 4: struct_X 6| 5: struct_Y 6|

2. Extraction from AB monitors [don't expect links to work in this environment!] (list of all ABs which have entered the current score result table, sorted by DATE, extracted from AB monitors)

DATE AB ([local]: in $DFO_AB_DIR) COMPL. AB LOG RECIPE RAW_TYPE SETUP STATUS LOG T QC_REPORT SCORE CERTIF 2008-10-09 GIRAF.2008-10-10T15:31:31.463_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.3 DONE (0/6) OK 2008-10-11 GIRAF.2008-10-12T10:14:44.424_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.3 DONE (2/6) OK 2008-10-12 GIRAF.2008-10-13T12:06:40.624_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.3 DONE (2/6) OK 2008-10-12 GIRAF.2008-10-13T18:39:35.736_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.3 DONE (2/6) OK [local] GIRAF.2008-10-15T12:45:03.281_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.3 DONE (0/6)   [local] GIRAF.2008-10-16T12:54:41.381_tpl.ab compl. OK   gimasterbias BIAS 1 OK P_LOG 0.4 DONE (0/6)

3. Details per plot and QC1 parameter [don't expect links to work in this environment!] This is a list of all QC1 parameters and scores for the BIAS report on the HC monitor. It is not complete in terms of ABs. This list can be used to correct, remove or override scores interactively.

Workflow for correction: 1a. modify configuration 1b. call trendPlotter for report --> update stat_text file and plots 1c. call scoreQC for AB --> write new scores into db 2. call scoreHC --> update HC monitor Workflow for deletion: 1. delete all scores for selected AB in db 2. call scoreHC --> update HC monitor Workflow for NOK override: 1. force all scores for selected AB to green/OK 2. call scoreHC --> update HC monitor

Action Plot QC1 table QC1 param DATE MJD_OBS score AB Plot 1: giraffe_bias median_master 2008-10-09 54749.646892 0 GIRAF.2008-10-10T15:31:31.463_tpl.ab Plot 1: giraffe_bias median_master 2008-10-11 54751.426903 0 GIRAF.2008-10-12T10:14:44.424_tpl.ab Plot 1: giraffe_bias median_master 2008-10-12 54752.504637 0 GIRAF.2008-10-13T12:06:40.624_tpl.ab Plot 1: giraffe_bias median_master 2008-10-12 54752.777497 0 GIRAF.2008-10-13T18:39:35.736_tpl.ab Plot 1: giraffe_bias median_master 2008-10-14 54754.531288 0 GIRAF.2008-10-15T12:45:03.281_tpl.ab Plot 1: giraffe_bias median_master 2008-10-15 54755.537979 0 GIRAF.2008-10-16T12:54:41.381_tpl.ab Plot 2: giraffe_bias sigma_raw 2008-10-09 54749.646892 0 GIRAF.2008-10-10T15:31:31.463_tpl.ab   Plot 2: giraffe_bias sigma_raw 2008-10-11 54751.426903 1 GIRAF.2008-10-12T10:14:44.424_tpl.ab   Plot 2: giraffe_bias sigma_raw 2008-10-12 54752.504637 1 GIRAF.2008-10-13T12:06:40.624_tpl.ab   Plot 2: giraffe_bias sigma_raw 2008-10-12 54752.777497 1 GIRAF.2008-10-13T18:39:35.736_tpl.ab etc. ...

The red button will call 'scoreQC -a <ab>'. This might be reasonable when you have modified thresholds definitions. Because of the complex link between config.scoreQC and HC configuration (dynamic thresholds), the HC report needs then to be refreshed, then scoreQC is called, and finally scoreHC is called (for a second time) to transfer the updated scores to the HC monitor.

The blue button (offered only for NOK scores) offers deletion in qc1_score. This might be reasonable when you discover that a certain file was invalid, and its scores would raise a false alert. The associated workflow also updates the HC monitor.

The green button (offered only for NOK scores) offers overriding a red score. This might become useful when you have analyzed an issue, find that in this particular case there is no issue, or the issue has been solved, but you have good reasons to not modify the configuration. The associated workflow also updates the HC monitor. Make sure to not call 'scoreQC' again for that AB: since you have not modified the configuration, you will see agin red scores.

Note that for instruments producing MEF files (like CRIRES or HAWKI), results from a particular AB will display in several lines. For instance, for CRIRES, there will be six lines (4 detector entries, plus AVG and RMS). For consistency, the blue buttons will delete all scores for all extensions from the AB. For our CRIRES example this means a single delete action will delete all six lines at once.

score_list_<instr>.txt file. This is a temporary output file used for the evaluation of the Health Check site. It is created under $TMP_DIR.

MULTIPLE. As a general rule, scoreHC requires single data sets per plot (where OPLOG data sets are not counted since they never get scored). This has two reasons: one is that the tool would then become overly complex. The other reason is that displaying multiple data sets in a single plot with different threshold values and rules makes the graph quickly unreadable. Therefore, scoreHC checks for these cases and returns "multiple dataset" instead of scores.

There is one exception to this general rule: if you define two or more data sets for a given plot in your trendPlotter configuration file, but make sure that you have only one dataset with a scoring condition (section SCORE_COND in the trendPlotter configuration), then that dataset gets scored and displayed as QUICK-LOOK (but all others not). A typical application would be that you want to display all flatfield fluxes in the J filter (dataset #1), and also overplot (in different colors) the ones for different plate scales (data sets 2...n). Then you can score dataset #1 and leave the others unscored, by defining SCORE_COND for that dataset only. As soon as the tool detects more than one SCORE_COND for a plot, this is handled as "multiple dataset" case. An example is found here.

Output

- score HTML files: all HC score results are wrapped in a piece of HTML code, exported to the Health Check web site and linked dynamically (through the virtual include mechanism) to the HC pages.

- score_overview.html and scoreManager.html

How to use

Type scoreHC -h for on-line help, scoreHC -v for the version number,

scoreHC -r <report_name>

to create the QUICK-LOOK version of report <report_name>. This syntax is exactly the same as for trendPlotter.

Type

scoreHC -P

to create the php scripts and directory structure for the comment interface, or to update its code.

Type

scoreHC -I

to create only the local scoreManager interface (without creating a score report, and hence faster).

You can call the tool on the command-line. The standard use is from trendPlotter.

There is also a special option

scoreHC -O [overview]

that creates the common score overview page including the 'last 7' and 'last 30' pages. It is *not* required to be run in routine operations. The score overview page all_score_overview.html itself is a static HTML page around the instrument scores. The 'last 7' and 'last 30' pages are created by the tool from a local memory file. That file is called SCORE_OVERVIEW and maintained in $DFO_MON_DIR of the maintenance account. The overview mode parses the all_score_overview.html file once a day, adds the new instrument score from all configured instruments to the SCORE_OVERVIEW file, updates the 'last 7' and 'last 30' pages and uploads them to the public web server of the HC monitor.

Configuration file

For normal use, there is no tool-specific configuration. Most required configuration is read from config.trendPlotter or from the report configuration file, config.tp_<report_name>.

Note that some information is also read from config.scoreQC, and that the usage of multiple configuration there (MULTI_CONFIG=YES) has certain implications:

If MULTI_CONFIG=YES in config.scoreQC, and if a REPORT contains plot(s) that have mixed RAW_TYPE, then you should specify the proper config file for scoreQC for each plot in the report. This has to be done in config.scoreQC, in the section 4.1A. Please check details there.

For the special mode 'scoreHC -O' that creates the all_score_overview pages for the last 7 and 30 days (accessible from the shiftleader's page), there is the configuration file config.scoreHC. It is called by the hard-coded account $CREATE_OVERVIEW_USER. This configuration file has the currently active TELescopes (UT1-4, VLTI, survey telescopes) and INStruments.

Operational aspects

• Manage the report news such as to have them always up-to-date and relevant. "News" will typically become outdated after two weeks or so. They should then be removed.
• See also sections Hints and Pitfalls in the scoreQC and trendPlotter documentation.
• The consistency checker of the scoreManager lists in section #4 all config.tp_<...> files that are not configured in the HC_PLOT section of config.scoreQC. Because of the operational importance of that section, any inconsistencies are also mailed, once a day. There are several options to resolve these inconsistencies, and get rid of the daily reminder mails:
  • A config.tp file is really obsolete: then you may want to delete it, or move it to some OBSOLETE folder; scoreHC will only scan $DFO_CONFIG_DIR/trendPlotter.
  • A config.tp file is obsolete for operations but you are not yet sure: rename to e.g. AA_config.tp; scoreHC will only scan for files named like config.tp_<...>.
  • A config.tp file is operationally used for a FULL plot but not configured in the HC_PLOT section: include it in the HC_PLOT section.
• How to include the scoreManager in the local monitor navigation bar:
  - edit config.dfoMonitor to include 'scoreManager' (col. #3) and '<expanded path to scoreManager>scoreManager.html' (col. #4)
  - push "navigation bar: [refresh]" on the dfoMonitor
  - now the scoreManager is embedded in the navigation bar of dfoMonitor; the other tools follow as soon as they are refreshed.