make printable	new:	see also:
	trendPlotter v4.0, and pet.py v2.0: - interactive close-up plots using Highcharts NOTE: any reference to THRESHOLD methods PER and OFF has been removed; it is discouraged, will be de-commissioned, and shall not be used on the HC monitor. Note: support for the data source LOCAL will be discontinued in the future.	trendPlotter tool trendPlotter operations qc1Plotter web interface navigation bars can be created by webNavBar
topics: general | section 0 | section 1 | section 2 | section 3.1 | section 3.2 | symbols, colors, sizes | statistics | section 4 | conditions | compute rules | delta configuration | report config file | group config file

trendPlotter: report configuration

Here the report configuration is documented. The tool documentation is found here, operational hints here. A description of the non-standard tool configuration file (designed for special tasks) is available here.

Configuration: general

trendPlotter has, due to its configuration complexity, its own configuration directory: $DFO_CONFIG_DIR/trendPlotter. There exist the following configuration files:

The typical report REPORT will require one configuration file, config.tp_REPORT, and the two text files REPORT.txt and REPORT.inf . You may also want to add REPORT.msg. For non-standard formats, the format definition file REPORT.fdf is required. For HISTORY reports, you may also need a delta configuration file. Finally, a report group is defined in a group definition file.

In the following, we describe the report configuration. The description follows the hierarchy scheme of data presentation in a trending plot:
- [if applicable, report group definition];
- report definition;
- plot definition;
- data set definition.

The table at bottom lists all configuration keys in detail.

Section 0. General

This is some general information about the report. It contains an optional URL (with tutorial information about the trending plot), an optional key REPORT_GROUP for grouping, and a key START_DATE for the history.

Report groups. Report groups are defined by the config key REPORT_GROUP and the name of the group file. The group configuration file is described below. The list of configured reports creates a horizontal navigation bar in each of the reports belonging to the group, e.g.:

same group:

U_BESS

B_BESS

V_BESS

START_DATE. This key is required to define a start date for the automatic creation of the HISTORY plots and their navigation bars. It must fit to the raster defined by RANGE. E.g., if your first data point is from 2003-09-15, and your raster is defined as 90 days, you can start with 2003-07 (and be complete), or with 2003-10 (loosing the first points). You cannot start with 2003-09.

FULL_NAME. If a related report exists with RANGE=FULL, you can configure its name here and it will be included in the HISTORY navigation bar.

VNAVBAR. This is an optional key to prepare for advanced navigation. By default, all reports get the same vertical navigation bar the name and location of which is hard-coded: /observing/dfo/quality/ALL/common/navbar_HC_<instr>.html . If this key, however, is specified under VNAVBAR, a dedicated navigation bar is included which should differ from the standard one only by highlighting the name of the current report. It should then be provided under /observing/dfo/quality/<instr>/common/ . See e.g. http://www.eso.org/observing/dfo/quality/GIRAFFE/common/ . This is an efficient way of improving the navigation since the user always knows the current page, no matter if related links have been called in between. The navigation bars can be created automatically by the TQS tool webNavbar.

MARK_NO_DATA. By default trendPlotter plots a tag [no data] if no data are currently found for a dataset. If that key is set to NO, this behaviour is turned off.

Section 1. Report definition

Report format. The plot format of a report is defined in the key REPORT_TYPE. The tools supports template and custom formats. Supported template formats are: REPORT2, REPORT4, REPORT6, REPORT8, REPORT10.

REPORT2 plot1   plot2

REPORT4 plot1 plot2 plot3 plot4

REPORT6 plot1 plot2 plot3 plot4 plot5 plot6

REPORT8 plot1 plot2 plot3 plot4 plot6 plot7 plot8 plot9

REPORT10 plot1 plot2 plot3 plot4 plot5 plot6 plot7 plot8 plot9 plot10

The purpose of template formats is to make configuration easier, and also to offer a certain standard look and feel for all QC trending plots.

You don't need to 'fill' a report completely. If you have just three plots to display, you could choose REPORT4 or even REPORT6, and the remaining boxes are left empty. Find here an example for the REPORT6 format filled with 5 plots.

REPORT_TYPE=CUSTOM. There is also the option to define a CUSTOM format, which then has to be provided by the user in an external format definition file (fdf). This file needs to have the name <report_name>.fdf and to be stored in $DFO_CONFIG_DIR/trendPlotter/. It contains all coordinate information about the plots in MIDAS syntax. Similar information for the standard reports is hard-coded in the tool, in section 0.3 under "#REPORT_TYPE", and can be used as a template for the fdf file. The installation package also has an example fdf file included. See here an example for a report created with a custom fdf file.

RANGE. The time range for the report can be configured under RANGE. The standard case is as number but there are also other cases (FULL, KPI):

• 60/90/180 or 365 (days). The most commonly used value is 90 which results in 4 reports per year in the HISTORY. The allowed values are restricted in order to provide a certain standardization of the reports in the HC monitor. There should be a proper balance between acquisition frequency and selected RANGE: 60 should be used for frequently measured QC values (maybe more than once per day), while 365 should be reserved for parameters taken only once per month or so.
  
• RANGE=FULL: this value can be used for reports which display all existing data points, starting with START_DATE. A FULL report is configured to display long-term trends. Such reports require particular care for scaling: the XAXIS key needs to be chosen such that labels remain readable. The YAXIS key requires careful configuration to be able to display the full historical data record.
  
  A FULL report makes sense for two cases:
  
  • there are reports with data taken only every month or even more rarely. Still one may want to trend those data (e.g. since they are measured regularly, or since they record an instrument property which changes only slowly over time but still is important). Then, even a 365 days range may be to short to reveal significant trends.
  • one may want, in addition to a "normal", short time range, also display the total variation in one report. Then all historical reports can be regarded as a "zoom", and the FULL report is of additional interest.
  
  In the first case, the FULL report would be included in the group bar just as a regular report of its own value. In the second case, one may prefer to offer it as additional HISTORY-type report.
  
  Both can be configured. The first case is covered by just preparing a configuration file with RANGE=FULL. You would then call that report just like all others.
  
  The second case requires two actions: first you configure a report with RANGE=FULL. This time, it will likely be a copy of an existing report, let's say with name BIAS. You would then call the new report BIASFULL (just an example), with almost identical content, except for RANGE=FULL, carefully chosen XAXIS and YAXIS values, and reasonable REMARKS. Second, you would configure the key FULL_NAME in the BIAS configuration file, to contain the name of the additional FULL report. Then the HISTORY navigation bar of the BIAS report will automatically be supplemented by a link to this FULL report:

	close window	history:	2003	2004	2005	2006					2007			HEALTH	FULL
			...	...	...

• RANGE=KPI: this is very much like FULL, with the additional feature that a reference value as configured in sect. 4.7 is overplotted.

Additional keys. Each report can be configured to mark the entries for the last 24 hours (MARK_LAST, counted backwards from the current last data point), to mark vertical date lines for better orientation (MARK_DATES), and to mark 'aliens' as red arrows (data points falling outside the defined Y range: MARK_ALIENS). You can define a short label to appear in the first line of the plot header (ADD_TEXT) and a description file name (DESCR_FILE), to be included in the HTML page ("This file").

Marking statistical outliers is configured per data set in the SYMBOLS section.

Section 2. Plot definition

Section 2.1: Plot properties.

Each plot needs a unique PLOT_INDEX (which determines its position in the report and ranges from 1 to the maximum number supported by the REPORT_TYPE) and a PLOT_NAME (used to match entries in PARSET_NAME and SYMBOLS sections). The PLOT_INDEX will show up in the report, the optional PLABEL will display within the graph on the upper left.

The plot properties are: XFORMAT, XLABEL; YFORMAT, YLABEL; XAXIS, YAXIS; PLABEL; SCORE_ENABLED, NO_MARK_LAST, SFORMAT, INTERACTIVE. See the table below for details.

SCORE_ENABLED is a key which by default is YES. With NO, you can control this report to not be linked to scoreQC (its QUICK-LOOK version displays as "not implemented"). This should be used as an exception. This key, like the following, is optional. You can drop it if this and the following keys are configured by their defaults.

NO_MARK_LAST: while MARK_LAST controls the marking of the last data point for the whole report, you can turn it off for a particular plot by configuring NO_MARK_LAST as YES. Default is NO. Note that this key takes effect only if MARK_LAST=YES.

#PLOT_NAME  P_INDEX PLOT_NAME  XFORMAT XLABEL  YFORMAT YLABEL          XAXIS   YAXIS      P_LABEL       SCORE_ENABLED   NO_MARK_LAST  SFORMAT  INTERACTIVE  
PLOT_NAME   1       PRESS_CRYO AUTO    mjd-obs AUTO    pressure_[Pa]   50,10   0,0.022    PRESS_CAMERA  NO              YES           AUTO     INTER 
PLOT_NAME   2       TEMP_CRYO  AUTO    mjd-obs AUTO    temperature_[K] 50,10   30,42,2,.5 TEMP_CAMERA   NO              NO            AUTO     STATIC

NO_MARK_LAST=YES means that the last data point is not marked for this plot, independently of the value for MARK_LAST in the REPORT_NAME section. NO_MARK_LAST=NO or any other value does not change the MARK_LAST behaviour. An omitted entry is the same as NO_MARK_LAST=NO.

SFORMAT: you can control the format of the average output (e.g. MEAN, MEDIAN). It displays in the result table, and if you mouse over the plot. Possible values are:
- AUTO: corresponds to .3g and is the default
- NONE: corresponds to g, the format used until tool version 3.0
- a format string like e.g. 8.3f
For most cases AUTO returns the best results, namely 3 significant digits (e.g. 0.0837 instead of 0.083764). There are exceptional cases, e.g. the CRIRES WAVETHAR plot which needs more than 3 significant digits for its mean value 1083.774 (which is a wavelength in Angstroem).

The default format of the results in the output table is AUTO (.3g).

INTERACTIVE: new with 4.0. Optional key to de-configure interactivity for a close-up plot. Can be 'STATIC' or 'INTER' (default). This should only be used and set to 'STATIC' for sub plots that have very specialized FIT_RULEs (see below) that are not supported for the close-up and which would make the interactive close-up plot not useful. Please note that the columns for SCORE_ENABLED, NO_MARK_LAST, and SFORMAT also have to be present when the INTERACTIVE column is to be used.

Section 2.2: Plot events.

In this section, EVENTs that have an impact on QC parameters can be marked for both the main plot and the corresponding close-up plot. Events have a defined mjd-obs and are marked with a vertical line in plots that show trending versus time. They are a much simpler way of entering vertical lines than the previous FIT_RULEs. A typical case that could be marked is a mirror re-coating for zeropoint or efficiency trending plots.

#EVENT   PLOT_NAME       ETIME   ECOLOR   SHORT_ELABEL	ELABEL
EVENT    IZeff           56970   7        wash		&&M1 washing&&
EVENT    IZeff           57620   4        coat		&&M1 coating&&

For events, the PLOT_NAME, the time ETIME, the colour ECOLR, and a short (SHORT_ELABEL) and a long label ELABEL have to be defined. The long label (multiple words, enclosed by &&...&&), is printed on the close-up plots, next to the vertical line that marks the event. The short label (one word!) is used for the main plot. This is useful to avoid overly crowded plots on the main plot.

Section 2.3: Lines.

Lines (other than events, e.g. horizontal) can be drawn as a straight line for both the main plot and the corresponding close-up plot. This is intended to support simple fits. (Note that more complex fits (e.g. parabolas) must be defined with a FIT_RULE and cannot be plotted on the close-up plots.)

#LINE    PLOT_NAME       LX1	LY1   LX2	LY2      LCOLOR
LINE     IFU_1           50000 0.4   60000 0.4      4 
LINE     IFU_2           0 	0     3 	3        2

The PLOT_NAME, the x and y coordinates of the start point of the line (LX1 and LY1), the x and y coordinates of the end point (LX2 and LY2), and the line color (MIDAS colour codes) have to be configured. In the above example, IFU_1 is a plot versus mjd-obs and has a horizontal line at y=0.4. LX1 and LX2 are outside the plot range but the line is cut appropriately. In the IFU_2 plot, a line is drawn from (0, 0) to (3, 3).

This is just about plotting lines, without labels.

Section 3. Data set definition

Each plot displays at least one data set. It is defined in Sect 3.1. The number of data sets per plot is not fundamentally limited. Each data set requires a definition of its plot symbols in Sect. 3.2.

Section 3.1: data set specification. A data sets needs to be uniquely identified by PARSET, its parent plot is identified by PLOT_NAME (that's why PLOT_NAMEs need to be unique). It has a data SOURCE. The only supported value is QC1DB (although still LOCAL is supported but this will become obsolete in the future).

The name of the QC1DB database table must be specified (e.g. fors2_bias).

The data set is further defined by XPARAM and YPARAM, the names of the parameters to be displayed and as used in the table. For QC1DB tables, these are the names of the QC1 parameters to be displayed, e.g. mjd_obs and flux.

Typically the X axis will be mjd_obs (which makes the report a true trending report), but the user may also choose any numerical parameter as X axis to create a correlation plot. In that case the tool will always auto-add mjd-obs to the download query in order to provide the time range selection.

Some more properties of the data set can be configured: DOWNLOAD (offer data for download, only possible for SOURCE=QC1DB); CONDITION (apply optional additional selection criteria), and REMARK (optional remarks entered in the result table). Conditions and remarks are entered by a unique name and are defined in Sect. 4.

Section 3.2: plot parameters. Here the plot parameters for each data set are defined, like symbols, connecting lines, statistics, units, compute and fit options.

trendPlotter uses the python tool pet.py to generate the report plots. The symbols and properties are configured in the following way (data set by data set):

YSYMBOL (plot symbols):

The table below list the original MIDAS plot symbols. Not all these symbols are available with Python Matplotlib (main plot) and with Highcharts (interactive close-up plot). The symbols that are identical for the main and the close-up plot are 2, 3, 4, 11, 17, 18, 19, 20, and 21.

YSIZE (symbol size): relative symbol size. "1" is the standard size for a full page. You can choose larger or smaller values. Since all plots are smaller than a full page, a size like "0.4" will probably display best in many cases. For the close-up plots, symbol size is set automatically by the tool.

YCOLOR (symbol colors): From the supported colors, only 1,2,3,4 will generally make sense. 6 and 7 also display well but may look a bit fancy. 5 and 8 cannot be recommended:

LINE: you can choose (YES/NO) to have data points connected by a solid line. Generally this is not recommended and not necessary, at least not for outlier detection since outliers beyond the Y limits are always marked. Furthermore, connecting scattering data by a line is misleading and confusing. Connecting lines could become useful if there is a trend to be highlighted and the scatter is small, e.g. to indicate a temperature slope.

Statistics. Here the user configures statistical parameters for the trend analysis.

AVG. The averaging options MEAN, MEDIAN, FIXED and NONE are supported. Their value is marked in the plot, and listed in the table. If you don't want averaging, enter NONE. MEAN and MEDIAN options are using outlier clipping: statistics are calculated a second time with exclusion of the outliers found in the first run. FIXED is an option to keep the AVG value fixed by the user, no average is calculated.

THRESH. One thresholding option VAL=val1,val2 is available. The other two (PER, OFF) has been de-commissioned and should be replaced by VAL.

The VAL thresholding is static and user-controlled. The threshold value is marked in the plot, and listed in the table. If you don't want thresholds, enter NONE.

Outliers. Statistical outliers (data outside the thresholds but within the Y plot limits) can be marked if a statistics option is set. Marking is done by a red asterisk (a blue one if the symbol color is red).

Units. Used for display in the result table.

COMPUTE_RULE, FIT_RULE. There is the option to configure compute rules which are applied to the data set, and a fit rule. Define their names here, and specify them in Section 4. There is no real fitting option for trendPlotter, but you can define (in python syntax) a curve to be plotted on top of the data. That line (e.g. a polynomial) could serve as a reference slope (to indicate nominal behaviour). It is always plotted as dotted line. New with v4.0: fit rules are applied to the main plot but cannot be applied to the interactive close-up plots. Only in exceptional cases with setting INTERACTIVE=NO in section 2, fit rules are also applied to the (then static) close-up plots.

CR_COMPENSATE (YES | NO). If scoreQC is using thresholds set to "HC", and if a compute rule is applied to the data in trendPlotter, then a compute rule compensation (CR_COMPENSATE) is needed to match the values scored with the thresholds given by trendPlotter. In other words, if your config.scoreQC report entry has a HC_FACTOR that is different from 1, then CR_COMPENSATE = YES. Otherwise, this can be left blank (default = NO).

Section 4. Special definitions

In sect. 4, the following special definitions are configured:

• CONDITIONS
• ADD_TEXT
• COMPUTE_RULEs
• FIT_RULEs
• REMARKs
• Reference values (for KPI only)

Conditions. If you have specified a condition name in sect. 3.1, you need to define it here. Conditions are optional. They can be used to add constraints to the data sets. Unfortunately, they need to be formulated in up to three different syntaxes: one (under CONDITION) is in SQL and constraints the QC1DB data set. The one under CGI_COND is in cgi syntax and is used to constraint the download link (actually it uses the qc1Browser cgi syntax). Finally, for the QUICK-LOOK mode (scoring), there is SCORE_COND which comes in SQL syntax and duplicates the CONDITION, with each key prefixed with 'QC.'. This is required for a well-formed join query.

Note that generally each condition needs to be formulated in all 3 different syntaxes.

SQL conditions (CONDITION, SCORE_COND). In SQL syntax, the condition needs to be configured for content after the WHERE clause. For instance, you may want to select data satisfying the WHERE clause "ins_filt_name=U_BESS". You would configure it as "&&ins_filt_name=U_BESS&&". Conditions need to be enclosed in double ampersands, like "&&<condition text>&&". No character is permitted after the last '&&'. The same condition in SCORE_COND syntax reads: "&&QC.ins_filt_name=U_BESS&&".

CGI conditions (CGI_COND). If you have defined an SQL condition, you should make an attempt to also translate it into the cgi syntax and configure it under CGI_COND (don't forget to repeat the COND_NAME). For instance, the above condition in cgi syntax would read "&&field_ins_filt_name=ins_filt_name&filter_ins_filt_name=U_BESS&&". The syntax of the cgi script is:

• each parameter section is separated from the others by a single '&'
• parameter sections always come either as field_<param_name>=<param_name> (definition of parameter to be downloaded), or as filter_<param_name>=<value> (filtering).

Each condition has to be enclosed in double ampersands. No character is permitted after the last '&&'. No spaces are permitted for cgi syntax.

Check out here how to encode special characters like "+" in the cgi condition.

In general, SQL is more flexible than the cgi script behind qc1Browser. E.g., the cgi script can only filter by certain columns, cannot use expressions other than '=' etc. For that reason, it might not always be possible to dynamically download the same data that have been plotted. For instance if the data have been modified (by a COMPUTE_RULE) or if they have been filtered (e.g. by EXPTIME > 10). To avoid such inconsistencies, a configurable option exists to suppress download links (DOWNLOAD NO), and you are encouraged to use it in those cases. A broken download link, or one which extracts different data than the ones plotted, can cause confusion and frustration. Unfortunately there is no automatic way to discover and fix this problem. At least you should mention this issue in the remark column (REMARK).

Download links. The output HTML page offers links to download the data displayed in the reports (only if the configured data source is QC1DB). There are three links which only differ in time range (this: the same time range as the plots; last_yr: last 365 days of data; all = full time range). For a trending plot, mjd_obs, civil_date and the Y parameter are queried by default. A correlation plot downloads mjd_obs, civil_date, X and Y. You only need to configure additional columns, as wanted, by specifying a CGI_COND.

ADD_TEXT. This is a short optional label for the plot header (first line), limited to probably 30 characters. It may contain spaces. Check out the closeup plots generated with a certain ADD_TEXT to see if it's too long.

Compute rules. These are optional rules to modify the downloaded data set before plotting. Define here the content (enclosed by &&...&&). It requires python syntax and is entered by the tool directly into the code. A simple example is when RON values are stored in ADU but should be displayed in electrons. In such case, be careful with the data download option (see above).

FIT_RULES. Optional FIT_RULEs can be configured here. Syntax is the same as for compute rules.

FIT_RULEs can only be applied to the main plot and not to close-up plots. If they correspond to logical events (vertical lines in MJD-OBS plots with a short marker), they should be replaced by an EVENT, as configured under Section 2.2. If they refer to the data and are actually straight lines, they should be replaced by a LINE, as configured under Section 2.3. Both EVENTs and LINEs have the advantage that they can consistently be plotted on the main and on the close-up plots. Only if a curve is more complex than a LINE (e.g. a parabola), it should be configured as a FIT_RULE and then displays only on the main plot.

Remarks. You can configure short remarks which show up in the result table. They come as free text and need to be enclosed by &&...&&. Make sure that the remark text is shorter than about 120 characters, otherwise the tool will fail without helpful comments. Anything longer than that limit could go to the "This plot" file.

Reference values. For KPI plots, it is important to have a reference value configured and plotted. It is defined in Sect. 4.7 of the configuration file as REF_VAL, one per plot.

Other remarks and text additions. The tool adds a general caption which is hard-coded. This is the section starting with "General information".

Information file. There is the option to add a specific information text which is meant as a short tutorial about what is displayed. That section is added after the statistics table, before the general caption. It is stored in the information file under $DFO_CONFIG_DIR/trendPlotter, its name is configured under DESCR_FILE. The main purpose of that text is to provide quick-look information about what is displayed in the graph. It is a rule of thumb to always have, as a minimum content, a one-line description of X and Y parameter(s) per plot. This cannot be given in the statistics table since this is simply too much information.

Delta configuration

In HISTORY plots, it may happen that a configured Y range is fine for the current plot but inappropriate for earlier ones. To support such incremental configuration changes, there is the mechanism of delta configuration files. These are called delta.tp_<report_name> and contain only those lines with a differing values. They have added a final column (YEAR_MONTH) specifying the applicable time range. This column always needs to be added, no matter what the other columns are.

There are three possible values for YEAR_MONTH:

• a specific YYYY-MM value, like 2005-01: the delta configuration is applied only if start date equals YEAR_MONTH.
• a more general value like 2005_ALL: this delta configuration applies to all 2005 reports.
• the value HISTORY_ALL: the delta configuration is applied to all HISTORY reports. This may make sense if you want e.g. to apply thresholding for HEALTH reports only but not for HISTORY reports.

For HISTORY reports, trendPlotter will always check if there is a delta config file applicable for the specified start date, and use it to overwrite the standard configuration.

Note: since in delta configuration files PLABEL is not the last column anymore, you must always have an entry for this key. Enter NONE as PLABEL if you don't want any plot label.

Configuration files

The tool reads its tool configuration file (config.trendPlotter), plus the specific report configuration. All configuration files for trendPlotter are collected under $DFO_CONFIG_DIR/trendPlotter.

1. config.trendPlotter

The tool configuration file is described here.

2. config.tp_<report>: the report configuration file has all the configurations for a specific report:

3. Optional delta configuration file for HISTORY plots:
Any arbitrary line from Section 3 can be configured in a delta config file, with the last column YEAR_MONTH added.

The example below changes the configured YAXIS to a new value, for 2003-10 only:

Another example to show how to change the THRESH value for all HISTORY plots (standard configuration is VAL=0,2000 and is effectively applied to HEALTH plots only):

4. Optional report group definition file <group_name>.grp: