Common Trending and QC tools:
Documentation

tqs = Trending and Quality Control System
make printable new: see also:
 

v1.4:
- supports KPI plots (interprets config file section 4.7 with REF_VALUES)

all trendPlotter config files

trendPlotter tool
trendPlotter operations

matplotlib documentation here

topics: COMPUTE and FIT rules | axis labels | installation

The python engine for trendPlotter: PET

Purpose

PET is the Python Engine for trendPlotter. Since summer 2012 it replaces the MIDAS plot engine which was used in trendPlotter before version 3.0. It has been developed by Burkhard Wolff.

It is not a stand-alone tool but works exclusively as plugin to trendPlotter. Therefore the following documentation does not cover operational issue but only configuration details, in particular related to the syntax for the COMPUTE and FIT rules.

Syntax changes from MIDAS to python:
[ top ] COMPUTE and FIT rules

Compute and fit rules (identifies by COMPUTE_RULE and FIT_RULE in the config.tp_ files) have to be rewritten into Python syntax. Old MIDAS syntax is ignored; the plot will be created but the result will not look as desired, and an email warning will be sent by the tool. (In the following, the term RULES is used for COMPUTE_RULEs and FIT_RULES).

Any Python statement that can be written in one line of code can be used to describe the RULES. If a statement cannot be comfortably written into one line, use a file to store the statements and refer to it with the execfile command (see below).

The x, y, and time values for each parameter set can always be accessed via the names xdat, ydat, and tdat.

Some examples are presented here. If necessary, a 'translation' of the python code is added as last line (the documentation of all matplotlib commands is available here). In some cases, the statements extend over two lines for better readability. In the actual configuration, only one line must be used.

Multiplying the y values of the data points by 1000
MIDAS COMPUTE_RULE CONV2 &&comp/tab plt_ch2.tbl #2 = #2*1000.0&&
PYTHON COMPUTE_RULE CONV2 &&ydat = 1000.0*ydat&&

Correct zeropoints from ADUs to electrons
MIDAS COMPUTE_RULE CORR_Q1 &&comp/tab new3Q1.tbl :zeropoint = :zeropoint + 2.5 * log10(1.75)&&
PYTHON COMPUTE_RULE CORR_Q1 &&ydat = ydat + 2.5 * math.log10(1.75)&&

Plot a linear function (x values from 0 to 60000, slope 0.0157, black colour, dotted line)
MIDAS FIT_RULE FITLINQ1 &&cre/ima linQ1 1,60 0,1000 POLY 0,1 ; comp/ima linQ1 = 0.0157 * linQ1 ; over/row linQ1 ? ? ? 2&&
PYTHON FIT_RULE FITLINQ1 &&pylab.plot([0,60000],[0,0.0157*60000],'k:')&&
      <<pylab.plot([x1,x2],[y1,y2],'black, dotted')>>
E.g. in GIRAFFE DETNOISE plot.  

Plot a square root
MIDAS COMPUTE_RULE FITSQRTQ1 &&cre/ima rootQ1 1,60 0,1000 POLY 0,1 ; comp/ima rootQ1 = sqrt(rootQ1/1.75) ; over/row rootQ1 ? ? ? 2&&
PYTHON COMPUTE_RULE FITSQRTQ1 &&xvals=numpy.arange(0,60000,1000,dtype=numpy.float32); yvals=numpy.sqrt(xvals/1.75); pylab.plot(xvals,yvals,'k:')&&
     

<<X values=numpy.arange(from,to,steps,data type=numpy.float32); Y values=numpy.sqrt(xvals/1.75); plot(xvals,yvals,'black, broken line')>>
E.g. in GIRAFFE DETNOISE plot.


Plot an additional vertical line to mark an MJD value:
PYTHON FIT_RULE MJDMARK &&pylab.plot([55814,55814],[0,0.22],'b-')&&
      <<plot line([X1,X2],[Y1,Y2],'blue solid line'>>
E.g. in many WISQ plots.

One can also use multi-valued Python lists to simultaneously plot many lines on a trending plot. For example, to mark horizontal lines indicating detectors number 65, 69, 73, 77, 81, 85, 89, and 93 one can do the following:

plot many horizontal lines in one go:
PYTHON FIT_RULE FIT_LINES &&hline = [65, 69, 73, 77, 81, 85, 89, 93]; pylab.plot([0,60000],[hline, hline], 'k:')&&
      <<plot line([X1,X2],[Y1,Y2],'black dotted line' where Y1=65,69,etc.>>

Labels

Plot an additional label in relative figure coordinates (from 0 to 1):
PYTHON FIT_RULE LAB1 &&pylab.figtext(0.1,0.9,'fixed-pattern noise',color='r')&&
      <<x_start,y_start in figure coordinates,'text','red'>>
Plot a label with the current median of the data values:
    LAB2 &&pylab.figtext(0.1,0.9,'median:'+str('%5.2f'%average)+'min',color='r')&&
or:
&&pylab.figtext(0.1,0.9,'median: %5.2f min'%average,color='r')&&
      <<x_start,y_start in figure coordinates,text plus formatted value of 'average' variable in pet.py,'red'>>
This trick is used in the WISQ exectimes plots .

Plot an additional label in data coordinates:
PYTHON FIT_RULE LAB2 &&pylab.text(50000,120,'photon noise',color='g')&&
      <<X_start, y_start in data coordinates,'text','green')>>

External files and programmes
If there are many plotting statements (more than can easily be accommodated in one line), one might collect them in an external file and execute it as Python code (it's a good idea to give it the extension .py and keep it under $DFO_CONFIG_DIR/trendPlotter):

Call external python command file:
PYTHON FIT_RULE PLOTALL

&&execfile('$DFO_CONFIG_DIR/trendPlotter/lots_of_plot_commands.py')&&

For example:

Plot lines in relative figure coordinates (from 0 to 1):
PYTHON FIT_RULE FIT_CLOSEUP &&execfile('$DFO_CONFIG_DIR/trendPlotter/ambient_PWV_closeup.py')&&
      in ambient_PWV_closeup.py:
      pylab.axes(main_ax)
pylab.plot([0.47,0.55],[0.185,0.185],'r:')
pylab.plot([0.47,0.55],[0.360,0.070],'r:')
pylab.plot(ax)
      Used in PWV monitor for the red broken lines

You may want to use the FIT_RULE, not to create a fit on the plot, but to execute an external program, e.g. to modify or create the data points before plotting:

Call external procedure:
MIDAS FIT_RULE FIT_PLOT4 &&$DFO_BIN_DIR/scores_for_trendplot.py 0 qc_raw_dome_median omegacam_dome r_SDSS&&
PYTHON FIT_RULE FIT_PLOT4 &&os.system('$DFO_BIN_DIR/scores_for_trendplot.py 0 qc_raw_dome_median omegacam_dome r_SDSS')&&

[ top ] Axis labels, tick formatting, mathematical symbols, plot symbols

The PLOT_NAME definitions in the configuration files can remain unchanged in general. XAXIS and YAXIS tick marks are automatically translated. If you have specified AUTO here, the Python defaults may be a bit different. You may wish to specify non-default values here.

For XFORMAT and YFORMAT, only C-like syntax (e.g. '3.1f') is understood. FORTRAN-like syntax (e.g. 'F3.1') has to be translated.

Mathematical symbols in XLABEL and YLABEL can be coded with TEX syntax, embraced in dollar signs. A 'SPACE' is automatically replace by ' ':

#PLOT_NAME   P_INDEX PLOT_NAME XFORMAT XLABEL  YFORMAT YLABEL                    XAXIS   YAXIS   PLABEL     
PLOT_NAME    1       Q2        AUTO    NONE    AUTO    $R=\lambda/\Delta\lambda$ 100,20  0,3000  Q2      
PLOT_NAME 1 Q2 AUTO NONE AUTO $\lambda\SPACE/\SPACE\Delta\lambda$ 100,20 0,3000 Q2

The plot SYMBOLs are translated automatically within the tool. In most cases, MIDAS and Python symbols are basically the same but differences may be observed for some. Please note also that open MIDAS symbols (like an open circle) are transparent whereas the Python equivalents are filled with white colour.

The plot sizes and colors are also translated automatically. For the colors, the following mapping is used:

MIDAS PYTHON color
0 w white
1 k black
2 r red
3 g green
4 b blue
5 y yellow
6 m magenta
7 c cyan
8 w white

[ top ] How to install

pet.py is installed with the standard mechanism, 'dfosInstall -t pet.py.

pet.py uses the python 2D plotting library matplotlib. Find the matplotlib documentation here. It is part of the standard python installation.


Last update: April 26, 2021 by bwolff