Common Trending and QC tools: |
tqs = Trending and Quality Control System |
make printable | new: | see also: |
v1.4: |
all trendPlotter config files matplotlib documentation here |
|
topics: COMPUTE and FIT rules | axis labels | installation |
Purpose
PET is the Python Engine for trendPlotter. Since summer 2012 it replaces the MIDAS plot engine which was used in trendPlotter before version 3.0. It has been developed by Burkhard Wolff.
It is not a stand-alone tool but works exclusively as plugin to trendPlotter. Therefore the following documentation does not cover operational issue but only configuration details, in particular related to the syntax for the COMPUTE and FIT rules.
Syntax changes from MIDAS to python:
COMPUTE
and FIT rules
Compute and fit rules (identifies by COMPUTE_RULE and FIT_RULE in the config.tp_ files) have to be rewritten into Python syntax. Old MIDAS syntax is ignored; the plot will be created but the result will not look as desired, and an email warning will be sent by the tool. (In the following, the term RULES is used for COMPUTE_RULEs and FIT_RULES).
Any Python statement that can be written in one line of code can be used to describe the RULES. If a statement cannot be comfortably written into one line, use a file to store the statements and refer to it with the execfile command (see below).
The x, y, and time values for each parameter set can always be accessed via the names xdat, ydat, and tdat.
Some examples are presented here. If necessary, a 'translation' of the python code is added as last line (the documentation of all matplotlib commands is available here). In some cases, the statements extend over two lines for better readability. In the actual configuration, only one line must be used.
Multiplying the y values of the data points by 1000 | |||
MIDAS | COMPUTE_RULE | CONV2 | &&comp/tab plt_ch2.tbl #2 = #2*1000.0&& |
PYTHON | COMPUTE_RULE | CONV2 | &&ydat = 1000.0*ydat&& |
Correct zeropoints from ADUs to electrons | |||
MIDAS | COMPUTE_RULE | CORR_Q1 | &&comp/tab new3Q1.tbl :zeropoint = :zeropoint + 2.5 * log10(1.75)&& |
PYTHON | COMPUTE_RULE | CORR_Q1 | &&ydat = ydat + 2.5 * math.log10(1.75)&& |
Plot a linear function (x values from 0 to 60000, slope 0.0157, black colour, dotted line) | |||
MIDAS | FIT_RULE | FITLINQ1 | &&cre/ima linQ1 1,60 0,1000 POLY 0,1 ; comp/ima linQ1 = 0.0157 * linQ1 ; over/row linQ1 ? ? ? 2&& |
PYTHON | FIT_RULE | FITLINQ1 | &&pylab.plot([0,60000],[0,0.0157*60000],'k:')&& |
<<pylab.plot([x1,x2],[y1,y2],'black,
dotted')>> E.g. in GIRAFFE DETNOISE plot. |
Plot a square root | |||
MIDAS | COMPUTE_RULE | FITSQRTQ1 | &&cre/ima rootQ1 1,60 0,1000 POLY 0,1 ; comp/ima rootQ1 = sqrt(rootQ1/1.75) ; over/row rootQ1 ? ? ? 2&& |
PYTHON | COMPUTE_RULE | FITSQRTQ1 | &&xvals=numpy.arange(0,60000,1000,dtype=numpy.float32); yvals=numpy.sqrt(xvals/1.75); pylab.plot(xvals,yvals,'k:')&& |
<<X values=numpy.arange(from,to,steps,data
type=numpy.float32); Y values=numpy.sqrt(xvals/1.75); plot(xvals,yvals,'black, broken
line')>> |
Plot an additional vertical line to mark an MJD value: | |||
PYTHON | FIT_RULE | MJDMARK | &&pylab.plot([55814,55814],[0,0.22],'b-')&& |
<<plot line([X1,X2],[Y1,Y2],'blue solid
line'>> E.g. in many WISQ plots. |
One can also use multi-valued Python lists to simultaneously plot many lines on a trending plot. For example, to mark horizontal lines indicating detectors number 65, 69, 73, 77, 81, 85, 89, and 93 one can do the following:
plot many horizontal lines in one go: | |||
PYTHON | FIT_RULE | FIT_LINES | &&hline = [65, 69, 73, 77, 81, 85, 89, 93]; pylab.plot([0,60000],[hline, hline], 'k:')&& |
<<plot line([X1,X2],[Y1,Y2],'black dotted line' where Y1=65,69,etc.>> |
Labels
Plot an additional label in relative figure coordinates (from 0 to 1): | |||
PYTHON | FIT_RULE | LAB1 | &&pylab.figtext(0.1,0.9,'fixed-pattern noise',color='r')&& |
<<x_start,y_start in figure coordinates,'text','red'>> | |||
Plot a label with the current median of the data values: | |||
LAB2 | &&pylab.figtext(0.1,0.9,'median:'+str('%5.2f'%average)+'min',color='r')&& or: &&pylab.figtext(0.1,0.9,'median: %5.2f min'%average,color='r')&& |
||
<<x_start,y_start in figure
coordinates,text plus formatted value of 'average' variable in pet.py,'red'>> This trick is used in the WISQ exectimes plots . |
Plot an additional label in data coordinates: | |||
PYTHON | FIT_RULE | LAB2 | &&pylab.text(50000,120,'photon noise',color='g')&& |
<<X_start, y_start in data coordinates,'text','green')>> |
External files and programmes
If there are many plotting statements (more than can easily be accommodated in one line),
one might collect them in an external file and execute it as Python code (it's a good
idea to give it the extension .py and keep it under $DFO_CONFIG_DIR/trendPlotter):
Call external python command file: | |||
PYTHON | FIT_RULE | PLOTALL |
&&execfile('$DFO_CONFIG_DIR/trendPlotter/lots_of_plot_commands.py')&& |
For example:
Plot lines in relative figure coordinates (from 0 to 1): | |||
PYTHON | FIT_RULE | FIT_CLOSEUP | &&execfile('$DFO_CONFIG_DIR/trendPlotter/ambient_PWV_closeup.py')&& |
in ambient_PWV_closeup.py: | |||
pylab.axes(main_ax) pylab.plot([0.47,0.55],[0.185,0.185],'r:') pylab.plot([0.47,0.55],[0.360,0.070],'r:') pylab.plot(ax) |
|||
Used in PWV monitor for the red broken lines |
You may want to use the FIT_RULE, not to create a fit on the plot, but to execute an external program, e.g. to modify or create the data points before plotting:
Call external procedure: | |||
MIDAS | FIT_RULE | FIT_PLOT4 | &&$DFO_BIN_DIR/scores_for_trendplot.py 0 qc_raw_dome_median omegacam_dome r_SDSS&& |
PYTHON | FIT_RULE | FIT_PLOT4 | &&os.system('$DFO_BIN_DIR/scores_for_trendplot.py 0 qc_raw_dome_median omegacam_dome r_SDSS')&& |
Axis labels, tick formatting, mathematical symbols, plot symbols
The PLOT_NAME definitions in the configuration files can remain unchanged in general. XAXIS and YAXIS tick marks are automatically translated. If you have specified AUTO here, the Python defaults may be a bit different. You may wish to specify non-default values here.
For XFORMAT and YFORMAT, only C-like syntax (e.g. '3.1f') is understood. FORTRAN-like syntax (e.g. 'F3.1') has to be translated.
Mathematical symbols in XLABEL and YLABEL can be coded with TEX syntax, embraced in dollar signs. A 'SPACE' is automatically replace by ' ':
#PLOT_NAME P_INDEX PLOT_NAME XFORMAT XLABEL YFORMAT YLABEL XAXIS YAXIS PLABEL PLOT_NAME 1 Q2 AUTO NONE AUTO $R=\lambda/\Delta\lambda$ 100,20 0,3000 Q2
PLOT_NAME 1 Q2 AUTO NONE AUTO $\lambda\SPACE/\SPACE\Delta\lambda$ 100,20 0,3000 Q2
The plot SYMBOLs are translated automatically within the tool. In most cases, MIDAS and Python symbols are basically the same but differences may be observed for some. Please note also that open MIDAS symbols (like an open circle) are transparent whereas the Python equivalents are filled with white colour.
The plot sizes and colors are also translated automatically. For the colors, the following mapping is used:
MIDAS | PYTHON | color |
0 | w | white |
1 | k | black |
2 | r | red |
3 | g | green |
4 | b | blue |
5 | y | yellow |
6 | m | magenta |
7 | c | cyan |
8 | w | white |
pet.py is installed with the standard mechanism, 'dfosInstall -t pet.py.
pet.py uses the python 2D plotting library matplotlib. Find the matplotlib documentation here. It is part of the standard python installation.
Last update: April 26, 2021 by bwolff |