Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO
*make printable new: see also:
 

v1.5:
- HdrDownloader called in separate directory

- output of this tool is operationally relevant for Paranal QC!
- the tool is closely linked to calChecker
 

Note: To enable calChecker for display of the tellTracker output page, you must set CHECK_TELLTRACK to YES in its config.calChecker file.

 

[ used databases ] databases sched_rep for SM_VM flag
[ used dfos tools ] dfos tools called fitsreport, HdrDownloader, getObInfo; output linked to calChecker
[ output used by ] output HTML output exported to qcweb
[ output used by ] upload/download download: twilight information from skycalc
upload: HTML output exported to qcweb
javascript javascripts jquery-latest.js, jquery.tablesorter.js, jquery.dataTables.js under ${JSCRIPT_URL}
topics: description | rules | results: issues overview enhancements all_matches | update | performance | output | how to use | configuration | workflow

tellTracker

You may also want to read the information linked to the public result page (here) and the general calChecker home page.

[ top ] Description

The tellTracker monitor is a plugin to the calChecker monitor. It tracks the completeness of time-critical nighttime calibrations, like telluric standard stars, for all science OBs of the indicated date.

This plugin is only enabled for certain instruments with red or infrared spectroscopy. When enabled, you find it linked to the main calChecker output like this:

 science    cal4cal     tellTracker   

[ top ] Rules

For all configured science data types, a matching telluric standard star is searched for. Matching is based on header information only. Unlike calChecker, the tellTracker accepts matches only for the same night. No quality information can be taken into account. This means that for instance a saturated telluric standard star cannot be recognized as such and will be used by the tool for a formally correct match.

After evaluation of the formal matches, two constraints are evaluated:

The configured values are indicated. Typical values would be 0.0875 (2 hr) for the time constraint, and 0.2 for the airmass constraint.

All configured science files are evaluated. This is generally a subsection of all science files measured during the night, because:

[ top ] Results

Results are displayed in terms of flags, in the last column RESULT. Note that tellTracker analyzes each single SCIENCE file (while calChecker analyzes ABs, regarding only the first file in a template). Like calChecker, the tellTracker has three flags, OK, NOK, MISS:

NOK or MISS flags are further analyzed in the last column. Here, a label LEFT+79 indicates that 79 minutes are still left (defined by the time constraint) for taking a new telluric standard which would then improve the NOK or MISS flag to OK. A label LOST-13 indicates that the time window has already expired, but still it might be reasonable to take a standard star, e.g. in order to improve a MISS to a NOK case. A label LOST (without a number) indicates that there is no way to improve the result since the night is over.

If only one constraint is violated (combinations 'OK NOK' or 'NOK OK') preference for display is given to the time constraint, i.e. if there is one 'OK NOK' and one 'NOK OK' record, the former is displayed and evaluated.

During the night, there is in the update section some information regarding the remaining time until morning twilight starts.

If there is an unrecoverable issue for the current last night, a flag ! is displayed on the tellTracker tab.

[ top ] Issues. An issue is defined as a NOK or MISS case. All issues, if any, are collected in the top part. That table is intended to provide a list of telluric standards to be taken. It is aggregated per combination of OBS_ID and setup. If a certain science OB has two different setups (e.g. XSHOOTER: 0.9x11_NIR and 100k/1pt/hg_0.9x11_VIS) and a total of 10 science files, there will be two entries in the issues section, one for the NIR and one for the VIS setup. If there are three OBs with two different setups with missing telluric calibration, there will be 6 entries in the issue table since the OBs might belong to different pointings and therefore to different airmass values.

If within an OB there are some files with 'OK NOK' and some with 'NOK OK', preference is given to the 'NOK OK' cases (time constraint violated but airmass ok) or to 'NOK NOK' since these are also found by calChecker. The 'OK NOK' issues (meaning AIRMASS is violated but time is ok) are also displayed as such but only if there is no 'NOK OK' issue.

The issues section is split into two parts, one for the recoverable issues (NOK+LEFT; MISS+LEFT; MISS+LOST) and one for the non recoverable issues (NOK+LOST). The first section should be used for scheduling the next telluric standard, the second section for OB comments or grades in the nightlog. The issue table has columns LST, RA and DEC to be used for input in the telluric star tool. The column UT is a short version of the ARCFILE column in the overview.

[ top ] Overview. The overview table is complete for all science files of the night. It has the same columns as the issues table, plus a few more like ARCFILE, DATE, and DATA_TYPE. This table has the science and, for orientation, also the telluric standard stars listed. The latter objects are linked to their score page (if that is available at runtime).

The tables are sortable. Press Shift key for multiple column sorting. Default sorting is by UT (issues) and ARCFILE (overview). Restore default sorting with the browser refresh button.

Select SETUP and then ARCFILE to sort all TELL and science files per setup.

[ top ] Table enhancements. The table output has features which go beyond HTML functionality and are provided by javascripts. Such enhancements are:

Filtering and sorting can be combined and can be useful for research on an issue. E.g. enter '50333' and see all files belonging to that OB_ID. Enter'TELL04' and see all files associating to that telluric star, including the TELL file itself. Then sort the output by AIRM, or dAirm. Do not forget to disable browser refresh for longer analysis (and to enable it again afterwards).

More technicalities about dataTables here.

Find an example output table here (click on any column to sort, enter a string to search/filter, refresh the browser to get back the original content):


Complete overview:  [issues]   [all matches]   top   bottom   [page auto-refreshes after 300 sec]   [stop | on]

Toggle: LST+RA+DEC | AIRM | DATA_TYPE | all of those

PROG_ID M OBS_ID GRD LST RA DEC AIRM ARCFILE DATA TYPE SETUP RESULT
FLAG

TELL

dTime

dAirm

remain
086.B-0407(A)SM501007A!08:06:59.93609:59:21.902:06:40.61.24XSHOO.2011-03-27T00:32:16.538.fitsSCI_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS OKTELL01 0.0410.02    
086.B-0407(A)SM501007A!08:32:32.11909:59:21.602:06:37.01.19XSHOO.2011-03-27T00:57:44.787.fitsSCI_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS OKTELL01 0.0230.03    
085.D-0144(A)SM442926A13:57:15.31813:20:44.800:18:53.31.12XSHOO.2011-03-27T06:21:34.766.fitsSCI_SLIT_VIS100k/1pt/hg_0.9x11_VIS OKTELL03 0.0400.07    
085.D-0144(A)SM442926A14:10:08.42813:20:44.700:18:57.61.13XSHOO.2011-03-27T06:34:25.471.fitsSCI_SLIT_VIS100k/1pt/hg_0.9x11_VIS OKTELL03 0.0320.06    
085.D-0144(A)SM442926A14:22:55.52313:20:44.700:18:57.61.15XSHOO.2011-03-27T06:47:11.025.fitsSCI_SLIT_VIS100k/1pt/hg_0.9x11_VIS OKTELL03 0.0230.04    
085.D-0144(A)SM442926A14:35:49.63713:20:44.800:18:53.31.17XSHOO.2011-03-27T07:00:02.810.fitsSCI_SLIT_VIS100k/1pt/hg_0.9x11_VIS OKTELL03 0.0140.02    
386.A-0620(A)SM503333A!15:09:25.14013:11:30.1-1:19:20.11.25XSHOO.2011-03-27T07:33:32.648.fitsSCI_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS NOKTELL01 0.2520.03LOST
386.A-0620(A)SM503333A!15:16:02.22413:11:29.9-1:19:16.71.27XSHOO.2011-03-27T07:40:08.661.fitsSCI_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS NOKTELL01 0.2560.05LOST
086.B-0407(A)SM501007A!08:07:04.95009:59:21.902:06:40.61.24XSHOO.2011-03-27T00:32:19.772.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL02 0.0410.02    
086.B-0407(A)SM501007A!08:32:37.13309:59:21.602:06:37.01.18XSHOO.2011-03-27T00:57:47.840.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL02 0.0230.04    
085.D-0144(A)SM442926A13:57:21.33413:20:44.800:18:53.31.12XSHOO.2011-03-27T06:21:39.265.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0400.07    
085.D-0144(A)SM442926A14:10:13.44213:20:44.700:18:57.61.13XSHOO.2011-03-27T06:34:28.382.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0320.06    
085.D-0144(A)SM442926A14:23:00.53713:20:44.700:18:57.61.15XSHOO.2011-03-27T06:47:14.131.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0230.04    
085.D-0144(A)SM442926A14:35:54.65013:20:44.800:18:53.31.17XSHOO.2011-03-27T07:00:05.866.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0140.02    
386.A-0620(A)SM503333A!15:09:30.15413:11:30.1-1:19:20.11.25XSHOO.2011-03-27T07:33:35.688.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0100.06    
386.A-0620(A)SM503333A!15:16:07.23813:11:29.9-1:19:16.71.28XSHOO.2011-03-27T07:40:11.615.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0140.09    
386.A-0620(A)SM503333A!15:22:42.31713:11:29.9-1:19:16.51.30XSHOO.2011-03-27T07:46:45.547.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0190.11    
386.A-0620(A)SM503333A!15:29:19.40113:11:30.1-1:19:18.91.32XSHOO.2011-03-27T07:53:21.472.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0230.13    
386.A-0620(A)SM503333A!15:35:57.48813:11:30.1-1:19:19.51.35XSHOO.2011-03-27T07:59:58.768.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL04 0.0280.16    
386.A-0620(A)SM503333A!15:42:34.57213:11:29.8-1:19:15.41.38XSHOO.2011-03-27T08:06:34.692.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL06 0.0270.11    
386.A-0620(A)SM503333A!15:49:11.65613:11:29.9-1:19:16.01.41XSHOO.2011-03-27T08:13:10.617.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL06 0.0220.08    
386.A-0620(A)SM503333A!15:55:48.74013:11:30.1-1:19:19.31.45XSHOO.2011-03-27T08:19:46.541.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL06 0.0180.04    
386.A-0620(A)SM503333A!16:02:26.82713:11:30.1-1:19:19.01.49XSHOO.2011-03-27T08:26:23.132.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL06 0.0130.00    
386.A-0620(A)SM503333A!16:09:03.91213:11:29.9-1:19:15.81.53XSHOO.2011-03-27T08:32:59.762.fitsSCI_SLIT_NIR0.9x11_NIR OKTELL06 0.0080.04    
60.A-9022(C)200193515X!09:05:47.56810:08:48.406:43:02.81.22
XSHOO.2011-03-27T01:30:54.734.fits0/6
STD_TELL_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS  Hip049704 TELL01  
60.A-9022(C)200193515X!14:55:35.87614:38:48.207:54:39.81.19
XSHOO.2011-03-27T07:19:46.149.fits0/6
STD_TELL_SLIT_VIS100k/1pt/hg_0.9x11_VIS  Hip071615 TELL03  
60.A-9022(C)200193515X!16:21:01.87212:47:53.6-24:51:04.51.49
XSHOO.2011-03-27T08:44:57.927.fits0/6
STD_TELL_SLIT_VIS100k/1pt/hg/1x2_0.9x11_VIS  Hip062448 TELL05  
60.A-9022(C)200193515X!09:05:52.58210:08:48.406:43:02.81.22
XSHOO.2011-03-27T01:30:58.135.fits1/3
STD_TELL_SLIT_NIR0.9x11_NIR  Hip049704 TELL02  
60.A-9022(C)200193515X!14:55:41.89214:38:48.207:54:39.81.19
XSHOO.2011-03-27T07:19:49.558.fits1/3
STD_TELL_SLIT_NIR0.9x11_NIR  Hip071615 TELL04  
60.A-9022(C)200193515X!16:21:06.88612:47:53.6-24:51:04.51.49
XSHOO.2011-03-27T08:45:01.320.fits1/3
STD_TELL_SLIT_NIR0.9x11_NIR  Hip062448 TELL06  
PROG_ID M__ OBS_ID__ GRD__ LST RA DEC AIRM__ ARCFILE DATA TYPE SETUP__ RESULT
FLAG

TELL

dTime

dAirm

remain


[ top ] Table with all matches. This additional output table provides a list of all possible TELL matches. It is thought to provide details in case of NOK issues. For instance, the main page lists the best possible NOK-OK match (dTime violated but dAirm ok), while the all-matches table also lists the other telluric matches, including the ones flagged OK-NOK. Find its link here: [all matches].


[ top ] Update information and documentation

Last update: 2017-02-22T07:12:07 (UT)
TWILIGHT morning:
2011-02-22T09:05 (UT)
Configured validity dTime:
2.0 hr (0.083 d)
Last header: XSHOO. 2017-02-22T05:21:51.420.hdr
next:
2:37 hr
Configured difference in airmass dAirm:
0.2

Last update: timestamp of creation of this page (UT)

Last header: name of last header file currently available; compare this timestamp to 'Last update' to see if monitor is up-to-date

Frequency: The tool is scheduled by a cronjob and is running during the Paranal night, from about 22:00 UT until about 11:00 UT, every 15 minutes.

[ top ] Performance

As a specialized tool, tellTracker is tuned for good performance. Instead of OCA, it uses simple pattern matching (unix grep) for classification and association (like the long-term calibration section of calChecker). All configured keys are read with fitsreport into a single table which is then analyzed for data types, matches and setups. tellTracker typically executes in less than a minute. Its execution times are written in the QC1 database table exec_time and monitored for the case of XSHOOTER here.

[ top ] Output

[ top ] How to install

This tool is not part of the standard dfos tool suite. Download the installation tarball from http://www.eso.org/~qc/dfos/ or directly from this link into $DFO_INSTALL_DIR/tellTracker. Then:

cd $DFO_INSTALL_DIR/tellTracker
tar xvf tellTracker.tar
cat README_tellTracker
mv tellTracker $DFO_BIN_DIR

The following should be done only if this is a new installation (not for updates!):

mv config.tellTracker tellTracker.cfg tellTracker_radec.cfg $DFO_CONFIG_DIR

Read the README file and follow the instructions. They are standard and straightforward.

[ top ] How to use

Type tellTracker -h | -v for on-line help and version number. Type

tellTracker

to create the output (per default for $TODAY).

tellTracker -d(ate): for a specified date

tellTracker -N: ($TODAY, no header download)

tellTracker -d(ate) -O: (specified date, with header download)

tellTracker -V(erbose): verbose output (otherwise the tool is silent)

The verbose output goes to the terminal. It displays all matches per science file and may be useful for analyzing issues.

Note that for $TODAY the default behaviour is header download (with HdrDownloader), while for older dates the default is no header download (since those header directories are assumed to be complete anyway).

[ top ] Configuration files

The tool has

1. The tool configuration config.tellTracker defines:

Section 1: general parameters
RANGE
120 validity range for telluric matches in minutes (beyond +/-$RANGE minutes a match will be considered NOK)
D_AIRM
0.22 airmass range for telluric matches (delta_airm beyond +/-D_AIRM will be considered NOK)

Section 2: 2. Classification and association

2.1 Classification: Definition of SCIENCE and TELLURIC data types

Define each DATA_TYPE (SCIENCE or TELLURIC) here; take the file behind "ASSOC-RULES"
as reference; pattern matching is based on the output of the tellTracker.cfg fitsreport file;
the 'egrep' part stands for alternative matches (OR), while all 'grep' parts are a logical 'AND'
Enclose DEFINITION by '&&', '&&' must be last characters in the row.
Each SCI_DATA_TYPE and TELL_DATA_TYPE in section 2.2 must correspond to a NAME here.

DATA_TYPE
NAME DEFINITION
DATA_TYPE SCI_SLIT_VIS &&grep " VIS " | egrep "OBJECT ECHELLE,SLIT,NODDING|OBJECT ECHELLE,SLIT,STARE|OBJECT ECHELLE,SLIT,OFFSET|SKY ECHELLE,SLIT,OFFSET" | grep SCIENCE&&
DATA_TYPE
STD_TELL_SLIT_VIS &&grep " VIS " | grep STD,TELLURIC | egrep " 2_SEQ| 3_SEQ" | egrep "ECHELLE,SLIT,NODDING|ECHELLE,SLIT,STARE"&&
etc.    

2.2 Association

Define for each DATA_TYPE the associated TELL_DATA_TYPE and the matchkey;
- all indexes refer to the fitsreport output list of tellTracker.cfg;
- make sure to arrange the keys sequentially, they are matched in a single 'grep'.
MATCH_SETUP: index(es) of key(s) used to find matching TELL_DATA_TYPE
DISPLAY_SETUP: index(es) of key(s) for defining DATA_TYPE setup

TELL_MATCH

SCI DATA_TYPE TELL_DATA_TYPE MATCH_SETUP DISPLAY_SETUP
TELL_MATCH
SCI_SLIT_VIS STD_TELL_SLIT_VIS $14,$15 $14,$15
etc.        

 

The following lines are auto-inserted by tellTracker, to avoid unnecessary checks for directories on safweb1 (don't remove):

# ======================================================================================
# DO NOT REMOVE: auto-added by tellTracker!
TELLTRACK_EXIST YES # flag to indicate that /home/qc/public_html/XSHOOTER/CALCHECK/TELLTRACK exists on qc@safweb1

2. Two configuration files are needed to define the classification and association keys, as well as the keys for the output table. One is called tellTracker.cfg, the other one tellTracker_radec.cfg. The latter file contains RA, DEC, and LST which are always used and need a fixed format and conversion. This file does not need any attention by the QC scientist. The other file comes as a fitsreport config file with minimum content. It has two parts, the first part being a standard part not to be modified by the user, and a second part for the instrument-specific setup definition:

General keys, always needed
ARCFILE display
DPR.CATG needed for classification
DPR.TYPE
DPR.TECH
MJD-OBS needed for time matching
TEL.AIRM.START needed for the airmass constraint; the mean airmass is evaluated
TEL.AIRM.END
OBS.PROG.ID used for display and for 60.A suppression
SETUP keys, user controlled; e.g.:
SEQ.ARM these are typical keys for setup definition. Their index (e.g. $13) is used in section 2.1 of config.tellTracker to define matchkeys.
DET.READ.CLOCK
INS.OPTI2.NAME
etc.

As usual with fitsreport config files, you can e.g. define 'Delimiter' strings to make selection of these keys by unix grep easier. See more in the fitsreport documentation.

3. There is an optional but recommended info file, info.tellTracker under $DFO_CONFIG_DIR. Its text is displayed on top of the page, right part. Its content should be similar to the info file for calChecker.

[ top ] Workflow

Details:

  1. $DATE=$TODAY: download headers (unless disabled) with 'HdrDownloader -d $TODAY' (into $DFO_HDR_DIR/TELLTRACK to avoid interference with operational headers)
  2. Classification
    2.1 call fitsreport and classify all files
    2.2 get OB grades, comments,
    SM/VM flags
  3. Association
    3.1 read SCI_DATA_TYPE, TELL_MATCH, SCI_DEFSTRING, TELL_DEFSTRING from config.tellTracker
    3.2 loop across SCI_DATA_TYPEs
    3.3 get closest-in-time match of TELL and SCIENCE, calculate deltas, evaluate OK/NOK/MISS
  4. Create HTML output tables
    4.1 extract issues per OB and setup
    4.2 provide full overview for science
    4.3 add telluric standard stars
  5. scp result HTML file to qcweb

Operational hints:

- The tool will download headers with 'HdrDownloader -d $TODAY', without suppressing the last (possibly) incomplete template; in order to avoid unwanted effects with autoDaily (incomplete ABs), its headers are downloaded into $DFO_HDR_DIR/TELLTRACK; the call of 'HdrDownloader -d' is incremental, i.e. only new headers are downloaded.

- Make sure to set up the cronjob file properly: call the tool like

01,16,31,46 00-11,22,23 * * * . /home/xshooter/.qcrc; /home/xshooter/bin/tellTracker

- Make sure to call tellTracker before autoDaily: if not it will possibly interfer with filterRaw (as triggered by autoDaily) which means it will search for headers which are temporarily not available.


Last update: April 26, 2021 by rhanusch