make printable | back to 'other' | close window	new:	see also:
	This page collects information about the getFAP script running on the fors2 account.	databases QC1 database, tables fors2_zp_frame /fors2_photometry/fors2_response dfos tools tool uses processAB and processQC output used by trendPlotter jobs upload/download uploaded HC jobs
getFAP | photometric nights | time window | mirror events | usage | pgi_getNightStatus | stable flag | QC script

getFAP

Description

The tool getFAP is the dfos-compatible version of the tool fors_photometry originally written by S. Moehler. Its functionality is to collect photometry products of processed standard star field images (STD_IMA) in a configurable time window, create and execute ABs (one per filter and detector) for the recipe fors_photometry, and write the output as QC1 parameters into the QC1 database table fors2_photometry. The FAP (FORS absolute photometry) project is described in The Messenger 149, p14. (2012).

It is based on the idea that by collecting photometric data from many nights it should be possible to safely disentangle the true instrumental zeropoint (i.e. a system property where the system is composed of the telescope and the instrument) and the extinction (i.e. a property of the atmosphere). The zeropoint then can be used to monitor the instrument health.

The tool is a workflow tool. Its main functionalities are:

• collect all suitable certified products of pro.catg=ALIGNED_PHOT (code: F2_PZPI; products of the fors2_zeropoint recipe that processes an input STD_IMA file into photometric products, in the course of standard incremental calibration processing)
• put them in a new AB for the recipe fors2_photometry
• execute the AB and ingest the result QC parameters into fors2_photometry
• do this for all 4 standard filters and 2 CCDs

The output is visible on the HC plots for the IMG zeropoints .

Photometric nights. The time window chosen has the width of 28 calendar days, formally symmetrical around the date of the night called (but see below). It turns out that it is important for the accuracy of the results that the input nights have a high photometric stability. To select those, a criterion has been developed which is applied by the helper plugin tool pgi_getNightStatus (find the definition there). For each night with certified photometric data, such a flag exists. The getFAP tool searches, in the selected time window, for stable nights (flag = 'S' or 'Sb') and collects frame measurements for them.

Delay. With a time window of 28 days, the latest date for which a search can be performed is TODAY minus 14. To be on the safe side, the default date for the call is TODAY minus 28 days. Then, with a window size of +/- 14 days, the effective last date is TODAY minus 14d. It is asumed that for this date certified product tables (ALIGNED_PHOT) already exist (which is usually not an issue with standard dfo operations, but occasionally might fail with a backlog).

Time window. The tool can handle non-standard time windows, they are specified by the parameter -D <N_days>. This might be useful if the width is fixed (option -F). If it is not fixed, then it is dynamic under certain circumstances, namely if not enough stable nights could be found. From test reports it is known that a minimum number of roughly 7 nights is required for good results (this number is controlled with the parameter -N). If it is not reached with the standard window size, then the window is increased to a maximum of 90 days (unless the flag -F is set to fix the window).

Mirror events. Another important entity for the time range is a mirror event (recoating). Since then a sudden change of the zeropoint is to be expected, the time window used by the tool is not allowed to cross such a date (breakpoint). Mirror events are expected in the configuration file.

Workspace. In order not to interfere with routine DFOS operations on the FORS2 machine, the tool creates its own workspace. It downloads products, processes them, creates the new coefficients as QC1 parameters and then deletes all output except for the processing log files and the ABs (both going to $DFO_LOG_DIR/FAP/<date>). Most directories are split by 'FAP', in $DFO_CAL_DIR 1999-01-01 is added.

Logs. The tool log is in $DFO_MON_DIR/FAP/GET_FAP. All relevant output (ABs, rblogs, tool log) is exported to qcweb, into .../logs/FAP.

How to use

	getFAP		default date for the nightly zeropoints (TODAY minus 28 days), for cronjob execution
	for command line execution:
	-d <date>		specify date for the nightly zeropoints
	-D <delta>		specify non-default time window (default: +/- delta with delta=14 days)
	-N <nph_min>		specify non-default number of stable nights (default: 7)
	-f <filter>		specify filter (default: any; selection: RSPECIAL IBESS bHIGH vHIGH)
	-c <chip>		specify chip (default: any; selection: C1 C2) (only MIT chips)
	-a		use asymmetric window as in cronjob operations (left boundary defined as usual; right boundary by delta)
	-B		deBug mode: all products kept, no QC1 parameter ingested, no cleaning up
	-F		keep time window fixed (then: all stable nights are collected, disables nph_min)
	-K		keep downloaded ALIGNED_PHOT (F2_PZPI) files in $DFO_CAL_DIR/1999-01-01 (for performance)
	-V		verbose output on the command line
	-v		print version
	-h		print help

For automatic processing, just call 'getFAP'.

For re-processing, use 'getFAP -d <date>' (standard case), or 'getFAP -d <date> -D <delta>' for a non-standard time window, or 'getFAP -d <date> -N <nph_min' for a non-standard number of stable nights, or 'getFAP -d <date> -F' to keep the time window fixed (and not open it dynamically if not enough stable nights exist). You can also combine different methods. You can as well call the tool for a specific filter and chip. Finally, you can call all options together with flag -B (debug), then ABs are processed but no QC1 parameter ingestion occurs, and all products are kept in $DFS_PRODUCT/FAP/STD_IMA for further inspection.

Logging is controlled with option -V (if set, the verbose output goes to the command line and to the log file; otherwise the tool output ends up in the log file).

The tool logs are found in $DFO_MON_DIR/FAP/GET_FAP and are sorted by <date>. All new processing of a <date> is appended to the existing log file.

Operational hints

• The tool creates so-called level-2 ABs (ABs containing products from previous ABs as input files). No raw files are involved. The input products are the F2_PZPI files (ALIGNED_PHOT) which are fits tables.
• The recipe fors2_photometry opens all product files, reads the coefficients, creates linear transformation equations and solves them for a common zeropoint and nightly extinction coefficients. In doing so, it assumes that all flux variations are due to airmass and extinction (constant per night). Hence the independent assessment of the input nights as photometrically stable is important.
• Currently the tool processes ABs sequentially; it could be tuned to execute jobs in parallel

The tool getFAP is currently called once per 24 hours as a fors2 cronjob, as 'getFAP' (i.e. with automatic calculation of the date). In that mode, it also updates the HISTORY of the ZEROPOINT HC files properly. The HEALTH CHECK plot is updated in JOBS_HEALTH, the current HISTORY in JOBS_TREND.

How to install

The tool is installed in the $DFO_BIN_DIR of the fors2 account. It also requires

• the configuration file config.getFAP (see below)
• the QC script photometry_qc.prg (in the standard $DFO_PROC_DIR of fors2)
• the pgi script $DFO_BIN_DIR/pgi_getNightStatus for determining the photometric flag

pgi_getNightStatus

This pgi script is called by the fors2 pgi_move_products script, in a separate workflow step, namely every time when a new night has been certified. This pgi takes care of classifying a night as S(table) or N(ot stable) or U(nknown).

The flag U is applied by default and applicable to all nights without a low-high-airmass pair (and of course to nights without any STD_IMA data). A night satisfying all criteria is flagged S if it has a second dataset ( so that a significant fraction of the night can be assessed), or at least 'Sb' (like stable in the beginning) if only the initial pair exists and satisfies the extinction criterion. A night with a low-high-airmass pair violating the extinction criterium is flagged N.

The flag is updated in the following QC tables:

• fors2_zp_frame
• fors2_photometry (where it is used to mark stable nights in the HC plots)
• fors2_response (where it is not actually used for the HC plots but may be generally useful).

There is also the hidden QC1 table fors2_stablenights which stores the output of the pgi_getNightStatus script.

The tool also writes the flag into a local trending table, $HOME/trending/STABLENIGHTS.DAT where it is expected by getFAP.

Find its workflow here.

photometry_qc.prg

The QC script corresponding to getFAP is $DFO_PROC_DIR/photometry_qc.prg. It calls, in a way which is standard for the fors2 QC system, the qc1Ingest procedure
$HOME/trending/ingest_photometry.awk. Each call of this awk script writes a line into $HOME/SM/ingest_save/ingest_photometry.save. This file could be used if the qc1 database content would need to be modified.

Configuration

The configuration file config.getFAP is expected under the usual directory ($DFO_CONFIG_DIR) and contains coating dates:

Coating event	date	comment
COATING	2012-05-31	UT1-M1
etc.

Don't forget to include a new coating date. Do not include CO2 washing since this kind of intervention does not modify the zeropoint significantly, but would constrain the time window for getFAP unnecessarily.

Workflow

1. Setup

1.1 check coating dates in config file
1.2 setting up workspace

2. Search certified standard star products for selected date (exit if none found)

3. Search for stable nights (stable='S' or 'Sb'

3.1 If nph_min or more found, or -F is set: collect all ALIGNED_PHOT (F2_PZPI) files from stable nights and continue
3.2 otherwise: open the fixed window and search for nph_min closest stable nights, collect all ALIGNED_PHOT files from them

4. AB creation and processing

4.1 Create ABs per filter and chip, with all F2_PZPI files as input, and proper gencalibs as MCALIB; put them into $DFO_AB_DIR/FAP
4.2 Create JOBS_FAP_<date> file
4.3 Call it (AB processing and QC reporting unless option -B is set)
4.4 QC reporting ingests new QC1 parameters into fors2_photometry (old ones are overwritten)

5. Cleaning up

5.1 ABs and logs are preserved (in $DFO_LOG_DIR/FAP and $DFO_MON_DIR/FAP, resp.)
5.2 fits files are deleted

Workflow for pgi_getNightStatus

1. read the frame extinction coefficients from the QC1 database table fors2_zp_frame for specified night

2. apply 'stable' criteria (see here)

3. Return S if all criteria are met; Sb if only the initial pair exists and the other criteria are met; N if they are violated; U if not enough measurements are available (delta_airmass criterium not satisfied)

4. Update the 'stable' flag in fors2_zp_frame, fors2_photometry, and fors2_response.

Types of photometric product files: