make printable	new:	see also:
	v1.4: - supports both DFOS and PHOENIX installations	The tool checks for open ingestion requests and sends a warning if found.
	v1.5: - downloads headers and checks for completeness	databases none (updates of daily_stat through extractStat) dfos tools calling: extractStat | createReport, histoMonitor | HdrDownloader called by: dfoMonitor   output products of dfos tools called upload/download qcweb: calling webFinishNight - deletes AB status files under monitor

finishNight

Description

This tool provides the last step for the daily workflow: cleaning up all listings, job files etc. related to that DATE which are not supposed to be stored. It also extracts statistics using extractStat. As the first step, the tool downloads all headers and compares them to the existing pool, in order to catch any delayed headers. This is done in non-PHOENIX installations.

It checks and delete all files carrying the $DATE string in their name in the following directories:

• $DFO_JOB_DIR
• $DFO_MON_DIR (except of the DFO_$DATE log file)
• $DFO_GUI_DIR
• $TMP_DIR
• /tmp

It also removes left-over ABs, AB logs, and trending logs in $DFO_AB_DIR; deletes multiple fitsreport and dataclient log files; and cuts the DFO_STATUS file to the latest 5000 entries. For DRS_TYPE=CON, it checks and removes all cascade logs under $CONDOR_DAG_DIR. Finally, it deletes $DFO_RAW_DIR/$DATE and all raw data therein, and updates the data report (calling createReport) and the histoMonitor. Obsolete status files are deleted on the QC web server (qc@qcweb: /home/qc/public_html/<instr>/monitor/status_<date>*.html).

There is the option -c, which bypasses the statistics update but does the cleanup part. This is useful if you have done some tests on a night which has already finished its regular workflow, and don't want to take care manually for cleaning up all the intermediate files. In that case, calling finishNight in the usual way would corrupt the statistics for raw data, which is avoided with the option -c.

Output

How to use

Type finishNight -h for on-line help, finishNight -v for the version number. Type

finishNight -d <date>

to clean up information related to <date> and update statistics, and

finishNight -d <date> -c

to avoid updating the statistics.

Configuration file

None.

Status information

Operational aspects

• The tool checks for open ingestion tasks in the ingestion job file (JOBS_INGEST under $DFO_JOB_DIR). If found, it displays a warning, which however can be overridden.
• If CONDOR is used, the tool checks and removes all directories underneath $BQSROOT/condor carrying the $DATE string.
• The tool updates the data report and the history. For the history, it calls histoMonitor -m <yyyy-mm> as read from $DATE (for dfos only, ignored in PHOENIX environment).
• Although rare, header delays may happen from time to time. Without the completeness check, they would get unnoticed. If a discrepancy is found, the QC scientist decides what to do. Depending on circumstances, there might be the need to process new CALIB ABs, or create new SCIENCE ABs, or the issue could be ignored since e.g. a missing TEST file is found. In any case, calling the tool again will remove the discrepancy since after each download the 'new' directory becomes the 'old' one.