make printable	new:	Note:
	v1.5: - new column 'pipefile'; mark multiple entries v2.0: - supporting new mode -S (static calibrations)	databases qc_metadata..qc_products; qc1_daily_comments; ngas..ngas_files dfos tools ngasClient dpDelete histoMonitor phoenixMonitor output $DFO_MON_DIR/productExplorer.html and $DFO_MON_DIR/productExplorer1.html upload/download none

productExplorer

Description

This is a helper tool for exploring and managing the content of the qc_metada..qc_products database. This database stores all QC-ingested product fits files (calib, science: until 2011-10, ancillary: until 2014-05). It offers a user-friendly way to query ("explore") its content, and fine-tune the query. There are options to download the results or to delete entries.

It launches a direct isql query to qc_metada..qc_products.

It displays its result in a html table which can be sorted and filtered in the standard way, plus an executable download file, plus a deletion file for ingested products (to prepare for ingestion of an improved version).

It offers a way to explore the content of static calibrations. These are calibration products which (in most cases) have no parent raw files and cannot be easily found with the normal date-oriented query.

How to use

Type productExplorer -h for on-line help, productExplorer -v for the version number,

productExplorer -d 2017-10-15

to explore calib for that date and the default $DFO_INSTRUMENT,

productExplorer -d 2017-10-15 -D 2007-10-30 -i CRIRES

for the specified time range and instrument,

productExplorer -d 2017-01-01 -p LINE_TABLE

for the specified pro_catg value only, and

productExplorer -d 2017-01-01 -p LINE_TABLE -s grism_256

to filter from the result set those files with setup value "grism_256". This filter option is done after the isql download, at the unix level, for performance reasons.

To launch a query for static calibrations, you need to know its product category (pro_catg) since there is no flag 'static calibration' in the database. Then, you can launch a query using the above options but replacing the date by the flag -S ('static'):

productExplorer -S -p FLUX_STANDARDS

or

productExplorer -S -p REFERENCE_DISP_SOLUTION -s Medusa1_L385.7nm

if you want to constrain the setup.

Click [info for static files] on the result page to get some more information.

Result HTML file. The result file comes as an HTML file, similar in format to the (raw) data reports created by createReport. It uses color-coding to distinguish calib (green) from science (blue) files. It also supports ancillary (log) fits files (displayed yellow). In standard (date) mode, it displays daily comments just like the raw data reports.

Product report for 2008-03-13 (products ingested into NGAS)
CAL SCI ANC ALL number: 77 size (MB): 115.1 3.2 8.8 127.1
Date comment:	Last night of old CCD Bruce; IFLATs taken; STD efficiency
[read query] | [read download file] | [read deletion file]

type*	origfile	setup**	pipefile	mjd_obs	pro_catg	ins_mode	dp_id
VIRTUAL	V.GIRAF.2008-03-14T02:29:04.977_005.fits		NULL	54539.163178	SCIENCE_RBN_SPECTRA	MED	V.GIRAF.2008-03-14T02:29:04.977_005.fits
SCIENCE	GI_SRBS_287119_2008-03-14T02:29:04.977_IFU2_L543.1nm_o3.fits	IFU2_L543.1nm_o3	r.GIRAF.2008-03-14T02:29:04.977_tpl_0001.fits	54539.103530	SCIENCE_RBNSPECTRA	IFU	S.GIRAFFE.2008-04-01T09:28:10.770
CALIB	GI_PIME_080313A_Argus_L427.2nm_o4_LTT3218.fits	Argus_L427.2nm_o4_LTT3218	r.GIRAF.2008-03-14T10:39:42.5009_tpl_0006.fits	54539.163178	STD_RCERRORS	ARG	M.GIRAFFE.2008-04-01T09:00:16.159

Filter and sort. The HTML table can be filtered (any string from the content can be used). You can filter e.g. to display only MBIA files, or files containing the string U_BESS etc. Sorting can be done by any column.

Virtual products. Virtual products (defining the calSelector datasets) are displayed. You can toggle between displaying and hiding them, nearby the 'Filter' option.

Multiple entries. In particular static calibration files could have been entered multiple times with different origfile names. These files, as being identified by the combination of pipefile, pro_catg and mjd_obs, are marked in red font color.

Download. The files can be downloaded, by using the executable file $TMP_DIR/mcalibDownload, either as it stands or after further editing. It uses ngasClient for the download. All downloads go into $DFO_CAL_DIR/DOWNLOAD and can then be further distributed by the user.

Deletion. The tool supports deletion of products. It provides the results of the query in a text file, with the proper syntax prepared for deletion of these files. However, without further editing, deletion will be performed only in simulation mode. Only if the user edits the deletion file to the full syntax, e.g.

dpDelete GI_MBIA_...fits -force

these files will get deleted. Deletion means physical removal and deletion in all database tables. Deletion is not hiding: after deletion, you cannot reconstruct the file content from information stored anywhere in NGAS. You can, however, ingest a new version of this file from your local file system, and this is the main purpose of the tool.

Obviously, the deletion mode should be used with care.

Installation

Use dfosInstall.

Configuration file

none.

Operational aspects

• Note that this tool applies $DFO_OFFSET in the definition of the time ranges. This is your local, current value, so it might deviate from the historical value on your colleague's machine when the data were named and ingested.
• Note that for MUSE IDPs, there is a special channel to ingest certain types of science files, in addition to the phase3 ingestion. This channel uses the standard dpIngest tool (otherwise used for master calibrations only, since 2011) and creates science products in NGAS with the label S. (liked in the pre-2011 aera). They become visible with the productExplorer tool.
• Multiple entries of static calibrations. The tool can make these entries visible. There is no strict rule what to do with them. Usually it might be reasonable to delete the ones with non-standard names and keep the one with standard name. But there are also cases when IDPs are processed from historical ABs where the "old names" are listed, and replacing the old names by the new names, one by one, in the historical ABs might become a hard and unsafe task. Whether "cleaning up" or not is the better choice depends on these circumstances.