make printable	new:	see also:
	v3.3: - enabled for MCAL ingestion for instruments that are operational with qcFlow   new config key WITH_QCFLOW	- called by moveProducts - dpIngest information - how to ingest general calibration products
	v3.2: - PHOENIX only: new optional config key ENABLE_UPDATES - up to 3 attempts if ingestion fails due to NGAS error related to disk space	databases qc_products, to verify successful ingestion; qc1 for updating file names dfos tools dpIngest; fitsverify - off-line calls collected in JOBS_INGEST and managed by dfoMonitor - status of ingestion is displayed on the dfoMonitor (open: yellow circle; done: green ok)   output log files list_ingest and list_qc1update in $DFO_LST_DIR upload/download ingestion into NGAS

ingestProducts

Description

This tool ingests pipeline products into the NGAS archive, after they have been certified, distributed, and renamed. Calibration products (master calibrations) as well as science products (IDPs) are supported. The tool ingests all, or selected, products from the product directory. The product directory is $DFO_CAL_DIR/<DATE> for master calibrations, and $DFO_SCI_DIR/<DATE> for IDPs.

The tool is enabled to ingest IDPs (internal data products, being science products generated by QC through the phoenix process). This feature is documented in more detail on the PHOENIX branch of the DFOS documentation here.

With version 3, the tool is enabled to ingest master calibrations created by the PHOENIX process. The only difference to the normal (operational) ingestion of master calibrations is the configurable behaviour how to handle previously existing instances. This is documented in more detail on the PHOENIX branch of the DFOS documentation here. If you are not using the tool for a PHOENIX project, you can safely ignore this part.

The tool is normally called from the ingestion file under $DFO_JOB_FILE that is maintained by moveProducts (this is the usual operational setup for larger data sets). It can also be called from the command line.

The tool writes a log file list_ingest_$MODE_$DATE.txt into $DFO_LST_DIR. The monitor tools dfoMonitor and histoMonitor check this file and display whether or not a certain night with calibration products has already been ingested. (Note: versions earlier than v1.2 had the log file called list_ingest_$DATE.txt.)

It uses the DFS tool dpIngest that comes with the standard DFS installation. If the configuration key QC1_UPDATE is set to YES in the configuration file, the key cdbfile in the tables of the QC1 database is updated with the new name of the calibration product (since it has been renamed from the PIPEFILE format to the CALIBFILE format in the meantime). Find more information about dpIngest here.

If configured, the tool will send an email if errors or warnings are found in the log file.

The product data are ingested into NGAS. Their metadata description is extracted from the header keys and is stored in the qc_products database (qc_files).

In order for dpIngest to work properly, two entries in $HOME/.dbrc are required:

Also, you need to make sure that the fitsio tools, in particular fitsverify, are in your $PATH. Type 'which fitsverify', and in case of 'not found', enter something like

in your $HOME/.qcrc.

cleanupProducts. The tool writes, in both modes (CALIB and SCIENCE), a call 'cleanupProducts -d $DATE' into the job file $DFO_JOB_DIR/JOBS_CLEANUP that is monitored by dfoMonitor and ready for execution after finishNight has been executed. Find more information about cleanupProducts here.

Output

Error handling

The tool scans the log file $DFO_LST_DIR/list_ingest_$MODE_$DATE.txt for error messages from dpIngest and for unusual content. It reports about occurrences such as WARNING or ERROR before finishing. It also sends an email if SEND_MAIL=YES in the configuration file (highly recommended!). Error and warning messages must be carefully observed and understood in order to ensure proper ingestion of files.

The following error message from dpIngest is especially worth mentioning:

• "already stored in the calib DB": a file with the same name already exists in the calibration archive. The newer version on disk is then not ingested. In order to ingest it, the older version in the archive has to be deleted first using dpDelete.

All other messages are likely caused by problems with connecting to the archive. It must be checked that files have been successfully ingested.

Example

ingestProducts -m CALIB -d 2004-10-12 ingest master calibrations for that date
[ingestProducts -m SCIENCE -d 2013-10-12 ingest IDPs, if PHOENIX enabled ]
[ingestProducts -m CALIB -d 2003-10-12 ingest MCALIBs, if PHOENIX and MCALIB enabled ]

How to use

Type ingestProducts -h for on-line help, ingestProducts -v for the version number, and

ingestProducts -m CALIB -d 2004-10-12

to ingest all (or specified) pipeline products for mode CALIB on that date into the product archive.

Type

ingestProducts -m CALIB -d 2004-10-12 -f

to call fitsverify only on the fits files in $DFO_CAL_DIR/<date>, and verify their fitsverify compliance. No other effect. If the tool returns an error (exit code != 0), you may want to investigate the files with an error, by following the instructions. In case the fitsverify error is a real one, and not just a warning, you should create a pipeline ticket, with the exact description of the error, to get the file structure fixed.

For MCALIB projects under PHOENIX, type

ingestProducts -m CALIB -d 2003-08-02 -D

to call the tool in DEBUG mode, with interactivity and confirmation of the important steps of deletion of old instances and ingestion of new instances. See here for more.

Configuration file

config.ingestProducts defines:

Status information and logging

The tool writes the DFO status cal_Ingested or sci_Ingested.

Operational aspects

• The tool is normally called from the ingestion job file, $DFO_JOB_DIR/JOBS_INGEST, that is maintained by moveProducts.
• The DFS tool dpIngest comes with the standard DFS installation.
• Strategies for ingestion:
  - all calib products should be ingested that are potentially useful for future users. This includes:
  • products required for science processing (anywhere in the cascade)
  • products useful for assessment of instrument performance and carrying signature that is not covered by QC1 parameters (e.g. detmon flats monitoring contamination)

  - all calib products must be ingested that are associated to science data later.

• Product selection is possible only on the basis of pro_catg values. Case by case selection is not supported.
• Fine-tuning of the ingestion can also be done at earlier steps, when you define criteria for renaming and shifting:
• any file not renamed will not be ingested
• any product not selected for versioning will be deleted (e.g. from six master bias frames from the same night with the same setup, only the latest one will survive if versioning is disabled)
• any product not selected for RENAME_SHIFT (in config.processAB) will not be moved to $DFO_CAL_DIR and of course not ingested.
• Use the key SEND_MAIL to enable email notification if errors or warnings are found.
• The tool produces calls of cleanupProducts in the job file JOBS_CLEANUP. These can be executed only after having called finishNight for the corresponding nights.
• For the usage of the tool in the PHOENIX environment check here.
  
• For calling fitsverify (the tool to verify the fits structure) only, use the flag -f.