This section is not maintained anymore, but is the frozen version used last with FORS1.
DATA MANAGEMENT AT ESO
Once a VLT/VLTI observing run is finished, the QC Group creates a data
package which is then delivered by the Archive Operations Group to the PI.
Data packages are created for all VLT and VLTI instruments for observations done in Service Mode.
Data packages for VLT pre-imaging runs are created and delivered
just like ordinary data packages.
The data packages contain raw and processed data (provided there is
pipeline support for the instrument mode used), calibration data as well as science
data. To distribute the data correctly, the FITS keyword 'HIERARCH ESO OBS PROG ID'
is evaluated (where PROG.ID actually is a RUN.ID).
The following rules apply when data are distributed in a data package:
SCIENCE and ACQUISITION files: a strict check on PROG_ID is made.
PIs receive all science files which carry that PROG_ID, and only those files.
CALIB files: PIs receive by default shared calibration files, i.e. those ones which were produced by the Observatory as part of the daily calibration plan. If in addition the user has defined specific calibrations all these files will carry their PROG_ID and will also be included in the data package.
File naming schemes
Throughout the data flow system, files are named following certain conventions. The following naming schemes are relevant for files in the data packages:
archive file name (FITS keyword: ARCFILE): this is the unique name of a raw file which consists of the parent instrument name and the timestamp of its generation. Raw FITS files produced at ESO always have extension '.fits'.
original file name (FITS keyword: ORIGFILE): this is the name of a raw file generated on the mountain on the instrument workstation. It consist of:
- the name of the parent instrument,
- the category (e.g. BIAS, FLAT, OBS, ... ),
- the day sequence number within the year,
- and a sequence number counted within the night.
Original file names are not unique (they will usually repeat after a year, sometimes even after a day).
data product file name (FITS keyword: PIPEFILE): this is the name of any pipeline product. It is composed of:
the prefix 'r.',
the ARCFILE of the parent raw file stripped off by the extension,
and a sequence number '_0000' etc. (since there may be multiple pipeline products per raw file).
Product files have the extension '.fits'. In some cases, ESO product files also may have the extension 'tfits' where 't' stands for 'table'. These are ordinary FITS files just like the ones with extension 'fits'.
In case that more than one raw file has been used to generate the product, the ARCFILE of the first one is used.
calibration product name: this is a name of calibration data products which has been chosen to have explicit information about the type of file, the date of creation, its version and the instrument parameters in the file name. It is generated after the pipeline product has been created. The FITS keyword PIPEFILE is preserved and can be used to map the two naming conventions.
science product names: with period 75 (April 2005), science products have also been renamed to a scheme which allows the user to recognize from the file name the parent OB, the product type and setting parameters.
Do I have to submit a request to receive my data package?
No. Service Mode (SM) programmes are declared completed by the Users Support
Department (USD) (info | mail).
This signal is sent to the Quality Control Group which then prepares the data package.
The data are copied onto media and sent to the PI as soon as possible.
When can I expect to receive my data package?
Once a run has been finished (VM) or closed by USD (SM),
the following series of steps can be expected:
Raw data are transfered over the internet and are available in Garching
essentially in real time.
They are processed, quality-checked and packed by QC Garching as quickly
as possible.
Once all data for a specific run have been processed, a signal is sent
to the archive group to copy this data package on media (presently DVDs).
The complete set of data is mailed to the PI.
In total this chain of events is finished on average a few days after the
last frames have been measured. Usually this time span is dominated by the shipping time
to the PI.
General information about Service Mode Observing can be found here.
The processing sequence for FORS1 data and the association between calibration and science data can be found here. Usually Service Mode packages should contain raw and processed science and calibration data (if the mode is supported by the pipeline). Sometimes, however, the automatic pipeline fails. In this case information should be available in the GEN_INFO directory.