version of this description: |
P87
(April 2011) |
Data packages have been delivered for period P87 (April-September 2011) and before.
For new data, acquired after the begin of October 2011, data packages are no longer
created. Users can access their raw data in electronic form through the ESO User Portal.
Data for VLT pre-imaging runs are processed and delivered as
before.
For completeness, the structure of the historical packagesis described
below.
Science data have been processed by the pipelines with the best available calibration data. Please note that ESO is not assuming any responsibility in respect to the usefulness of the reduced data. The adopted reduction strategy may not be suitable for the scientific purpose of the observations.
top PACKAGE DIRECTORY STRUCTURE
|
The top-level structure of the data package is as follows:
For each observation block (OB) that has been executed on Paranal, you find all measured raw data (FITS files) in a directory named by the OB number (FITS key HIERARCH.ESO.OBS.ID). If pipeline products exist, these are also added in the OB directory.
The GEN_CALIB directory collects all those calibration files (raw and products) that have been measured as part of the regular calibration plan, and calibration frames of a general nature (like static line tables). The GEN_INFO directory has general information, like data reports and night logs.
The tree shown above is the logical structure, which means that this is the way the data have been organized before they have been put onto media. Depending on the size of your package, the directories may be distributed across several media. It is a good idea to create the original tree on your local disk and then copy all files from the media into this tree.
top CONTENT
OF DIRECTORIES
|
<OBS_ID> (e.g. 179211)
For each executed observation block of your run, the package contains a directory with all measured data from that OB. All data under <OBS_ID> carry your run ID.
Note that some of your OBs may have been executed more than once. In particular, if time permitted, observatory staff try to re-execute OBs which produced data clearly out of the specified constraints. Check out the NIGHTLOG.html file for details (go to "OB information"). All data from OBs that have been executed multiple times are found in the same directory.
Each OB directory is further subdivided into subdirectories for science frames, calibration frames, and log files. In many cases, there will be science data only, but there may also be OBs with attached calibration data:
|
acq |
sci_raw |
sci_proc |
cal_raw |
cal_proc |
logs |
|
<OBS_ID>/acq
All acquisition frames (DPR.CATG=ACQUISITION) from the OB are contained in this directory. This directory only exists if such data exist.
<OBS_ID>/sci_raw
All raw science frames (DPR.CATG=SCIENCE) from the OB are contained in this directory.
<OBS_ID>/sci_proc
Here you find the pipeline-processed science data. The naming scheme can be found here.
<OBS_ID>/cal_raw
If measured, raw calibration frames (DPR.CATG=CALIB) produced by the OB are contained in this directory ("attached calibrations"). These are the ones which have been taken upon user's request in addition to the ones from the calibration plan.
Calibrations measured as part of the regular calibration plan are stored under the GEN_CALIB directory.
<OBS_ID>/cal_proc
The pipeline products of the raw attached calibrations are delivered here.
The CALIB products are renamed. The naming scheme can be found here.
<OBS_ID>/logs
This directory holds logging information about processing and packing
of your data:
- Association Blocks (.ab)
- association logs (.alog)
- logs of the pipeline processing (extension .rblog)
- scoring results (.html) (optional)
- extraction from the nightlog, OB grade, QC0 report (.qcm)
- a reduction comment (.cmt) (optional)
Description:
- Association Blocks (ABs) are text files which contain all the
information required to pipeline-process and pack data. This information includes
the reduction recipe, the input raw file(s), the calibration products needed
for processing, and the names of the final products. More ...
- Association logs are delivered since P80. They are a simplified version of ABs, designed to provide the association information essential for the user. More ...
- The pipeline processing log is a record of the science reduction process, with a detailed log of reduction steps, results etc.
- The scoring report is intended to give some feedback
about the data quality. It is still experimental. More ...
- The nightlog file is an extracted version (per OB) of the summary
qc0 report and the NIGHTLOG.html file (see below). More ...
That directory in addition holds QC plots, if available.
GEN_CALIB
This directory collects all calibration frames from the regular calibration plan that are associated to your science data. It also contains their pipeline products, and calibration frames of a general nature (like static line tables). Calibrations that have been measured by user-defined OBs and that have been used for pipeline processing of science data may be included here in addition.
The directory has four subdirectories (gen, logs, proc, raw), two of which have further fine-structure:
GEN_CALIB/raw
Raw calibration files. These divide into raw file types (e.g. BIAS, FLAT etc.; see instrument specific section below).
GEN_CALIB/proc
Calibration products derived from the raw calibrations. These divide into file types like the raw calibration files, see instrument specific section below.
The CALIB products are renamed. The naming scheme can be found here.
GEN_CALIB/logs
Association Blocks, association logs and processing logs for the
calibration files under GEN_CALIB. There might also be scoring logs (.html files).
GEN_CALIB/gen
General calibration data of static nature.
Additional or missing raw calibration files may be retrieved anytime from the generic
ESO Archive form, or from the instrument specific forms.
Calibration data are public immediately while SCIENCE data normally have a proprietary period of one year.
GEN_INFO
This directory hosts some general information. It has the following subdirectories:
DIRECTORY |
CONTENT |
ObservingReports |
nightlogs, OB report (HTML files), association report |
scripts |
executable scripts (presently one: print_all_reports) |
The data package contains the following report files:
FILE |
CONTENT |
WHERE |
README.html |
the package portal page: point your browser here to find all information |
top |
ServiceMode.html |
this file |
top |
product_codes.html |
a table describing the naming scheme for product files |
top |
|
archive_<RUN_ID.txt |
list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive |
GEN_INFO |
qc0_<RUN_ID>.txt |
list of all SCIENCE files, containing the comparison between the user constraint set and the actual values |
GEN_INFO |
|
NIGHTLOG.html |
set of html files with nightlog, OB and association information |
GEN_INFO/ObservingReports |
|
list_sciRaw_<OBS_ID>.txt etc.
|
summary report of the fits files in each directory (these files are provided in text [*.txt] and PostScript format) |
all data directories |
The executable script print_all_reports under GEN_INFO/scripts can be used to print all postscript files in your package.
Archive report: archive_<RUN_ID>
While the above listings are about files in the package, the archive report is the result of a query to the ESO Archive. It is useful as a check on the completeness of the data package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable.
archive report |
keyword table
|
|
sample file |
|
QC0 report: qc0_<RUN_ID>
This report is sent only for Service Mode runs.
This file contains a report of quality control parameters ('QC level 0' where level 0 stands for Quality Control without pipeline processing) for your raw SCIENCE files. These parameters are airmass, seeing, moon distance, and fractional lunar illumination. They have been measured on site (column 'msrd'). They are compared to the required values as defined in your OBs ('targt') and flagged (OK/NOK).
The list is intended to give a rough indication of whether or not the required constraints have been fulfilled. They should not be interpreted in a too formal way, however. E.g., there may be cases where the seeing was worse than required, but this was compensated by a longer exposure time. Check the night reports for details.
Note that the seeing values reported here are DIMM seeing values, they are not measured on the frame. If the alarm flag ("NOK") is set in the SEEING column, the DIMM seeing value was larger than your seeing constraint during the indicated obseration. However, in many cases, the delivered seeing in the instrument focal plane is better than the DIMM seeing. Whenever possible, the on-site astronomer has measured the focal plane immediately after or during execution to determine the success or failure of your observation. Thus, your observation may have been completed within your specifications, even if the SEEING alarm flag is set. Please review the affected observation carefully and check the night reports for details.
QC0 report |
keyword table |
|
sample file (.txt) |
|
Night logs, OB logs and Association report
This is a set of HTML files with night log information, OB grading information and data association information. All relevant information about the nights contained in your package is included here, as well as information about each OB in your delivery.
Point your browser to GEN_INFO/ObservingReports/NIGHTLOG.html (or start from the package portal page, README.html) and navigate per night (labeled as 1), per OB (2) or per set of files (3).
The HTML files also come as stripped-down, printer-friendly versions. The files are organized to have a summary on top, and details below.
You can use either the navigation bar to jump to a specific night/OB/set of files, or use the up/down arrows (night logs only) to browse sequentially. The OB navigation bar (2) uses colour coding to give you a quick impression about OB grading. There are additional links to ambient condition information.
The association report (3) organizes your data and their association. It has two main levels: the OB (observing block), and the AB (association block) which collects raw file(s) and associated information like product files, calibration files, log files etc. This report gives you an impression how the data in your package are logically linked, while the listings in each directory give you a table of contents. File names in the association report may show up several times, e.g when a calibration file has been used for processing more than one science file.
NOTE:
- The external links (like the ASM links: seeing, sky transparency etc.) will only work with network connection.
- The ASM links require java-enabled browsers.
- The navigation bars read best with style-sheets and java-enabled browsers.
Sample nightlog files |
NIGHTLOG.html |
|
Known IRAF problems
- Filename Length. To display or manipulate the FITS files
with older versions of IRAF (before 2.11), you can:
- copy these FITS files to your hard-disk and rename them with filenames <= 32
characters in length;
- create symbolic links with filenames <= 32 characters in length to your DVD
files.
- Header Interpretation. ESO FITS files use the ESO HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot automatically interpret the information provided in ESO HIERARCH header lines. The problem may be solved using the tool hierarch28. Find information about this tool here.
- RA, DEC. Please note that the RA and DEC keywords are recorded in degrees. To translate these keywords so that they can be used by IRAF you have to use the asthedit task in the noao.astutil package. The help file for this task gives an example of how to translate the ESO format to the IRAF format.
Stand-alone FITS handling tools
Find information about FITS header handling tools (e.g. dfits, fitsort, hierarch28) here.
The rest of this page contains an overview of the structure and content
of the data packages for UVES as a standalone instrument (also
called UVES-Echelle) and for UVES as part of the FLAMES instrument (FLAMES/UVES,
UVES-MOS, or UVES-Fibre). FLAMES programmes using both UVES and GIRAFFE
spectrographs ("COMBINED mode") receive two separate sets of
media, one for the FLAMES/UVES data and one for the FLAMES/GIRAFFE data. Both sets are
usually shipped in one single package.
The description of the Service Mode package for
FLAMES/GIRAFFE data is found here.
For further information about UVES data and QC have a look at:
top UVES SCIENCE PRODUCTS |
The renaming scheme for UVES and FLAMES/UVES SCIENCE product files can
be found
here.
The ONLINE version is here.
top UVES CALIBRATION PRODUCTS |
The UVES raw and processed calibrations are mostly produced by the
calibration plan. Only exception are attached flat calibrations as
specified by the user.
More about calibrations ...
The GEN_CALIB directory for UVES data is structured as follows:
This directory collects all raw calibration frames that are associated
to your science data, and that have been measured as part of the regular
calibration plan. It also contains their pipeline products and calibration
frames of a general nature (like static line tables). Calibrations that
have been measured by user-defined OBs and that have been used for pipeline
processing of science data may be included here in addition. The directory
has four subdirectories, gen, logs, proc and raw. The proc and raw
directories have further fine-structure:
TYPE/SUBDIRECTORY |
UVES FLAVOUR |
CONTENT |
DPR.TYPE |
DET
|
UVES-Echelle, UVES-MOS |
Detector bias frames |
BIAS |
SPEC_ARC |
UVES-Echelle |
Spectroscopic arc lamp exposures |
LAMP,WAVE |
SPEC_FLAT |
UVES-Echelle |
Spectroscopic flat field exposures |
LAMP,FLAT
LAMP,DFLAT
LAMP,IFLAT
|
SPEC_FLAT |
UVES-MOS |
Slit flat fields without fibre system |
LAMP,SFLAT |
SPEC_OTHER |
UVES-Echelle |
Order definition flat fields |
LAMP,ORDERDEF |
SPEC_OTHER |
UVES-MOS |
Order definition flat fields |
LAMP,ORDERDEF,SimCal |
MOS_ARC |
UVES-MOS |
Arc lamp exposures through fibre system |
LAMP,WAVE,OzPoz
LAMP,WAVE,SimCal |
MOS_FLAT |
UVES-MOS |
Flat field exposures with fibre system |
LAMP,FLAT,ALL,OzPoz
LAMP,FLAT,ODD,OzPoz
LAMP,FLAT,EVEN,OzPoz
LAMP,FLAT,ALL,SimCal
LAMP,FLAT,ODD,SimCal
LAMP,FLAT,EVEN,SimCal |
Raw calibration data are described here: UVES
and UVES-MOS
Calibration products derived from the raw calibrations. Please look
at the UVES
calibration pipeline or UVES-MOS
calibration pipeline pages for descriptions of the products.
The renaming scheme for UVES and FLAMES/UVES CALIBRATION product files can be found
here.
The ONLINE version is here.
You can expect the following static calibrations in the gen directory
FILE NAME |
PRO.CATG |
DESCRIPTION |
thargood_3.tfits |
LINE_REFER_TABLE |
reference line table |
atmoexan.tfits |
EXTCOEFF_TABLE |
extinction coefficients table (only UVES-Echelle) |
UV_MRSP*fits |
MASTER_RESPONSE_BLUE, MASTER_RESPONSE_REDL,
MASTER_RESPONSE_REDU |
master response curves for flux calibration (only
UVES-Echelle), available for central wavelengths 346, 390, 437,
564, 580, and 860nm |
Additional or missing raw calibration files may be retrieved anytime
from the ESO
Archive. Calibration data are public while SCIENCE data have a proprietary
period of one year.
Some notes on calibration frames
For operational reasons, calibration products are created
only for standard configurations. No solutions are produced for non-standard
configurations. Hence such SCIENCE data are not reduced either. The
UVES pipeline may well be able to reduce such non-standard data.
Bias frames usually come in sets of five raw files
which are stacked to one master bias. They are provided only once per
week since they are in general very stable.
The order definition flats (and the format check frames,
usually not delivered) are for pipeline use. They are required for automatic
recognition of the spectral format. If you plan to reduce your data
with your own software, or with MIDAS in interactive mode, you will
not need them.
UVES-Echelle flat fields: Day-time lamp flat fields
usually come in sets of five raw files which are stacked into one master
flat. The extreme blue settings (WLEN = 346 nm) have two different types
of lamp flats: those taken with the usual flat lamp, and those taken
with a deuterium lamp which provides higher response in the UV but has
emission lines at longer wavelengths (see User Manual). Both sets of
flats are delivered. Master flats are also created from both sets. The
automatic pipeline reduces science data with the normal master flats
only. Two different sets of flats also exist for science data using
the Iodine absorption cell (600 nm): one using the normal flat lamp
and one using the lamp plus the absorption cell. Both sets are delivered
and used for master flats. The automatic pipeline uses only flats without
absorption cell
UVES-MOS flat fields: There are two different varieties.
The fibre flat fields are used to recognise the fibre postitions on
the CCD frame. They come in sets of three raw files which have only
odd numbered fibres, only even numbered fibres, or all fibres illuminated,
respectively. Corrections of the blaze function and of pixel-to-pixel
variations are provided by slit flat fields. They are taken at three
different slit positions with a stack of (typically) three frames at
each position.
Flux calibration (UVES-Echelle): For the purpose of
flux calibration, a set of master response curves (UV_MRSP) is available
in the GEN_CALIB/gen directory (for 346, 390, 437, 564, 580, and 860nm
only). Further information about response curves and flux calibration
can be found here.
top PROBLEMS, ISSUES, HINTS
|
Known reduction problems (UVES-Echelle)
Instrumental artefacts. Please look at the page about recent
problems for recently discovered instrumental artefacts that are
not yet mentioned in the User Manual.
High signal. Optimum extraction may face problems in cases
of high signal. Check out here
for details.
High airmass. In cases of blue spectra taken at high airmass
(> 1.5), without the ADC, atmospheric dispersion may shift the bluest
orders out of the extraction window, and their flux is lost to the automatic
pipeline procedure. The data can, however, be recovered by manual extraction.
Even at airmass 1.1, differential extinction spreads the spectrum by
about 3 pixels for the bluest setup (346 nm).
Order indentification. In a few cases problems have occured
with order identification in settings with 346nm central wavelength:
the orders found by the pipeline in the order defintion flat and during
extraction/wavelength calibration of the science frame may be different.
This is not recognised by the pipeline and the lines in the final wavelength-calibrated
spectrum appear shifted by one order. An example is given here
where the overlap between an observation with 346nm and 437nm central
wavelength is shown. The problem may be resolved by choosing a different
order definition flat.
Sky emission lines. Optimum extraction may over/under-corrected
bright sky emission lines by 5% or more.
Artificial lines. Series of artificial lines are occasionally
seen around 607 nm and 638 nm in spectra reduced with optimum extraction.
These features may be mistaken as telluric or real physical lines.
Attached red flats. The recommended exposure times for attached
flat fields as given in UVES user manuals until version 19/02/2003 (including)
have been longer than the values used for day-time calibrations. Therefore,
one or two orders in the upper (MIT) chip of attached flats taken with
the FF4 lamp (used for 860nm settings) may have maxima above 40000 ADU.
This chip is not linear any more in this region and saturates soon.
It is recommended to check carefully attached flats taken with FF4 before
using them for calibrating science data. Flat fields taken with other
lamps or taken within the regular day-time procedure are not affected
by this problem.
BIAS level. Note in cases of high-precision reductions of faint
sources taken in 2x2 binnings that there is a temperature dependence
of the mean BIAS level especially for the red upper CCD, by about 2
ADU/K.
Seeing. If the seeing encountered during the SCIENCE exposure
is less than the slit width, it might happen that wavelength shifts
are introduced in the SCIENCE data if the object is not perfectly centered.
This may be checked and corrected, at least in the red, with the telluric
absorption lines.
Radial velocity correction. Until 2001-09-17 (date of acquisition),
a bug in the radial velocity correction as calculated by the pipeline
may have caused wrong values. Please check carefully (FITS keywords
HIERARCH ESO QC VRAD BARYCOR and HELICOR; also stated in the listing
file 'list_of_all_red.txt'). Remember that no correction is applied
to the data. This bug is present in version 1.2 and earlier of the uves
pipeline, and solved with version 1.3.
Known reduction problems (UVES-MOS)
Instrumental artefacts. Please look at the page about recent
problems for recently discovered instumental artefacts that are
not yet mentioned in the User Manual.
Extraction of 860nm data. The part of the spectra above 980nm
is usually not extracted well by the pipeline recipe.
High signal. Optimum extraction may face problems in cases
of high signal. Check out here
for details.
Artificial lines. Series of artificial lines are occasionally
seen around 607 nm and 638 nm in spectra reduced with optimum extraction.
These features may be mistaken as telluric or real physical lines.
Errorbar
values. The values in the ERR_MWXB_SCI files are about
a factor of two higher than the statistical 1sigma errors.
UVES solar spectrum
A combined solar spectrum ranging from 3100 A to 10,200 A, taken at highest
possible resolution with UVES, is available for reference purpose. Find
details here.
|