VERSION:
|
P80.1
(December 2007) |
Observing Runs performed in Service Mode (SM) on La Silla receive a set
of data on media which presently are DVDs. Since February 2006 the structure
and content of these data packages is aligned with the ones delivered
for SM runs on the VLT and VLTI. The packages presently contain
raw data only. The only exceptions are FEROS and HARPS packages
which also contain reduced SCIENCE files which have been processed by the
pipelines on-site. Contrary to VLT packages, there are no calibration
products in the packages, and there are no quality control and
certification processes in place.
For a given ESO run ID, e.g. 079.A-0123(A), the files included in the
data package are selected as follows:
- All RAW files with DPR.CATG=SCIENCE and OBS.PROG.ID=prog-id
- All RAW files with DPR.CATG=CALIB and OBS.PROG.ID=prog-id
- All other RAW with DPR.CATG=CALIB and acquired within the
noon-to-noon period(s) of the night during which each RAW SCIENCE of the
program was acquired. This is on the basis of the operational paradigm
that the relevant calibrations for each Service Mode program observed
during a given night were either acquired in the afternoon immediately
before or in the morning immediately following the SCIENCE
observations.
This means that the following files do NOT in general get included:
- Any other RAW files with any other DPR.CATG, even if they have
OBS.PROG.ID=prog-id
- Relevant calibrations acquired OUTSIDE the 24hr, noon-to-noon
period(s) centered on the night(s) of the SCIENCE observations. This policy
unfortunately often misses SKY-FLATs which are often difficult to
acquire in advance or in the following morning.
If you feel you are missing calibrations, they have probably been
acquired outside the 24hr, noon-to-noon
period(s) centered on the night(s) of the SCIENCE observations. You
should then look for such missing calibrations in the
ESO Archive.
If you are unable to locate suitable calibrations for your data in the
ESO Archive, then please contact
La Silla.
The following table summarises the instruments supported by this
process:
Instrument |
date of implementation |
EFOSC (Notes) |
April 2007 |
EMMI (Notes) |
October 2006 |
FEROS (Notes) |
February 2006 |
HARPS (Notes) |
October 2006 |
SofI (Notes) |
October 2006 |
SuSI2 (Notes) |
October 2006 |
PACKAGE
DIRECTORY STRUCTURE
|
The top-level structure of the data package is as follows:
For each observation block (OB) that has been executed on La Silla,
you find all measured raw data (FITS files) in a directory named by the OB number
(FITS key HIERARCH.ESO.OBS.ID). If pipeline products exist (FEROS
and HARPS), these are also added in the OB directory.
The GEN_CALIB directory collects all those calibration files
(raw data only) that have been measured as part of the regular calibration plan,
and if any calibration frames of a general nature (like static line tables).
The GEN_INFO directory has general information such as data
reports.
The tree shown above is the logical structure, which means that this
is the way the data have been organized before they have been put onto media. Depending
on the size of your package, the directories may be distributed across several media.
It is a good idea to create the original tree on your local disk and then copy all
files from the media into this tree.
<OBS_ID> (e.g. 179211)
For each executed observation block of your run, the package contains
a directory with all measured data from that OB. All data under <OBS_ID> carry
your run ID.
Note that some of your OBs may have been executed more than once. In
particular, if time permitted, observatory staff try to re-execute OBs which produced
data clearly out of the specified constraints.
All data from OBs that have been executed
multiple times are found in the same directory.
Each OB directory is further subdivided into subdirectories for science
frames, calibration frames (if any), and log files. In many cases, there will be science data
only, but there may also be OBs with attached calibration data:
|
acq
|
sci_raw
|
sci_proc
|
cal_raw
|
logs
|
|
<OBS_ID>/acq
All acquisition frames (DPR.CATG=ACQUISITION) from the OB are contained
in this directory. This directory only exists if such data exist.
<OBS_ID>/sci_raw
All raw science frames (DPR.CATG=SCIENCE) from the OB are contained
in this directory.
<OBS_ID>/sci_proc
Here you find the Pipeline Product Tarballs
(PPTBs) produced by the online pipelines running at La Silla, HARPS and FEROS only.
This directory only exists if such data exist.
<OBS_ID>/cal_raw
If measured, raw calibration frames (DPR.CATG=CALIB) produced
by the OB are contained in this directory ("attached calibrations").
These are the ones which have been taken upon user's request in addition to
the ones from the calibration plan.
Calibrations measured as part of the regular calibration plan are
stored under the GEN_CALIB directory.
This directory only exists if such data exist.
<OBS_ID>/logs
This directory has logging information about the packing
of your data:
- Association Blocks (extension .ab)
- Association Logs (extension (.alog)
For VLT instruments, Association Blocks (ABs) are text files which
contain all the information required to pipeline-process and pack data. This
information includes the reduction recipe, the input raw file(s), the
calibration products needed for processing, and the names of the final
products. They are described here.
For La Silla packages, they are only used (and useful) for their packing
information.
Association logs are delivered since P80. They
are a simplified version of ABs, designed to provide the association
information essential for the user. More ...
GEN_CALIB
This directory collects all calibration frames from the regular
calibration plan that are associated to your science data
and calibration frames of a general nature (like static line tables).
Calibrations that have been measured by user-defined OBs from
other programs may be included here in addition.
The directory has three subdirectories (gen, logs, raw), one
of which has further fine-structure:
|
|
|
|
RAW_TYPE1,
e.g. BIAS |
RAW_TYPE2,
e.g. FLAT |
RAW_TYPE3,
e.g. WAVE |
RAW_TYPE4,
e.g. STD |
|
|
|
GEN_CALIB/raw
Raw calibration files. In general, for La Silla pacakages they
are currently all grouped together in one sub-directory called
CALIB. At some point in the future they may be separated by
RAW_TYPE, which would depend on the instrument used (e.g. BIAS,
FLAT, WAVE, STD, etc.)
GEN_CALIB/logs
Association Blocks and association logs for the
calibration files under GEN_CALIB.
GEN_CALIB/gen
General calibration data of static nature.
Additional
or missing raw calibration files may be retrieved anytime from the generic ESO
Archive form.
Calibration data are public immediately while SCIENCE data normally
have a proprietary period of one year.
GEN_INFO
This directory hosts some general information. It has the following subdirectories:
DIRECTORY |
CONTENT |
ObservingReports |
association report |
scripts |
executable scripts (presently one: print_all_reports) |
The data package contains the following report files:
FILE |
CONTENT |
WHERE |
README.html |
the package portal page: point your browser here to find all information |
top |
ServiceMode.html |
this file |
top |
|
archive_<RUN_ID>.txt |
list of all proprietary files (SCIENCE, attached CALIBs) as read from the archive |
GEN_INFO |
|
list_sciRaw_<OBS_ID>.txt etc.
|
summary report of the fits files in each directory (these files are provided in text [*.txt] and PostScript format) |
all data directories |
The executable script print_all_reports under GEN_INFO/scripts will print
all ps files in your package.
Archive report: archive_<RUN_ID>.txt
While the above listings are about files in the package, the archive report is the result of a query to the ESO Archive. It is useful as a check on the completeness of the data package. All files created by OBs which have been generated by the PI are listed here. The list includes all SCIENCE files, and the attached calibrations, and acquisitions, if applicable.
archive report |
keyword table
|
|
sample file |
|
Association report
The association report organizes your
data and their association. It has two main levels: the OB (observing
block), and the AB (association block) which collects raw file(s) and
associated information like product files, calibration files, log files etc.
This report gives you an impression how the data in your package are
logically linked, while the listings in each directory give you a table of
contents. File names in the association report may show up several times,
e.g when a calibration file has been associated to more than one science file.
Point your browser to
GEN_INFO/ObservingReports/ASSOCLOG.html (or start from the package portal
page, README.html).
The Association report also comes as
stripped-down, printer-friendly version. The file is organized to have a
summary on top, and details below.
Known IRAF Problems
- Filename Length Problem. To display or manipulate the FITS
files with older versions of IRAF (before 2.11), you can:
- copy these FITS files to your hard-disk and rename them with filenames <= 32 characters
in length;
- create symbolic links with filenames <= 32 characters in length to your DVD files.
- Header Interpretation Problem. ESO FITS files use the ESO
HIERARCH FITS keyword extensions standard to all ESO telescopes. Note that IRAF
treats all ESO HIERARCH header lines as COMMENT lines, i.e. IRAF and IDL cannot
automatically interpret the information provided in ESO HIERARCH header lines.
The problem may be solved using the tool hierarch28.
Find information about this tool here.
- RA, DEC. Please note that the RA and DEC keywords are recorded
in degrees. To translate these keywords so that they can be used by IRAF you have
to use the asthedit task in the noao.astutil package.
The help file for this task gives an example of how to translate the ESO format
to the IRAF format.
Stand-alone FITS handling tools
Find information about FITS header handling tools (e.g. dfits,
fitsort, hierarch28) here.
Instrument Specific Information
|
EFOSC
- Additional Calibrations:
Can be downloaded from the ESO
Archive.
EMMI
- Additional Calibrations:
Can be downloaded from the ESO
Archive.
FEROS
- Additional Calibrations:
DomeFlat (until mid 2006) and DARK calibrations were acquired according to
the FEROS calibration plan
(see FEROS Calibration Plan)
on a semi regular basis. If such
calibrations were acquired during the afternoon before or
the morning after your service mode observations, they will
be included in this package. Otherwise they can be
obtained from the
ESO Archive.
For DomeFlats Search for Orig Name =
FEROS_ech_cal_df, for DARKs search for Orig Name =
FEROS_ech_cal_dark.
Additionally, calibrations are grouped together on the basis of
the "same day" which for FEROS is defined to be from UTC 12:00 +
4.0hrs in the afternoon before the night till UTC 12:00 + 4hrs in
the afternoon after the night. This means that if calibrations are started
later than usual, some of the calibration exposures may have been
started "the next day" and they may therefore not be
included in this package. In such cases any missing calibration
files can also be obtained from the
ESO Archive.
- FEROS Pipeline Product
Tar-Balls: The
FEROS MIDAS Data Reduction System
(DRS a.k.a Pipeline) is run "online" at La Silla. The
principal purpose of running the DRS at the telescope is to
provide a real-time reduction of the observations as they are acquired.
Various simplifying assumptions are made and as
a result, these reductions, or DRS products, are intended only for
"quick look" purposes . They are not
intended to be of Publication Quality. More to the point
ESO definitely does NOT guarantee them to be of publication
quality. They can be used to check approximate Signal to Noise
level and simple things like that. None the less, since even these results
can be of further use to service mode PIs
these Pipeline products are archived in the ESO archive and are
included in these Service Mode data packages in the format of a
Pipeline Product Tar-Ball (PPTB). They are not
yet available for general users to download from the archive, but
hopefully one day will be.
However for various
reasons, from time to time it can happen that there is no PPTB for a given
RAW file or that, due to problems with the setup or initialisation of the
DRS, any given PPTB can be completely useless for any purpose at all. The
contents of FEROS PPTBs is described in this
README
(which is also included in each FEROS PPTB).
HARPS
- Additional Calibrations:
Can be downloaded from the ESO
Archive.
- HARPS Pipeline Product
Tar-Balls: The
HARPS Pipeline
is run "online" at La Silla and reduces every RAW file.
These pipeline products are archived in the ESO archive and are
included in these Service Mode data packages in the format of a
Pipeline Product Tar-Ball (PPTB). They are not
yet available for general users to download from the archive, but
hopefully one day will be.
However for various
reasons, from time to time it can happen that there is no PPTB for a given
RAW file or that, due to problems with the setup or initialisation of the
pipeline, any given PPTB can be completely useless for any purpose at all.
SofI
- Additional Calibrations:
Can be downloaded from the ESO
Archive.
SuSI2
- Additional Calibrations:
Can be downloaded from the ESO
Archive.
|