Quality Control and
Data Processing

General information

PIPELINES AT ESO

Pipelines are used at ESO to process both calibration data and science data and to retrieve quality information. There is a dedicated pipeline for each VLT and VLTI instrument. Find general information about ESO reduction pipelines here.

FUNCTIONALITIES

The main functionalities of the pipelines are:

• create master calibration data,
• reduce science frames,
• extract QC information from the data.

For selected instrument modes, we offer science-grade data products processed with the pipelines.

OPERATIONS

There are two instances of the data reduction pipelines:

• at the instrument workstation on Paranal, running in automatic mode,
• at HQ Garching, run by the Quality Control Group in the optimized mode.

The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked.

HAWKI part

The HAWK-I data processing pipeline is operational since April 2008. All operational HAWK-I observing modes are supported although with some caveats:

• JITTER offsets greater than 1500 can not be handled
• Limitations of the OS may impose limits on the number of RAW frames that can be handled in one recipe exection.

Changes to the pipeline

The following is s summary of some changes in the pipeline. Please note that the list may not be complete:

• 1.4.2-Public Release:
  • Monolithic Science recipe hawki_img_jitter split into several component recipes to allow use with Reflex.
  • MASTER DARK product is now divided by DET.DIT value, i.e. pixel values are now [ADU/sec] (previously were [ADU])
  • A number of bugs and improvements in hawki_img_zpoint
• 1.3.5: Better memmory allowing the hawki_img_jitter recipe to handle up to ~135 frame stacks
• 1.3.5: Median level and RMS level filtering for auto selection of Flatfield sample
  
• 2015-10-01 New, CASU prepared pipeline (v.2.0.2) adopted. It is dramatically different from previous versions. New master calibrations are not compatible with the old ones. In particular, measurements of the zeropoints differ from previous values. For more information go here

CALIBRATION, REDUCTION

Find the description of HAWK-I data processing and pipeline recipes here:

• calibration
• science reduction

ESO internally processed HAWKI science data products are available on the phase3 archive interface.

RAW DATA

HAWK-I has a 2x2 mosaic of 4 2048x2048 photon-sensitive pixels. There are NO pre- or post- overscan columns or rows.

Further details regarding the chips can be found in Appendix B of the User Manual.

In total, HAWK-I raw frames have 4096x4096 pixels. There is only one read-mode available, which is unbinned, un-correlated. Raw frames have a size of 65MB each.

Extensions: Raw data come as FITS files with FIVE HDUs (header units), so-called Multi-Extension FITS files, or MEFs. The primary HDU has the header with all primary keywords and telescope, positioner, detector, observation, instrument etc. keywords. The pixel data, plus chip-specific header info for each of the FOUR chips are then stored in four further HDUs. HAWKI data obtained in the AO (Adaptive Optics) mode have additional 1 or 2 extensions containing information from the GRAAL AO system module.

Products:Pipeline products are either FITS images or FITS tables, and either have a single HDU (i.e. traditional FITS files) or 5 HDUs as in case of the RAW data, i.e. a primary HDU containing the header with all primary keywords and telescope, positioner, detector, observation, instrument etc. keywords and then, pixel or table data, plus chip-specific header info for each of the FOUR chips in four further HDUs. Further information can be found here.