Quality Control and
Data Processing

General information

PIPELINES AT ESO

Pipelines are used at ESO to process both calibration data and science data and to retrieve quality information. There is a dedicated pipeline for each VLT and VLTI instrument. Find general information about ESO reduction pipelines here.

FUNCTIONALITIES

The main functionalities of the pipelines are:

• create master calibration data,
• reduce science frames,
• extract QC information from the data.

For selected instrument modes, we offer science-grade data products processed with the pipelines.

OPERATIONS

There are two instances of the data reduction pipelines:

• at the instrument workstation on Paranal, running in automatic mode,
• at HQ Garching, run by the Quality Control Group in the optimized mode.

The optimized mode is the mode, which uses all data of a night, including the daytime calibrations. The calibration data are sorted and grouped according to their dependencies. Master calibration data are created. Their quality is checked.

MUSE part

MUSE observations in wide-field-mode (WFM) are offered with no AO (NOAO) and AO (since early 2018) correction in two different settings: E - extended range, to include the blue part of the spectrum and N - nominal range of the spectrum.

In 2018, new, narrow-field-mode (NFM) was commissioned. The data are taken only using AO system.

The MUSE pipeline is publicly available here. Under this link you can also find the pipeline Users Manual.

CALIBRATION and SCIENCE data

Find the description of MUSE data processing and pipeline recipes for calibrations here and for science data here.

The pipeline supports all MUSE settings.

RAW DATA

The MUSE raw data file consists of 24 FITS extensions, one for each IFU. It contains the detector data as well as the IFU/detector specific keywords, and a primary FITS HDU containing only keywords. The 24 extensions with detector data of the IFUs can be identified by EXTNAME FITS keyword. The extensions are called CHAN01, CHAN02, etc. and, for technical reasons, the numbering of extensions does not correspond to the numbers of the MUSE channels. However, the sequence of the channels as they are stored in the FITS file is fixed. Find more details about the MUSE CCD detectors format and properties here.

The User Manual has also a detailed description of the MUSE raw data structure.

file type	size of raw file
calibration or science frame	0.9 GB

PRODUCT FILES

The MUSE data reduction system (DRS) creates pipeline products in three different formats:

• Images
• FITS tables
• Pixel Tables
• Data Cubes

Pipeline Products - Images: A MUSE image, i.e. the reduced image of a single CCD (channel) has three 2D FITS image extensions:

• DATA - the reduced image
• DQ - the data quality flags encoded in an integer value according to the Euro3D standard
• STAT - the variance of the reduced data

The calibration recipes write product files which contain data belonging to a single IFU, thus produce 24 files for each kind of product. In addition, the combined format is available where all 24 products are stored in a single FITS file in extensions. For example, a combined master bias consists of 72 extensions (24 x 3 (data, error and data quality)) with extensions called CHAN01.DATA, CHAN01.DQ, CHAN01.STAT, CHAN02.DATA, CHAN02.DQ, CHAN02.STAT, etc.

Starting from 2014-12-08, when pipeline version v.1.0 was installed, all the products come in combined format.

Pipeline Products - FITS tables: Some of the products of the calibration recipes come in a format of standard FITS tables.

Pipeline Products - Pixel Tables: The pixel table format is used to store intermediate products of the pre-processing recipe muse_scibasic and a few post-processing recipes. It is a tabular format where 1 raw corresponds to each CCD pixel of a single IFU and the value of each pixel is stored together with it's "coordinates" (position and wavelength).

Pipeline Products - Data Cubes: The final output of the MUSE science recipes is in data cube format. Similar to the image format the data cubes consist of up to three 3D FITS cube (NAXIS=3) extensions. They are also named DATA, DQ and STAT. By design, the data quality extension is not included. Instead, pixels which do not have clean data quality status are directly encoded as NaN in the DATA extension.

file type	size of product file
Images combined	4.9 GB
Fits tables	0.1-0.8 MB
Pixel tables	up to 9 GB
Data Cubes	up to several GB

More details are described here.