VLT Instrument Pipelines

Latest News:

April 10, 2014: New release of all the pipelines compatible with CPL 6.4.1 and Reflex 2.6

April 01, 2014: Two new software tools to correct observations for atmospheric telluric absorption (Molecfit) and subtract sky emission for observations without plain sky information (Skycorr) are released for the first time. The tools have been developed for ESO by a team of astronomers at the Institute for Astro- and Particle Physics at the University of Innsbruck, as part of an in-kind contribution to ESO.

Nov 15, 2013: Reflex 2.5 has been released together with pipelines and workflows FORS 4.11.12, KMOS 1.2.6, SINFONI 2.4.0, UVES 5.3.0, VIMOS 2.9.9 and X-SHOOTER 2.3.0

Sep 19, 2013: KMOS 1.2.4 has been released.

Sep 09, 2013: UVES 5.2.5 has been released.

May 24, 2013: GIRAFFE 2.11.1 has been released.

May 06, 2013: New releases for all pipelines and Reflex 2.4 are now available.

Introduction

In collaboration with the various instrument consortia, the Pipeline Systems Department has undertaken to implement data reduction pipelines for the most commonly used VLT/VLTI instrument modes. These data reduction pipelines have three main purposes:

  • Data quality control - pipelines are used to produce the quantitative information necessary to monitor instrument performance (see the ESO Quality Control web pages for more information).
  • Master calibration product creation - pipelines are used to produce master calibration products (e.g. combined bias frames, super-flats, wavelength dispersion solutions).
  • Science product creation - using pipeline-generated master calibration products, science products are produced for supported instrument modes (e.g. combined ISAAC jitter stacks; bias-corrected, flat-fielded FORS images, wavelength-calibrated UVES spectra). The accuracy of the science products can be limited both by the quality of the available master calibration products and by the algorithmic implementation of the pipelines themselves. In particular, adopted reduction strategies may not be suitable for all scientific goals. Therefore, ESO assumes no responsibility for the usefulness of reduced data for any specific scientific project.

Pipelines Running Environments: Reflex, Gasgano and Esorex

Reflex is the recommended way to reduce ESO data. It automatically organizes input files according to their category and runs the entire reduction chain at the push of a button. It supports break points in the reduction sequence in order to inspect and interact with intermediate and final products and rerun the corresponding step if necessary. Please see this table for a list of the supported instrument modes.

Gasgano is a Java-based data file organizer developed and maintained by ESO. It can be used to manage and organize in a systematic way the astronomical data observed and produced by all VLT compliant telescopes. Gasgano is a graphics-based software tool for organising and viewing data files produced by the VLT Control System. Gasgano offers functionalities for data viewing, grouping, sorting, classification, searching, and filtering of data. And, of course, Gasgano will run any requested CPL recipe on the selected data.

Esorex, a command-line utility for running pipeline recipes is also available. It may be embedded by users at their home institute into data reduction scripts for the automation of processing tasks.

Both Gasgano and Esorex are included in the pipeline public release packages.

Pipeline Reflex-based workflows

This table shows the pipelines released with a Reflex workflow. Follow these steps to install the software (it includes Reflex itself, the pipelines and demo data sets):

  • Check the software prerequisites and installation instructions in the Release Notes of the new Reflex 2.6 workflows
  • Check the "Important Notes and Bug Reports" section of the Release Notes
  • Download the install_reflex script.
  • If necessary, make it executable: chmod u+x install_reflex
  • Execute the command and follow the instructions: ./install_reflex
Pipeline
With links to the instrument pages
Release Information Release Kit Documentation Demo Data Sets Workflow scientifically validated1 Workflow contains interactive windows2
FORS Reflex 2.6 workflows release notes and
4.12.8 pipeline release notes
4.12.8 (2014-04-09) FORS Reflex Tutorial and
FORS Pipeline User Manual
FORS demo data
(downloaded by installation script)
Yes Yes
KMOS Reflex 2.6 workflows release notes and
1.3.1 pipeline release notes
1.3.1 (2014-04-09) KMOS Reflex Tutorial and
KMOS Pipeline User Manual
KMOS Cookbook
KMOS demo data
(downloaded by installation script)
Yes Yes
SINFONI Reflex 2.6 workflows release notes and
2.5.2 pipeline release notes
2.5.2 (2014-04-09) SINFO Reflex Tutorial and
SINFO Pipeline User Manual
SINFONI demo data
(downloaded by installation script)
No No
UVES Reflex 2.6 workflows release notes and
5.4.0 pipeline release notes
5.4.0 (2014-04-09) UVES Reflex Tutorial and
UVES Pipeline User Manual and
FLAMES-UVES Pipeline User Manual and
FLAMES-UVES Reflex Tutorial
UVES demo data
(downloaded by installation script)
Yes Yes
VIMOS Reflex 2.6 workflows release notes and
2.9.14 pipeline release notes
2.9.14 (2014-04-09) VIMOS MOS Reflex Tutorial, VIMOS IFU Reflex Tutorial and
VIMOS Pipeline User Manual
VIMOS demo data
(downloaded by installation script)
No No
X-SHOOTER Reflex 2.6 workflows release notes and
2.4.0 pipeline release notes
2.4.0 (2014-04-09) XSHOO Reflex Tutorial and
XSHOO Pipeline User Manual
XSHOO demo data
(downloaded by installation script)
Yes Yes
  • NOTE 1: Scientifically validated workflows are those that have been confirmed to produce science grade data products. All validated workflows are based on pipelines that have gone through an extensive end-to-end review
  • NOTE 2: This column indicates whether there is at least one workflow for that pipeline that includes an interactive component to adjust pipeline parameters

Pipeline Releases Based on the most Recent CPL Release

The Pipeline Systems Department makes also these data reduction pipelines available to the user community. The table below provides access to the pipelines currently available.

Installation of Multiple Pipelines

The procedure for a simultaneous installation of multiple pipelines on a single account is:
  • Download the desired pipeline kits from the "Release File" column in the table below into an empty, temporary directory.
  • Download the Pipeline-kit installer into the same directory.
  • If necessary, make it executable: chmod u+x install_pipelinekit
  • Execute the command: ./install_pipelinekit *-kit-*.*
  • After the installation the temporary directory with its contents can be deleted.

 

The below pipelines are delivered with CPL 6.4.1, gasgano 2.4.5 and esorex 3.10.2, they can be installed together using the above procedure.
Pipeline
With links to the instrument pages
Release Information Release File
Includes CPL,Gasgano,EsoRex,
and static calibration data
Users Manual and Documentation Data Sets
AMBER 4.3.1 (2014-04-09) 4.3.1 (2014-04-09) AMBER Pipeline User Manual
CRIRES 2.3.2 (2014-04-09) 2.3.2 (2014-04-09) CRIRES Pipeline User Manual
FORS 4.12.8 (2014-04-09) 4.12.8 (2014-04-09) FORS Pipeline User Manual Demo Data (29 MB)
GIRAFFE 2.12 (2014-04-09) 2.12.1 (2014-04-09) GIRAFFE Pipeline User Manual Please check the GIRAFFE Standard Calibration Files page
HAWK-I 1.8.14 (2014-04-09) 1.8.14 (2014-04-09) HAWK-I Pipeline User Manual Demonstration Package
(2,5GBytes)
ISAAC 6.1.4 (2014-04-09) 6.1.4 (2014-04-09) ISAAC Pipeline User Manual Static Calibration Files
(50 MB, included in release kit)
KMOS 1.3.1 (2014-04-09) 1.3.1 (2014-04-09) KMOS Pipeline User Manual
KMOS Cookbook 0.9
KMOS Reflex Tutorial 1.1
MIDI 2.8.3 (2014-04-09) 2.8.3 (2014-04-09) MIDI Pipeline User Manual
NACO 4.3.4 (2014-04-09) 4.3.4 (2014-04-09) NACO Pipeline User Manual
SINFONI 2.5.2 (2014-04-09) 2.5.2 (2014-04-09) SINFO Pipeline User Manual
The SINFONI pipeline
(paper on talk presented at ADA IV 2006)
Calibration Database
(255MBytes, complete set of master calibration FITS files provided only for comparisons)
Demonstration Package
(1.2GBytes)
SOFI 1.5.6 (2014-04-09) 1.5.6 (2014-04-09) SOFI Pipeline User Manual
UVES 5.4.0 (2014-04-09) 5.4.0 (2014-04-09) UVES Pipeline User Manual
FLAMES-UVES Pipeline User Manual
Demonstration Package
(2.0 GBytes)
VIMOS 2.9.14 (2014-04-09) 2.9.14 (2014-04-09) VIMOS Pipeline User Manual Demonstration Package
(1.7GBytes)
VISIR 3.5.1 (2014-04-09) 3.5.1-1 (2014-04-09) VISIR Pipeline User Manual
XSHOO 2.4.0 (2014-04-09) 2.4.0 (2014-04-09) XSHOO Pipeline User Manual Additional Telluric Model catalog for NIR arm Standard Star data reduction
(190MBytes)
DETMON 1.2.7 (2014-04-09) 1.2.7 (2014-04-09) DETMON Pipeline User Manual

 

Additional tools for data visualisation and analysis.
Package
Release Information Release File
Includes CPL,Gasgano,EsoRex,
and static calibration data
Users Manual and Documentation Data Sets
CASA 3D Viewer Current CASA Release Current CASA Release 3D Visualisation Tool Manual
3D Visualisation Tool Cookbook
Visualisation Tool Data Format Specifications
Cookbook Demo Data
Sky Correction Tools First public release 1.0.0 (2014-04-01)   Molecfit User Manual
Molecfit GUI and Tutotial
(Molecfit) Smette et al. (2014, A&A, submitted)
(Molecfit) Kausch et al. (2014, A&A, submitted)
(Skycorr) Noll et al. (2014, A&A, submitted)
The Cerro Paranal Advanced Sky Model
(Sky model) Noll et al. (2012, A&A 543, A92)
(Sky model) Jones et al. (2013, A&A 560, A91)
Moehler et al. (2014, A&A, submitted)
Telluric transmission spectra, see Moehler et al. (2014, A&A, submitted)

 

Previous Pipeline Releases

The Pipeline Systems Department also makes these data reduction pipelines available to the user community. They are based on older versions of CPL and can in general not be installed together.

Pipeline
With links to the instrument pages
Release Information Release File
Includes CPL,Gasgano,EsoRex,
and static calibration data
Users Manual and Documentation Data Sets
EFOSC 2.2.2 (2011-04-07) 2.2.2 (2011-04-07) EFOSC User Manual 1.0 (2011-02-02) Static Calibration Tables (included in kit) Demo Data (100 MB)
UVES (DISCLAIMER) 2.9.10 (2012-06-13) 2.9.10 (MIDAS based) UVES Pipeline User Manual 2.9.10
FLAMES-UVES Pipeline User Manual 2.9.10
A set of reference and master calibration data is included in the pipeline package.
DISCLAIMER:
MIDAS based release is offered with very limited support. We recommend users to use the Reflex based kit release.

Publications, Reference Documents, and Links


VLT-SPE-ESO-19000-1618, Data Flow for VLT/VLTI Instruments Deliverable Specifications (version 3)

VLT Science Products Produced by Pipelines, 2004, D.R. Silva & M. Peron, ESO Messenger 118.

The ESO Quality Control Group processes VLT data and monitors the instrument and data quality and also provides a summary of current pipeline status.