ESO Data Reduction Pipelines and Workflow Systems
ESO provides data reduction pipelines for most current VLT instruments and some La Silla instruments and will provide them for all new VLT and ELT instruments (c.f. complete list of all current instrument pipelines + manuals + source code).
ESO pipelines consist of of series of reduction steps called "recipes" that can be executed one at a time. These recipes can be encoded in Python or C. In order to execute the recipes, they need to be fed with raw data, static calibration data and (possibly) the output from other recipes.
Two fully automated workflow systems to organise data, configure and execute recipes in the appropriate sequence, and graphically visualize the results are available, EsoRelfex and EDPS.
- EsoReflex is currently the most comprehensive system that supports all pipelines, however further development has now been discontinued, and its retirement is expected in 2026.
- EDPS is a modern system that will eventually replace EsoReflex. It is fully functional and provides better performance than EsoReflex, but its GUI is still under development.
The above systems use EsoRex or PyEsoRex to execute individual recipes. These tools can also be used to execute recipes one at a time directly from the command line or in shell scripts or other data processing systems. Recipes can also be called directly from Python using the PyCPL interface. There are currently no supported tools to carry out the data organisation, association or visualization outside of the EsoReflex and EDPS.
Test and Science data
Test data sets to demonstrate the reduction are available for each pipeline. They can be obtained as part of the installation procedure, or downloaded with this script. Public science data from ESO telescopes including the necessary calibrations can be downloaded from the ESO Science Archive Facility.
Installation
ESO pipelines are distributed for Linux and macOS and are known to work in Windows. Several installation options are available for each operating system. The different installation options for each operating system are tabulated below in order of recommended preference from higest to lowest, top to bottom. A more detailed discussion of the Pros & Cons of each installation method is provided here in case you want to make a more informed decision regarding the best install method for you/your use case.
The operating system dependent installation methods described below all include installation of EsoReflex, whereas the installation instructions for EDPS are operating system independent (except for RPM, MacPorts and HomeBrew methods which provide EDPS packages).
In all cases installation from source code should be considered as "for experts", or at least users experienced with building/installing from source code either specifically the ESO pipelines or "software packages" in general.
| Recommended installation methods | |||||
|---|---|---|---|---|---|
| macOS | Ubuntu Systems |
Fedora CentOs |
Other Linux |
Windows | |
most
least |
MacPorts(*)
|
|
Homebrew(**)
|
|
WSL2(***) with Ubuntu and Homebrew
WSL2(***) with Ubuntu |
(*) Note, if you already have MacPorts installed, we recommend you keep using MacPorts, unless you want to try out or prefer HomeBrew. Homebrew binary packages are available for most recent versions of macOS and hardware, whereas MacPorts will install from source for current versions of macOS and hardware (except Monterey on Intel).
(**) For Fedora versions that are supported by ESO RPMS, we recommend RPM installation over Homebrew installation, but we often do not have RPMs available for the very latest one or two releases of Fedora, and then in this case we recommend Homebrew installation.
(***) Ensure you have the very latest version of WSL2.