A release of the CR2RES pipeline recipes is now available !

A release of the CR2RES pipeline recipes is now available.

Together with the data processing modules, we also provide standalone applications for running them:

  • EsoReflex, a GUI which provides an easy and flexible way of runing pipeline execution workflows.
  • Gasgano, a GUI for data browsing and organisation, providing an intuitive and comprehensive interface to pipeline recipes that are based on the ESO Common Pipeline Library (CPL).
  • EsoRex, a command-line driven utility to launch pipeline recipes, which can be used either as an interactive tool, or can be embedded into scripts for automating some of the data reduction tasks.

Changes:

Version 1.4.2 Public Release (08/02/24)

  • Fixes output of wrong spectral resolution values in wavecal

Version 1.4.1 Public Release (20/12/23)

  • Fix the number of RAW files and the TEXPTIME in the IDP products
  • static calibration header keywords PRO REC RAW removal
  • IDP science product show SNR = -1. in their header
  • spectrum depression in CRIRES science
  • Barycentric correction, using HDRL
  • lower estimates of resolution

Version 1.4.0 Public Release (30/11/23)

  • New QC.FPI.CONTRAST/SEPARATION parameters
  • Added QC UNE FLUX parameter
  • Recurring detlin issues fixed
  • Additional QC parameters for the cr2res_cal_detlin recipe
  • gain value (detector dependant) needs to be passed to cr2res_extract.c and used in the extraction
  • science stare recipe: combined product missing
  • CRIRES pipeline mistakes metrology for science
  • Additional QC parameters for the cr2res_cal_flat recipe
  • Additional QC parameters for the cr2res_cal_wave recipe
  • number of saturated pixel in crires science recipes
  • Polarization demo data in cr2res pipeline
  • Improve documentation
  • Null and Stokes spectrum mixed up, for linear polarization
  • IDP Complient Products
  • Add Correction to the traces in Polarimetry
  • Errors underestimate SNR

Version 1.3.0 Public Release (04/05/23)

Important: This release is solving the issue reported as Important in the 1.2.0 release.

  • Add DRS.TMID keyword in the Obs products
  • Obs 2d Recipe : Save the calibrated combined images as in the other Obs recipes
  • PRO.TYPE is replaced by DRS.TYPE. PRO.TYPE set to REDUCED in all products. Static calibration files updated : PRO.TYPE = 'STATIC'
  • IDP output is now available via the --idp flag for the recipes cr2res_obs_nodding and cr2res_obs_staring
  • Additional QC parameters in cr2res_cal_detlin, cr2res_cal_dark, cr2res_cal_flat
  • QC TILT parameter to measure slit tilt
  • New QC parameter for the cr2res_obs_nodding recipe
  • Add a new recipe parameter (slit curvature degree) to cr2res_util_slit_curv, cr2res_cal_wave
  • The recipes cr2res_obs_nodding, cr2res_obs_pol and cr2res_obs_staring divide the extracted spectra by the blaze if provided as input
  • CRIRES Slit curvature not static anymore: In the reflex WKFs, the wavecal recipe computes the slit curvature from FPET, and stores it in the output TW. This solves the Important issue of the 1.2.0 release announcemnt
  • Poor repeatability of the 1D extraction - fixed
  • CRIRES+ incorrect inter-order light subtraction - fixed
  • The propagated errors do not depend on the extraction width - fixed
  • cr2res nodding workflow fails on M4211 data - fixed
  • cr2res_util_calib error when reducing only 1 detector - fixed
  • Failed swaths affect extraction results - fixed
  • Wrong slit tilt in 2D spectra - fixed
  • Cosmic rays removal added to the Obs recipes
  • Improvement of inter order background subtraction
  • IDP completed for cr2res_obs_nodding and cr2res_obs_staring

Version 1.2.3 Public Release (25/11/22)

  • Fix input filtering of several Actors in the 4 Workflows in order to avoid missing inputs in some rare edge cases
  • Improve documentation
  • Astrometry/Polarimetry data not always reduced in the observatory when the recipe time exceeds 20mn

Version 1.2.0 Public Release (16/09/22)

Important: As of September 2022 the slit has slightly moved. The slit curvature information contained in the static calibration files delivered with the pipeline distribution is therefore not valid any more. It is recommended to use the delivered cr2res_util_slit_curv recipe to regenerate properly the necessary calibration files. See the Pipeline Manual for more detailed information

  • Astrometry/Polarimetry data not always reduced
  • Workflow issue in Paranal
  • 3 New workflows for the Observation modes
  • Traces correction n Polarimetry
  • Update of the User Manual
  • Errors computation issue reported
  • Issue with wavelenght calibration in the reflex workflow

Version 1.1.7 Public Release (21/06/22)

  • Better error handling avoiding its unwished propagation in some cases
  • Move to HDRL 1.5.0
  • Wrong DIT values in products are corrected in cr2res_cal_dark
  • Remove SEQ.NODPOS from nodding products headers
  • Support WAVE_SKY data (SkyObs template) and trigger the cr2res_obs_staring recipe

Version 1.1.6 Public Release (06/05/22)

  • No Change with regard to 1.1.5. Only Dependencies are updated for the yearly common public release

Version 1.1.5 Public Release (05/05/22)

  • Improvement in the Reflex demo data set in K Band

Version 1.1.4 Public/Paranal/QC Release (09/03/22)

  • Significant improvement of the Pipeline Manual
  • Update the HDRL to version 1.4.0
  • Add many new products to cr2res_obs_pol
  • Add the support fot SKY frames in cr2res_obs_2d
  • Add the support for astrometry mode in cr2res_obs_nodding
  • Fix the missing PRO REC CAL in the products
  • Detector linearity recipe and correction have been improved
  • Add --subtract_interorder_column to the four obs-recipes
  • Improvements on the Etalon wavelength calibration
  • QC.RESOL : Use the minimum FWHM, not the median anymore. Use a new selection of catalog lines. Log the position of the retained line, reject outliers.
  • fix normalization of slit_smooth parameter in optimal extraction
  • numerous minor bug fixes and improvements
  • updated static calibrations, in particular TWs with much improved wavelength calibrations
  • Added QC.THROUGPUT

CR2RES pipeline recipes public version:

The public version of the CR2RES pipeline recipes is 1.4.2, and it is included in the pipeline distribution kit cr2re-kit-1.4.2-1.tar.gz together with the following packages:

  • the CFITSIO library, version 4.2.0
  • the Common Pipeline Library (CPL), version 7.3.2
  • Gasgano version 2.4.8
  • EsoRex, version 3.13.8
  • Third-party libraries used by CPL (wcslib, fftw)
  • Third-party libraries required by the CR2RES pipeline

CR2RES pipeline documentation

User Manual: 1.4.2
EsoReflex Tutorial: 1.4.2

CR2RES pipeline recipes System Requirements

The CR2RES pipeline kit version 1.4.2 is verified and supported on the VLT target platform:
  • CentOS 7 (x86_64), using gcc v4.8.5.
It is also verified that the pipeline kit can be built on the following platforms (x86_64, unless specified otherwise):
  • CentOS 7
  • Rocky Linux 8
  • Fedora 31
  • Fedora 34
  • Fedora 35
  • Fedora 36
  • Fedora 37
  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 22.10
  • Debian 10
  • Debian 11
  • macOS 10.15
  • macOS 11
  • macOS 12 (Intel & Apple Silicon)
  • macOS 13 (Intel & Apple Silicon)

Installing and running the CR2RES pipeline recipes 1.4.2

There are several ways to install the pipeline on your machine. The recommended installation procedure depends on whether you are working on a Mac OSX or on Linux.

Installation using RPM repositories

For Fedora 35/36/37, CentOS 7 and Scientific Linux 7, it is recommended to install the pipeline from our RPM repository. The RPM repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

Please refer to the instructions provided here.

Installation using MacPorts repositories

For Apple macOS 11/12/13, it is recommended to install the pipeline with MacPorts. The MacPorts repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

Please refer to the instructions provided here.

Installation using the EsoReflex installation procedure

Please refer to EsoReflex Software Prerequisites and Installation Instructions for detailed instructions

Installation using the public pipeline kit

To install the CR2RES pipeline recipes 1.4.2 you will need to retrieve the pipeline distribution kit, unpack and install it.

Prerequisites:
  • Please check the software prerequisites here.

Installation procedure:

  1. Change directory to where you want to retrieve the CR2RES pipeline recipes 1.4.2 package. It can be any directory of your choice but not:
    • $HOME/gasgano
    • $HOME/.esorex
  2. Download the software and the static calibration data (line catalogs etc.) necessary for running the pipeline recipes:
  3. Unpack using for example the following command:     tar xzf cr2re-kit-1.4.2-1.tar.gz
  4. Install: after moving into the installation directory,     cd cr2re-kit-1.4.2-1 it is recommended to perform the installation using the supplied installer script:     ./install_pipeline (BEWARE: The installation may take several minutes). By default the script will install the CR2RES recipes, Gasgano, EsoRex, all the necessary libraries, and the static calibration tables, into a directory tree rooted at $HOME. A different path may be specified as soon as the script is run. For instance (user input is boldfaced):
    $ ./install_pipeline
    I am about to install the following software packages:
    • cfitsio4.2.0.tar.gz
    • cpl-7.3.2.tar.gz
    • esorex-3.13.8.tar.gz
    • gasgano-2.4.8.tar.gz
    • cr2re-kit-1.4.2-1.tar.gz
    with the following pipeline calibration file(s):
    • cr2re-calib-1.4.2.tar.gz
    The software installation will be organised in bin/, lib/ and include/ directories, while the calibration files will be installed in a calib/ directory. Where should I install the software packages ? [/home/userid] /home/dummy/pipelines
    Where should I install the pipeline calibration files ? [/home/dummy/pipelines] /home/dummy/calibrations
    In this case the software is installed under the directory /home/dummy/pipelines and the static calibration tables under /home/dummy/calibrations/cr2re The only exception to all this is the Gasgano tool, that will always be installed under the directory $HOME/gasgano Note that the installer will move an existing $HOME/gasgano directory to $HOME/gasgano.old before the new Gasgano version is installed.
    Important: the installation script would ensure that any existing Gasgano and EsoRex setup would be inherited into the newly installed configuration files (avoiding in this way any conflict with other installed instrument pipelines).
    Please follow the instructions shown at the end of a successful installation script: they will indicate how to set your PATH and CPLDIR variables in your machine in order to run the pipeline.
    Alternatively, it is possible to perform a manual installation (experienced users only): the README file located in the top installation directory contains more detailed information about a step-by-step installation.

Executing the pipeline 1.4.2

Using EsoReflex

  • To run the EsoReflex workflow, please refer to the CR2RES workflow tutorial available in this table under column Documentation.

Using Gasgano

  1. To run the Gasgano GUI, just add $HOME/gasgano/bin to your PATH environment variable:     export PATH="$HOME/gasgano/bin:"$PATH You should also define an environment variable CPLDIR to point to the same path specified for the installation. Possible files to update are:     $HOME/.bashrc     $HOME/.profile Finally, enter the command     gasgano
  2. If the Gasgano main panel appears, then you have successfully installed Gasgano. Make sure, however, that the expected Gasgano 2.4.8 release is running: the version number is listed at the top of the main Gasgano panel.
  3. Refer to the Gasgano User's Manual to learn how to browse data and run pipeline recipes. Refer to the Pipeline User Manual(s) for detailed information about the recipes usage.

Using EsoRex

  1. The EsoRex executable should be available in the bin directory under your installation path (see above). The PATH environment variable should be upgraded accordingly (similarly as done for Gasgano).
  2. Refer to the EsoRex web page for details about the related features and options.

Documentation

The EsoReflex Users' Manual is availale for download

The GASGANO Users' Manual (~1.2 Mb, 66 pages) is available for download in the Gasgano web page.

On the EsoRex web page some online documentation about EsoRex can be found.

The CPL manuals are available on the CPL web pages.

In case of problems when opening the documents directly from your web browser, the files may be first saved on disk, and then opened with Acrobat Reader.

Bug Reports

If you experience an unexpected behavior of any component of the CR2RES pipeline recipes package, please, first verify that you are using one of the above mentioned supported platforms and refer to the list of known problems and limitations in the pipeline manual of the current CR2RES pipeline release.

For any other issues or requests, please, send a report to the ESO User Support Department mentioning CR2RES pipeline in the subject, describing:        

  • the CR2RES pipeline version (currently 1.4.2), and the version of other components (e.g., Gasgano, EsoRex, ...) you are using
  • the version of your OS and C compiler.
  • the exact sequence of actions that were performed before the problem occurred
  • what were precisely the symptoms and the possible error message(s)
  • whether the problem is repeatable