MUSE pipeline recipes v2.10.10 are available!

This release is a maintenance release. The software distribution can be obtained here!

The current version of the MUSE Data Reduction Software supports the processing of all MUSE data, and has been thoroughly tested. At the time of this release there are a few known issues, which are summarized in section 4 of the MUSE Pipeline User Manual. The MUSE Data Reduction Software is still actively being worked on. Improvements and updates addressing the known issues will be made available on the download page as soon as they are available and validated.

Users should note that processing of MUSE data is quite demanding in terms of computer equipment. Please have a look at the system requirements before using the MUSE Data Reduction Software!

If you use the MUSE pipeline for your research, please consider citing the MUSE pipeline paper Weilbacher et al. 2020, A&A, 641, A28 ( DOI 10.1051/0004-6361/202037855). It provides a detailed description of the algorithms underlying all the recipes. The MUSE pipeline also has a citable entry in the Astrophysics Source Code Library (ASCL) (see ADS).

In order to further improve the MUSE Data Reduction Software we encourage all users, and in particular the PIs of an observing program to provide feedback to the ESO User Support Department mentioning MUSE pipeline in the subject.

Changes:

Version 2.10.10 (2024-07-19)

  • Public release with EDPS support.

Version 2.9.0 (2024-05-15)

  • Updated static calibrations and EsoReflex oca rules.

Version 2.8.9 (2023-05-31)

  • Public release version.

Version 2.8.7 (2022-05-19)

  • Support for latest version of CPL.

Version 2.8.5 (2021-06-01)

  • Support for latest version of CPL.

Version 2.8.4 (2021-01-29)

  • Use of an improved method, based on the Strehl measurements of the AO system, to compute the effective spatial resolution for NFM mode observations if the sdpCube format is used to save the final data cube.

  • Improvements to the technical recipe muse_lingain:

    • The input frame tags of the recipe were changed from DETMON_LAMP_ON and DETMON_LAMP_OFF to LINGAIN_LAMP_ON and LINGAIN_LAMP_OFF, respectively.
    • The list of QC parameters related to the gain computation have been revised and extended.

  • Reflex workflow fixes:

    • Fixed overplotting of symbols for the detected sources in the interactive actor of the MUSE workflow.
    • Fixed python2 compatibility issue in the MUSE ZAP workflow.
    • Reset the ROOT_DATA_PATH variable of the MUSE ZAP workflow to the standard default value.

Version 2.8.3 (2020-05-20)

  • Changed the order of the Raman contamination correction and the auto-calibration. This addresses residuals visible in the relevant wavelength range due to the non-uniformity of the Raman signal across the field.

  • SPECSYS FITS keyword was added to the data cube headers indicating the spectral reference frame of the cube.

  • ESO Phase 3 format updates: BUNIT keyword was removed from the data cube primary header, since the cubes do not have a primary data unit. The syntax used for FITS keywords containing data unit strings was changed to comply with the ESO Data Interface Control Document v6, sec. 8, when cubes are saved in sdpCube format. The value of EXPTIME is updated to be equal to TEXPTIME for single exposures in sdpCube format.

Version 2.8.1 (2019-10-22)

  • Fixed computation of the extraction radius of the circular aperture flux extraction method for intermediately bad seeing conditions.

Version 2.8 (2019-09-16)

  • New default flux extraction method for WFM observations (multi-step approach, see the packaged NEWS file for details). For NFM observations the flux extraction uses the circular apperture. The extraction method is chosen automatically depending on the field-mode (WFM, NFM) of the observation.

  • An improved algorithm (kd-tree) is used to match the sources detected in astrometric fields with the reference catalog. The new algorithm works for WFM as well as for NFM observations. The list of detected sources is available as a FITS extension in the ASTROMETRY_WCS product file and the astrometric solution is also written to the DATACUBE_ASTROMETRY product as secondary world coordinate system.

  • The default parameters for creating the model of the overscan have been optimized, which results in a cleaner bias correction.

  • The exposure alignment task (muse_exp_align) now also creates a quick look image (PREVIEW_FOV) of the combined field-of-view taking into account the determined field offsets. The resampling is done using the nearest neighbor interpolation scheme. The quick-look image allows for a visual validation of the computed offsets.

  • The option to create a smoothed model of the MASTER_DARK was added to the muse_dark recipe.

  • The package contains updated versions of the geometry table, the astrometric solutions (for both modes, WFM and NFM). Also the Raman line list has been updated as a result of a better physical understanding.

Version 2.6.2 (2019-05-20)

  • Improved determination of the size of the circular (option profile=circle) aperture for the flux calibration in NFM.

  • Improvements of muse_exp_align:

    • The list of detected sources is cleaned before the field offsets are calculated. Sources which are too close to a bad pixel are ignored when computing the field offsets.
    • The SOURCE_LIST frame type is allowed as ancillary input. The source detection can be skipped using the source positions from the input catalog to compute the field offsets.

  • Added QC parameter to muse_geometry to track invalid entries in the geometry_table.

  • Added support for the RAMAN_LINES static calibration to the MUSE Reflex workflow to allow cleaning of the Raman line contamination.

Known Issues

  • Matplotlib 3.2.1 has a bug which causes the muse_exp_align interactive actor of the workflow to crash. The details can be found in the matplotlib ticket. To solve the issue an older or a newer version of matplotlib, which is compatible with the general software prerequisites for workflows, should be used.

  • MUSE Pipeline source kit releases prior to MUSE 2.8 cannot be built with the GCC 9 compiler series!

  • A report from a user was received that installing the MUSE pipeline on a parallel file system (OrangeFS) causes the MUSE recipes to fail (independent on the version of the MUSE pipeline being used), while installations on a standard Linux file system work as expected. Currently ESO does not use parallel file systems for running VLT instrument pipelines and does not recommend it at this point due to the lack of testing on such systems.

    Users having experience with using the MUSE pipeline together with a parallel file system (as installation target and/or for data storage) are kindly asked to provide feedback to the ESO User Support Department mentioning MUSE pipeline in the subject. Please include which file system is used, how it is used (installation target, data storage), and, whether you could use it successfully or experienced any problems. In the latter case please also report your issues.

Legacy Static Calibrations

The pipeline distribution contains the latest set of static calibration files which is available at the time of the package release. However these calibration files may not suitable for processing data taken during previous periods, for instance if an instrument maintenance took place. In particular this applies to the geometric calibration and the associated astrometric calibration.

Therefore, the geometric calibration and the associated astrometric calibration to be used with datasets from previous periods (including Commissioning and Science Verification) are available as a legacy calibration dataset from the download page.

The necessary information on which calibrations should be used for data from the different periods is available in the README file included in the data set.

EsoReflex support

With the release of version 1.0 of the MUSE pipeline package the standard EsoReflex support is available, and covers the complete reduction chain. In addition, the support for aligning and co-adding multiple exposures is available. It can be obtained from the download section.

The workflow is distributed with a demo data package, which will be downloaded by the installer. Note that the size of the demo data set is about 12GB!

A list of scenarios for which the workflow has been successfully used can be found in Section 9 of the MUSE EsoReflex Tutorial. We encourage all users to provide feedback on the workflow to the ESO User Support Department mentioning MUSE pipeline in the subject.

Recovery procedure for vignetted SV data sets

During the first MUSE Science Verifiaction run data sets taken at low temperatures may be affected by vignetting. The cause has been fixed before the last commissioning run and thus will not affect data taken after this last commissioning run. However for the affected SV data sets the pipeline may fail to produce a valid tracing of the slice edges of IFU 6 (most likely slice no. 10). At very low temperatures even the wavelength calibration may fail due to a complete loss of flux in these slices.

A recovery procedure for such cases is available in Section 8.2 of the MUSE Pipeline User Manual. This procedure needs an additional utility, muse_pixtable_erase_slice, which has is available in the MUSE pipeline distribution since version 0.18.2.

In addition, a set of valid trace tables for IFU 6, for both, nominal and extended wavelength range has been prepared and is available here. They are ready to be used in the recovery procedure.

MUSE pipeline recipes version 2.10.10:

The current release version of the MUSE pipeline recipes is 2.10.10, and it is included in the pipeline distribution kit muse-kit-2.10.10.tar.gz together with the following packages:

System Requirements

The MUSE Data Reduction Software is only supported on 64-bit platforms and requires a 64-bit compiler to build correctly!

Minimum requirements:

  • 32 GB of memory
  • 4 (physical) CPU cores
  • 1 TB of free disk space (data storage)

Recommended requirements:

  • 64 GB of memory
  • 24 (physical) CPU cores
  • 4 TB of free disk space (data storage)

The minimum requirements are sufficient to process standard calibration sets (5 exposures) and create the final data cube from a single exposure.

This release of the MUSE pipeline kit is verified and supported on the VLT target platforms:

  • CentOS 7 (x86_64), using gcc 4.8.5
It is also verified that the pipeline kit can be built on the following platforms (x86_64):
  • CentOS 7
  • Rocky Linux 8
  • Alma Linux 9
  • Fedora 31
  • Fedora 34
  • Fedora 36
  • Fedora 37
  • Fedora 38
  • Fedora 39
  • Ubuntu 20.04
  • Ubuntu 22.04
  • Ubuntu 23.04
  • Ubuntu 23.10
  • Debian 11
  • Debian 12
  • macOS 12 (Intel & Apple Silicon)
  • macOS 13 (Intel & Apple Silicon)
  • macOS 14 (Intel & Apple Silicon)
In addition the MUSE pipeline has been built successfully on the following systems using the default system compiler:
  • openSUSE Leap 15.5 (x86_64)

Note for macOS Users:

The MUSE pipeline recipes can be built on the 64-bit Intel platform. However it lacks certain features compared the to a Linux installation, which lead to a degraded performance. In general it is recommended to run the MUSE pipeline recipes on Linux systems!

Installing and running the MUSE pipeline recipes

There are several ways to install the pipeline on your machine. The recommended installation procedure depends on whether you are working on macOS or on Linux.

Installation using RPM repositories

For Fedora 38/39, CentOS 7 and Scientific Linux 7, it is recommended to install the pipeline from our RPM repository. The RPM repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

General installation instructions for RPM packages are provided here.

User-contributed workflows are available for MUSE 2.10.10. The RPM packages of these workflows can be installed using:
dnf list esopipe-*contrib* # (Fedora 33 or newer)

Installation using MacPorts repositories

For Apple macOS 12/13/14, it is recommended to install the pipeline with MacPorts. The MacPorts repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the EsoReflex workflow, demo data and the EsoReflex tool itself.

General installation instructions for MacPorts packages are provided here.

User contributed workflows are available for MUSE 2.10.10. The MacPorts packages of these workflows can be installed using:
sudo port install esopipe-*contrib*

Installation using the EsoReflex installation procedure

Please refer to EsoReflex Software Prerequisites and Installation Instructions for detailed instructions

User-contributed workflows are available for MUSE 2.10.10 and they are installed automatically with the standard instrument workflows when using the install_esoreflex script.
Please note that these user-contributed workflows may have additional, non-standard software dependencies! These must be installed by the users themselves. For detailed information on additional software dependencies please refer to the documentation of the individual workflows!

Installation using the public pipeline kit

To use the MUSE pipeline recipes you will need to retrieve the pipeline distribution kit, unpack and install it.

Prerequisites:

  • The default system C/C++ compiler for one of the OS versions listed above, and standard GNU tools including make
  • To use the latest graphical front-end Gasgano version 2.4.8 a Java Development Kit (JDK) version 1.8.0 or later must be available on your system. You may use either OpenJDK, or Oracle JDK. Most likely, OpenJDK is already part of your Linux distribution and can be installed from its software repositories. The environment variable JAVA_HOME must be set correctly, and the related java executable must be in your PATH.

    Gasgano is known to be resource-intensive: see details of memory and CPU usage in Appendix A of the Gasgano User's Manual.

    Using Gasgano to run MUSE recipes is not supported! It can however be used as a filebrowser.
  • Please check for further software prerequisites here

Installation procedure:

  1. Download the MUSE pipeline kit to a directory on your computer. This may be any directory with the exception of $HOME/gasgano
    $HOME/.esorex
  2. Unpack the distribution file, using the following command:
        tar -zxvf muse-kit-2.10.10.tar.gz
  3. Install: after moving into the top level directory of the unpacked distribution,
        cd muse-kit-2.10.10 it is recommended to perform the installation using the supplied installation script:
        ./install_pipeline This may take a few minutes. Please be patient!

    Note for macOS users:
    On macOS the environment variable JAVA_HOME should be set to JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home prior to running the installation script in order to avoid compilation errors during the installation.

    By default the script will install the MUSE recipes, Gasgano, EsoRex, all the necessary libraries, and the static calibration tables, into a directory tree rooted at $HOME A different path may be specified as soon as the script is run. For instance (user input is boldfaced):
    $ ./install_pipeline
    I am about to install the following software packages:
      gasgano-2.4.8.tar.gz
      cfitsio-4.2.0.tar.gz
      wcslib-4.24.tar.bz2
      fftw-3.3.4.tar.gz
      cpl-7.3.2.tar.gz
      esorex-3.13.8.tar.gz
      muse-2.10.10.tar.gz

    - with the following pipeline calibration file(s):
      muse-calib-2.10.10.tar.gz

    The software installation will be organised in bin/, lib/ and include/
    directories, while the calibration files will be installed in a calib/ directory.

    Where should I install the software packages ? [/home/dummy] /home/dummy/pipelines
    Where should I install the pipeline calibration files ? [/home/dummy/pipelines] /home/dummy/calibrations
     In this case the software is installed under the directory /home/dummy/pipelines and the static calibration tables under /home/dummy/calibrations/calib/muse-2.10.10
    The only exception to all this is the Gasgano tool, that will always be installed under the directory $HOME/gasgano Note that the installer will move an existing $HOME/gasgano directory to $HOME/gasgano.old before the new Gasgano version is installed.

    Important: the installation script would ensure that any existing Gasgano and EsoRex setup would be inherited into the newly installed configuration files (avoiding in this way any conflict with other installed instrument pipelines).

    Alternatively, it is possible to perform a manual installation of the individual components (experienced users only): the README file located in the top installation directory contains more detailed information about a step-by-step installation.

Using the MUSE pipeline recipes

Using EsoRex

  1. To run the EsoRex command line tool, just add the bin of the installation path (see above) to your PATH environment variable:     export PATH="$HOME/pipelines/bin:"$PATH You should also define an environment variable CPLDIR to point to the same path specified for the installation. Possible files to update are:
        $HOME/.bashrc
        $HOME/.profile     Finally, enter the command     esorex --recipes
  2. If the list of MUSE recipes appears, then you have successfully installed EsoRex and the MUSE Data Reduction Software.
  3. Refer to the EsoRex web page for details about the related features and options.
  4. Refer to the MUSE Pipeline User Manual for detailed information about using the recipes.

Using Gasgano to run MUSE pipeline recipes is not supported!

Documentation

The  MUSE Pipeline User Manual (ca. 3.5 MB, 156 pages), is available here for download. Note that the manual is also included in the MUSE pipeline kit.

The GASGANO Users' Manual (1.2 MB, 66 pages) is available for download in the Gasgano web page.

On the EsoRex web page some online documentation about EsoRex can be found.

The CPL manuals are available on the CPL web pages.

In case of problems when opening the documents directly from your web browser, the files may be first saved on disk, and then opened with Acrobat Reader.

Bug Reports

If you experience an unexpected behavior of any component of the MUSE pipeline recipes package, please, first verify that you are using one of the above mentioned supported platforms and refer to the list of known problems and limitations in the current MUSE pipeline release, in section 4 of the MUSE Pipeline User Manual.

For any other issues or requests, please, send a report to the ESO User Support Department mentioning MUSE pipeline in the subject, describing:        

  • the MUSE pipeline version, and the version of other components (e.g., Gasgano, EsoRex, ...) you are using
  • the version of your OS and C compiler.
  • the exact sequence of actions that were performed before the problem occurred
  • what were precisely the symptoms and the possible error message(s)
  • whether the problem is repeatable