Table of Contents

Instructions 

Installation of ESO Pipelines under Ubuntu

We do not provide pipeline binaries for Ubuntu. Although you can compile the pipelines from source, a more convenient method to install the pipelines and ESOreflex on Ubuntu is to use Apptainer to install the Fedora RPMs in a container. These instructions will guide you through using Apptainer to set up and run the ESO VLT pipelines.

Apptainer, formerly known as Singularity, is a container platform designed for scientific and high-performance computing. It provides a way to encapsulate software environments and dependencies, making it easier to run complex applications in a consistent manner across different systems.

Other Linux distributions 

While the instructions below are for Ubuntu and other debian system Linux distributions, Apptainer can also be installed under any other modern Linux distribution, including Fedora. The advantage of using Apptainer over a native installation is that the pipelines can be installed in user space, and if necessary several versions of a pipeline can be installed without interfering with each other

Pre-requisites

1. Installation of  Apptainer 

The first step is to install Apptainer. You can refer to the official documentation for detailed installation instructions for any modern Linux distribution. Here, we re-produce the specific commands that be used to install Apptainer under Ubuntu:

sudo apt update
sudo apt install -y software-properties-common 
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer

Addressing user privileges restrictions

Ubuntu 24.04 restricts user namespaces by default, which can cause errors when creating mount namespaces during the execution of build task in apptainer. Before running apptainer you must run the following command with sudo privileges:

echo "kernel.apparmor_restrict_unprivileged_userns = 0" | sudo tee /etc/sysctl.d/99-userns.conf
sudo sysctl --system

To build containers on versions prior to Ubuntu 24.04, you must execute the following command with sudo privileges also.

sudo apptainer config fakeroot --add username

This command is a helper to create entries in /etc/subuid and /etc/subgid to allow a non-root user to perform actions within a container that normally require root privileges,

How do I know the Ubuntu version?
lsb_release -a

Building and Running the Apptainer Container

1. Download Apptainer definition file for VLT pipelines

To start building your Apptainer image, download first the definition file for VLT pipelines.

2. Build the Apptainer Image

Use the definition file to build the Apptainer image, use the --build-arg command argument to select the specific pipeline to be installed inside of the container or a list of pipelines separated by a semi-colon; like in the examples bellow:

apptainer build  --build-arg pipelines="eris;gravity;kmos" pipelines.sif eso-pipelines.def

After this a ".sif" file is created, whit the following content:

  1. The VLT pipeline binaries
  2. The VLT pipeline workflows definitions for the given pipeline

After installation of the selected pipelines, the following command can be run to display the list of all available workflows:

apptainer exec pipelines.sif esoreflex -l

Important: no demodata is included inside of the container. Real processing data must be located in your $HOME directory or binded to the container as described in the Binding directories section.

List of available pipelines package names
  • amber 
  • cr2re
  • crire
  • detmon
  • efosc
  • eris
  • esotk
  • espda
  • espdr
  • fors
 
  • giraf
  • gravity
  • harps
  • hawki
  • iiinstrument
  • isaac
  • kmos
  • matisse
  • midi
  • molecfit
  • muse
  • naco
  • nirps
  • sinfo
  • sofi
  • spher
  • uves
  • vcam
  • vimos
  • visir
  • xshoo

3. Running the ESO VLT Pipeline with ESOReflex

Esoreflex can now be started loading for example the gravity workflow with the following command:

apptainer exec pipelines.sif esoreflex gravity_wkf

Also, creating aliases in bash (eg: ~./bashrc) for the apptainer commands could be useful to enhance the user experience.

alias esoreflex='apptainer exec /`path-in-your-machine`/pipelines.sif esoreflex'

You can then start and use EsoReflex by typing esoreflex, e.g. type "esoreflex -l" to see the installed workflows.

In the same line EsoRex can be executed directly to run the different available recipes:

 

apptainer exec  eso-pipelines.sif esorex --help

Also, the cmdline.sh script produced after a workflow execution can be called in the same way:

apptainer exec pipelines.sif $HOME/reflex_data/reflex_book_keeping/...../cmdline.sh

4. Demo data and local directories binding

The installation assumes that raw_data are located in the directory ~/reflex_rawdata.

Demo data for ESO pipelines is available and can be downloaded from the following page

To download the demo data for the latest version of a specific pipeline, download and run the following script to retrieve the tar files. After using this script, the content will be extracted into the $HOME/reflex_rawdata directory by default.

./eso_download_demodata

By default, the pipelines running in apptainer have access to the following locations:

  1. The user’s home directory ($HOME)
  2. The current working directory from which the apptainer command was started.

Furthermore, if you need to mount directories outside of your $HOME, an easy method is binding your data within the container. Pipelines cannot access data stored on other disks, even when a symbolic link points to them. If your data is stored on another disk, you need to mount this disk using the following command:

apptainer exec --bind /data:/data pipelines.sif esoreflex gravity_wkf

Recommendations

  1. Monitor Disk Usage: Be mindful of the disk space used by Apptainer containers, specially your $HOME directory. Apptainer will cache SIF container images generated from remote sources, under $HOME/.apptainer/cache by default, although the location of the cache can be changed by setting the APPTAINER_CACHEDIR environment variable.
  2. RPM packages: Installing pipelines with top-level meta packages, can lead to the use of a lot of disk space since all standard components are included. If you only need certain parts, such as a specific recipe or demodata, you should run the individual command as specified in the RPM installation section.
  3. Security: It is important for the user to take responsibility for keeping the Apptainer images updated to ensure optimal performance and system security. We suggest keeping the Apptainer definition files (.def) updated to the most recent one published in the ESO portal.

Troubleshooting

  1. DISPLAY: If your OS is using a different DISPLAY identifier, this can easily be modified by passing the DISPLAY environment variable to the apptainer exec command: apptainer exec --env "DISPLAY=:1" eso-pipelines.sif ... Additionally, in some circumstances, it may be necessary to grant the Apptainer container access to your local display. This can be achieved by running the following command: xhost +local: