Pipeline Containers
Table of Contents
- Introduction
- Prerequisites
- Building and Running the Apptainer Container
- Installing new/additional Pipelines
- Deleting Installed Pipelines
- Recommendations
- Troubleshooting
Instructions
Installation of ESO Pipelines under Ubuntu
We do not provide native pipeline binaries for Ubuntu. Although you can compile the pipelines from source, a more convenient method to install the pipelines and ESOreflex on Ubuntu is to use Apptainer to install the Fedora RPMs in a container. These instructions will guide you through using Apptainer to set up and run the ESO VLT pipelines.
Apptainer, formerly known as Singularity, is a container platform designed for scientific and high-performance computing. It provides a way to encapsulate software environments and dependencies, making it easier to run complex applications in a consistent manner across different systems. A particular advantage of using Apptainer over a native installation is that the pipelines can be installed in user space, and if necessary several versions of a pipeline can be installed without interfering with each other.
Other Linux distributions
While the instructions below are for Ubuntu and other debian system Linux distributions, Apptainer can also be installed under any other modern Linux distribution, including Fedora.
Pre-requisites
1. Installation of Apptainer
The first step is to install Apptainer. You can refer to the official documentation for detailed installation instructions for any modern Linux distribution. Here, we re-produce the specific commands that should be used to install Apptainer in Ubuntu:
sudo apt update
sudo apt install -y software-properties-common
sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt update
sudo apt install -y apptainer
Addressing user privileges restrictions
Ubuntu 24.04 restricts user namespaces by default, which can cause errors when creating mount namespaces during the execution of the apptainer build task. Before running apptainer you must execute the following command with sudo privileges:
echo "kernel.apparmor_restrict_unprivileged_userns = 0" | sudo tee /etc/sysctl.d/99-userns.conf
sudo sysctl --system
To build containers on versions prior to Ubuntu 24.04, you must execute the following command (to create entries in /etc/subuid and /etc/subgid to allow a non-root user to perform actions within a container that normally require root privileges) with sudo privileges.
sudo apptainer config fakeroot --add username
How do I know the Ubuntu version?
lsb_release -a
Building and Running the Apptainer Container
1. Download Apptainer definition file for VLT pipelines
To start building your Apptainer image, download the definition file for VLT pipelines.
curl -O https://eso.org/sci/software/apptainer/eso-pipelines.def
2. Build the Apptainer Image
Use the definition file to build the Apptainer image, use the --build-arg command argument to select the specific pipeline(s) to be installed inside of the container or a list of pipelines separated by semi-colons as in the example bellow:
apptainer build --build-arg pipelines="eris;gravity;kmos" pipelines.sif eso-pipelines.def
After this a "SIF" image is created, with the following content:
- The binaries of the specified VLT pipeline(s)
- The workflow definitions (if available --not all ESO pipelines have EsoReflex Workflows) of the specified VLT pipeline(s)
- esorex
- EsoReflex
After installation of the selected pipelines, the following command can be run to display the list of all available workflows:
apptainer exec pipelines.sif esoreflex -l
The installation is configured to search for RAW data in the $HOME/reflex_rawdata directory, but this can be configued interactive once the workflow is opened in EsoReflex, or on the command line when starting EsoReflex.
Important: no demodata is included in the container. Data to be processed must be located either in your $HOME directory (or a subdirectory thereof) or in a directory (or a subdirectory thereof) bound to the container as described in the Binding local directories section below.
| List of available pipelines package names |
|
|
|
|---|
3. Following the EsoReflex Tutorial
If you are new to EsoReflex and/or the pipeline(s) you wish to use, we recommend you follow the Reflex Tutorial for the instrument(s) available from here.
To follow the tutorial of a given instrument, you will need to download the accompanying Demo data package. To download the latest version of the demo data package of a specific pipeline, download and run the following script to retrieve the tar files and extract the demo data into the $HOME/reflex_rawdata directory by default (you can change the directory the data will be extracted to at the first prompt when runnig the script).
curl -O https://eso.org/sci/software/apptainer/eso_download_demodata.sh
bash ./eso_download_demodata.sh
Be aware, depending on the instrument, the extracted demo data will take up between ~200MB (molecfit) and 26GB (espdr). If you do not have room for this in your home directory, see the Binding local directories section below.
Demo data for ESO pipelines is can also be 'directly' downloaded from the following page.
Once the demo data has been downloaded, you can start EsoReflex and follow the tutorial with:
apptainer exec pipelines.sif esoreflex <wkf_name>
where <wkf_name> is the name of the workflow as listed in the 'esoreflex -l' command above.
The demo data packages are ONLY required to follow the tutorial, so when you are finished with the tutorial, you can simply remove the demo data:
rm -fr $HOME/reflex_rawdata/<inst>
where <inst> is the usually the name of the instrument whose demo data package you downloaded.
4. Running the ESO VLT Pipeline with ESOReflex on your data
Esoreflex can now be started loading for example the gravity workflow with the following command:
apptainer exec pipelines.sif esoreflex gravity
To process data in the directory $HOME/my_reflex_data:
apptainer exec pipelines.sif esoreflex -RAW_DATA_DIR $HOME/my_reflex_data gravity
Also, creating aliases in bash (eg: ~./bashrc) for the apptainer commands could be useful to enhance the user experience.
alias esoreflex='apptainer exec /`path-in-your-machine`/pipelines.sif esoreflex'
You can then start and use EsoReflex by typing esoreflex, e.g. type "esoreflex -l" to see the installed workflows.
In the same line EsoRex can be executed directly to run the different available recipes:
apptainer exec pipelines.sif esorex --help
Also, the cmdline.sh script produced after a workflow execution can be called in the same way:
apptainer exec pipelines.sif $HOME/reflex_data/reflex_book_keeping/...../cmdline.sh
5. Binding local directories
By default, the pipelines running in apptainer have access to the following locations:
- The user’s home directory ($HOME)
- The current working directory from which the apptainer command was started.
If you need to access other directories outside of your $HOME (making a symbolic link in the $HOME directory will not work), an easy method is to bind that/those directories to the container using the following command:
apptainer exec --bind /my/data/dir:/data pipelines.sif esoreflex gravity
Installing new/additional Pipelines
Installing new or additional pipelines in an exisiting container is not possible. One simply has to create a new container with the pipeline(s) one requires. Creating one container for each pipeline is possible, but will take up more space in total than one container for all required pipelines, but there are use case where one PL per container might be best.
To make a new container restart from the Building and Running the Apptainer Container section. If it has been a while (more than say 1month) since you previously built an ESO Pipelines container, it's probably a good idea to download the latest version of the eso-pipelines.def file to be sure you have the latest version of the pipelines and core system.
Deleting Installed Pipelines
Simply remove the SIF image, e.g.
rm -fr pipelines.sif
Recommendations
- Monitor Disk Usage: Be mindful of the disk space used by Apptainer containers, specially your $HOME directory. Apptainer will cache SIF container images generated from remote sources, under $HOME/.apptainer/cache by default, although the location of the cache can be changed by setting the APPTAINER_CACHEDIR environment variable.
- RPM packages: Installing pipelines with top-level meta packages, can lead to the use of a lot of disk space since all standard components are included. If you only need certain parts, such as a specific recipe or demodata, you should run the individual command as specified in the RPM installation section.
- Security: It is important for the user to take responsibility for keeping the Apptainer images updated to ensure optimal performance and system security. We suggest keeping the Apptainer definition files (.def) updated to the most recent one published in the ESO portal.
Troubleshooting
- DISPLAY: If your OS is using a different DISPLAY identifier, this can easily be modified by passing the DISPLAY environment variable to the apptainer exec command:
apptainer exec --env "DISPLAY=:1" pipelines.sif ...Additionally, in some circumstances, it may be necessary to grant the Apptainer container access to your local display. This can be achieved by running the following command:xhost +local: