Installing PyCPL

Installation from Pre-Built Packages

Pre-built, RPM packages for Fedora 38 – 40 and MacPorts packages for macOS 12 and later are available from the respective ESO repositories.

Note

This installation method aims at using PyCPL and PyEsoRex in a production environment, i.e. using it for data processing. For developing PyCPL pipeline recipes, additional software tools may be required. For example, support for unit testing and generating documentation. These need to be installed separately depending on the requirements and conventions of the individual development project.

RPM Package Installation

Installing PyCPL and PyEsoRex using the RPM packages will utilize the default Python 3 interpreter of the system. In addition to PyCPL and PyEsoRex itself all dependencies needed to run pipeline recipes will be installed automatically too.

Step 1:

In case the ESO RPM repository is not yet configured on the machine where PyCPL and PyEsoRex will be installed, then please follow the ESO RPM repository configuration instructions to set up the repository.

Step 2:

With the ESO repository configured, a system wide installation of PyCPL and PyEsoRex can be set up running the following command (requires sudo or root privileges) in a terminal:

$ sudo dnf install python3-pycpl python3-pyesorex

MacPorts Package Installation

Installing PyCPL and PyEsoRex using MacPorts will use a suitable Python 3 interpreter from MacPorts. Both packages are available for a variety of Python 3 versions which can be installed explicitly. Alternativley, if no specific version is requested, the active default Python 3 version will be used. Installing PyCPL and PyEsoRex from MacPorts will also install all required dependencies.

Step 1:

In case the ESO MacPorts repository is not yet configured on the machine where PyCPL and PyEsoRex will be installed, then please follow the ESO MacPorts repository configuration instructions to set up the repository.

Step 2:

With the ESO Repository configured, a MacPorts installation of PyCPL and PyEsoRex can be set up running the following command (requires sudo privileges) in a terminal:

$ sudo port install py312-pyesorex

This will install PyCPL and PyEsoRex for Python 3.12. It is possible to install PyCPL and PyEsoRex for multiple Python 3 versions if needed.

An installation for the active default Python 3 version is done by installing the generic package, which does not have the encoded Python version in its name:

$ sudo port install py-pyesorex

This will install the packages either for the MacPorts default Python 3 version, or for the Python version which was previously selected as the default using the MacPorts port select command.

Step 3: (optional)

After Step 2 is finished, PyEsoRex can for example be called using the pyesorex-3.12 command. Of course, the version suffix may vary depending on the Python 3 version PyEsoRex was actually installed for.

To be able to call PyEsoRex without a version suffix simply as pyesorex, an installed version of PyEsoRex can be set as default. For example, to make the Python 3.12 version of PyEsoRex the default, run:

$ sudo port select --set pyesorex py312-pyesorex

Installation from Sources

While a system wide installation of PyCPL and PyEsoRex is of course supported, the following installation instructions will show an installation using a Python virtual environment. It is expected that this is a more common approach followed by individual users when installing python packages.

It is however assumed that the dependencies needed to build and use PyCPL and PyEsoRex can be installed using the system’s package management system or by installing an ESO instrument pipeline source kit.

Software Prerequisites

In order to build, install and use PyCPL and PyEsoRex the following software packages, with the given minimum version (if indicated), is needed:

  • C++ compiler supporting C++17 (GCC 9.5, Clang 5)

  • CMake 3.12

  • Python 3.9 (including the development package, i.e. headers and libraries)

  • Python packages:

    • setuptools 45.0.0

    • pybind11 2.8.0

    • astropy

    • pandas (optional)

    • Sphinx (optional, for generating the API documentation)

    • pytest (optional, for running the testsuite)

    • scipy (optional, for running the testsuite)

  • CPL 7.2.2

Depending on whether PyCPL and PyEsoRex will be installed system wide, local to the user, or in a Python virtual environment (or alike) the required packages must be available in the respective environment. While the compiler, Python itself and CPL are generally expected to be provided by the system, or possibly the user’s local environment. CMake and the required Python packages however may be installed into the target virtual environment only, in case a virtual environment approach is chosen.

Getting the Source Code

The most recent version of the PyCPL and PyEsoRex source distributions is available here:

Older releases of PyCPL and PyEsoRex can be found in the PyCPL archive and the PyEsoRex archive.

Users with access to the ESO Gitlab may also clone the PyCPL and PyEsoRex repositories directly (ESO internal access only):

$ git clone https://gitlab.eso.org/cpl/pycpl
$ git clone https://gitlab.eso.org/cpl/pyesorex

Installing Development Tools

The following shows how to install the standard dependencies for an ESO pipeline source installation. These dependencies are also needed to build PyCPL. These dependencies can be installed using the package manager of the different systems. Please skip to the section that applies to your environment.

Fedora 36 and newer:
$ sudo dnf install gcc gcc-c++ zlib-devel make gzip bzip2 tar perl gawk sed grep coreutils pkgconfig
$ sudo dnf install cmake python3-devel
Ubuntu 20.04 and newer:
$ sudo apt install gcc g++ zlib1g-dev make gzip bzip2 tar perl gawk sed grep coreutils pkg-config
$ sudo apt install cmake python3-dev python3-venv
macOS 12 and newer:
$ sudo port install pkgconfig autoconf automake libtool bzip2
$ sudo port install cmake python312

To set Python 3.12 as the default Python version to use on macOS, run

$ sudo port select --set python python312
$ sudo port select --set python3 python312

Installing CPL

There are several ways to install CPL. Depending on the use case one may choose one way or the other. If CPL is already installed on your system, because you have already installed an ESO instrument pipeline for instance, it is recommended that you use this CPL installation for building PyCPL, provided that the existing CPL installation includes the CPL header files, and meets the PyCPL requirements.

Attention

Regardless of which of the following methods you use to install CPL, the root directory of the CPL installation will later be needed to correctly build PyCPL.

Installing CPL from the ESO repositories

Binary packages of CPL are available from the ESO repositories for a number of different platforms. The information for which platforms binary packages of CPL are currently available can be found on the VLT Instrument Pipelines page.

Detailed instruction on how to configure and use the ESO repositories are available at the pipeline webpage for Linux (RPM) and macOS systems.

With the repositories configured you will have to install the CPL packages:

Fedora 36 and newer:

$ sudo dnf install cpl-devel

macOS 12 and newer:

$ sudo port install cpl

This will install CPL, including the required CPL headers, into /usr on Linux systems, and into /opt/local on macOS systems.

All pipeline packages which are available from these repositories can be used together with PyCPL.

Installing CPL using a Pipeline Source Kit

When installing an ESO instrument pipeline using one of the pipeline source kits which are available from the VLT Instrument Pipelines page, the CPL which is part of the source kit will be automatically installed.

For use with PyCPL the source kit maybe be installed using either the pipeline installer script, or the Reflex installer script. Instructions on how to install an ESO pipeline source kit using any of the two installer scripts can be found on the VLT Instrument Pipelines page.

The list of available pipeline source kits can be found here. All kits shown in this table provide a CPL which meets the PyCPL requirements.

Using this method CPL will be installed into the directory that is given when the installer scripts ask for an installation directory.

Installing CPL from Sources

It is also possible to install CPL using the CPL source distribution. This is the recommended way for developers working on an instrument pipeline project, or expert users who are familiar with the CPL build system and the needed developer tools. Using this method should hardly be necessary for a normal user.

To install CPL, and possibly its dependencies, using their respective source distributions, please refer to the installation instructions coming with the individual source packages. The installation instructions of CPL are included in the source package, which is available from the CPL download page

Choosing this method to install CPL gives you complete freedom where CPL, and possibly its dependencies, will be installed. It is however up to you to know where to point the PyCPL build system so that it finds the CPL installation.

Creating a Python Virtual Environment

To prepare for building and installing PyCPL and PyEsoRex a Python virtual environment may be created. To do so choose a location where the virtual environment is going to be installed. If you have used virtual environments before, you may have such a location already and it may be used to host the Python virtual environment for PyCPL and PyEsoRex as well.

In the following <venv-root-dir> will have to be replaced by the path you have chosen as the installation directory of the virtual environment.

$ python3 -m venv <venv-root-dir>/pycpl

Installing PyCPL and PyEsoRex

Once the required dependencies are installed the PyCPL and the PyEsoRex source distributions can be installed.

Preparing the Environment

The first step is to setup the environment so that PyCPL uses the correct CPL installation. In order to point the PyCPL build system to the appropriate CPL installation the environmet variable CPLDIR has to be set to the root directory of the CPL installation. For instance, if the CPL libraries are installed in the directory /usr/lib, then CPLDIR must be set to /usr. Hints on where CPL might be installed are given in the Section Installing CPL.

$ export CPLDIR=<cpl-root-dir>

Note

If in doubt where CPL is installed, try to find the file libcplcore.so or libcplcore.dylib depending whether you use a Linux system or a macOS system, respectively.

Make sure the libraries from the CPL installation located in CPLDIR are preferred when looking for libraries. This is particularly important if you have more than one CPL installation on your system, but it is also recommended if there is only a single CPL installation.

This is done by adding the CPL library directory to the beginning of the library search path. The environment variable names differ, depending on the OS you are using:

Linux:

$ export LD_LIBRARY_PATH=$CPLDIR/lib64:$CPLDIR/lib:$LD_LIBRARY_PATH

macOS:

$ export DYLD_LIBRARY_PATH=$CPLDIR/lib:$DYLD_LIBRARY_PATH

Note

If you exit from your shell session and start a new one later you will have to set these previously defined environment variables again before you can continue with the installtion.

The same applies if you are using PyCPL or PyEsoRex in a newly started shell environment.

The next step is to activate the Python virtual environment:

$ . <venv-root-dir>/pycpl/bin/activate

At this point it is recommended to update pip, setuptools, wheel and build, in particular if your virtual environment was created from a somewhat older Python installation:

(pycpl)$ python3 -m pip install --upgrade pip setuptools build wheel

As a last step before actually building PyCPL it may be necessary to change the default PyCPL recipe search path. By default PyCPL expects recipes to be found in a subdirectory of the CPL installation. This default may however not be suitable, for instance if CPL is installed into a system location where a normal user is usually not able to put his own recipes.

Attention

If the default recipe directory does not exist, a warning will be issued when PyCPL is built and when the Python module is imported. Usually this default recipe location does not exist if CPL is installed stand-alone, i.e. not as part of a pipeline source kit installation.

This default location can be changed when building PyCPL using the environment variable PYCPL_RECIPE_DIR and point it to an appropriate, existing directory.

Typical locations where recipes are installed may depend on the method used to install pipeline packages, and the operating system. For instance, using the package managers to install pipeline packages from the ESO repositories will install recipes into /usr/lib64/esopipes-plugins and /opt/local/lib/esopipes-plugins on 64-bit Fedora systems and macOS, respectively. If pipeline packages are installed using a source kit the recipe location will be <prefix>/lib/esopipes-plugins, where prefix is the root directory of the installation given by the user during the installation process.

Note

It is of course always possible to change this default recipe location later, at runtime by changing the recipe search path from the Python interpreter prompt, a script, or through a PyEsoRex command line option. Setting PYCPL_RECIPE_DIR will only change the built-in default. However, this path will persist as a default setting until PyCPL is eventually re-compiled. Thus, a reasonable location should be chosen.

Assuming that all pipeline recipes will be installed locally in a user account into the existing location $HOME/pipelines/lib/esopipes-plugins the default PyCPL recipe location can be changed setting the variable like this:

(pycpl)$ export PYCPL_RECIPE_DIR=$HOME/pipelines/lib/esopipes-plugins

The environment variable PYCPL_RECIPE_DIR is only used for building PyCPL. Once the installation of PyCPL is finished, the variable may be unset again.

Building PyCPL and PyEsoRex

With the environment set PyCPL and PyEsoRex can be build and installed from the source distribution packages in the one of the following ways:

Option 1:

The easiest way to install the latest available version of PyCPL and PyEsoRex, without the need for downloading the source packages first, you can use the following command:

(pycpl)$ python3 -m pip install --extra-index-url https://ftp.eso.org/pub/dfs/pipelines/libraries pycpl pyesorex
Option 2:

To install PyCPL and PyEsoRex after getting the source distributions from the locations given in Getting the Source Code and placing them into a temporary directory <tmpdir>, use the command:

(pycpl)$ python3 -m pip install <tmpdir>/pycpl-1.0.2.tar.gz <tmpdir>/pyesorex-1.0.2.tar.gz
Option 3:

Installing PyCPL or PyEsoRex from a working copy of their respective GitLab repository, or a directory containing the unpacked source distribution package of PyCPL or PyEsoRex, the following needs to be done. For instance in the case of PyCPL:

(pycpl)$ cd <tmpdir>/pycpl-1.0.2
(pycpl)$ python3 -m pip install .

At this point the installation of PyCPL and PyEsoRex is finished and you may unset the environment variable PYCPL_RECIPE_DIR as it is no longer needed:

(pycpl)$ unset PYCPL_RECIPE_DIR

Verifying the Installation

Finally verify that the installation actually succeeded:

(pycpl)$ cd <tmpdir>
(pycpl)$ python3

From the Python interpreter prompt import the cpl and the pyesorex modules:

>>> import cpl
>>> import pyesorex

If this succeeds without any messages printed to the terminal then you have successfully installed PyCPL and PyEsoRex API!

To verify that the PyEsoRex CLI is correctly installed:

(pycpl)$ pyesorex --version

or

(pycpl)$ pyesorex --help