What is PyCPL?

The Common Pipeline Library (CPL) is the toolkit used to build data reduction pipelines for ESO instruments since 2003. CPL is a set of stable and well tested libraries, implemented in C for reasons of performance and portability. These libraries provide a rich API of fundamental objects and the essential algorithms operating on these objects. Both are the building blocks needed to implement complex, instrument specific data reduction tasks, known as recipes.

The recipes, in turn, are the building blocks of the ESO Instrument Pipelines. They are the tasks which are executed in the different data processing workflows. CPL has been in use for building data procesing pipelines for ESO instruments since 2003, and will continue to be used also in the future for the ELT instruments.

Considering the increasing popularity Python has gained in the astronomical community over the past few years being used for data processing and the prototyping of algorithms, and the choice of Python as the scripting language for operating the ELT, shows the need for providing access to CPL from Python. This is provided by a new python library called PyCPL.

While the native C API of CPL is ideal to implement production ready recipes which have to be maintained over a long period of time, there are a number of use cases where the use of an interpreted language, such as Python, is more efficient or more intuitive. These use cases are what PyCPL will address in two ways:

  • Provide the ability to run existing pipeline recipes (implemented using the CPL C API) and access the results from Python.

  • Provide the ability to implement pipeline recipes in Python.

Taken together makes it possible to develop data processing workflows which can seamlessly combine existing CPL recipes with new Python recipes that leverage the functionality provided by CPL, together with other scientific Python libraries such as numpy, scipy, astropy, pandas, and matplotlib.

Using PyCPL will thus allow astronomers to integrate ESO’s Intrument Pipelines much better in their day to day workflow, it will ease the use of CPL in scripts executed as part of operating the future ELT instruments, and it will allow for rapid recipe prototyping during the developement of instrument pipelines.

The possibility to execute pipeline recipes implemented using the CPL C API from a Python environment was already available before using the python-cpl (see References) package maintained by O. Streicher. PyCPL goes beyond the features offered by python-cpl in that it provides also the ability to implement pipeline recipes in Python using the CPL API to its full extend, and the seamless, transparent use of both kinds of pipeline recipes by the user.

PyCPL Features

PyCPL provides:

  • Language bindings for the complete CPL C API (except for a few special purpose interfaces). The design of the language bindings is such that they have a natural look and feel for Python developers, i.e. they are pythonic.

  • Language bindings for the CPL plugin interface, which is used by tools like EsoRex to execute the recipes of an ESO Instrument Pipeline.

  • A plugin interface for recipes written in Python. With this plugin interface, and the Python bindings of the CPL API, it is possible to implement pipeline recipes in Python which have the same look and feel as a recipe implemented using the CPL C API directly.

PyCPL Current Release

The current release is PyCPL 1.0.0.

This is the first release of PyCPL which is made available for production use.

PyCPL 1.0 is feature complete, and provides a stable API, which was made even more pythonic than the API of the previous early-release versions. The revised API is a result of the feedback that was obtained from the beta testers of the early-release versions. This however means that when moving from a early-release version to PyCPL 1.0 changes to exsisting code may be required.

What is PyEsoRex?

PyEsoRex is new recipe execution tool that complements the PyCPL library. It fills the role of the existing EsoRex recipe execution tool, but extends it by facilitating the execution of recipes implemented either in C using CPL or in Python using PyCPL.

PyEsoRex Features

PyEsoRex provides:

  • A command line interface intended as a ‘drop in’ replacement for EsoRex. The CLI reproduces the functionality and syntax of EsoRex (to the maximum extend possible with a pure Python application), include the use of compatible config file formats.

  • An object-orientated, idomatic Python API for the execution of recipes from within Python code or interactive Python sessions. This builds upon the plugin interface included in PyCPL, providing a higher-level, more full featured interface that includes equivalent functionality for all of the features of EsoRex.

  • Seamless execution of both C and Python recipes using a common interface.

PyEsoRex Current Release

The current release is PyEsoRex 1.0.0.

This is the first release of PyEsoRex which is made available for production use.

PyEsoRex 1.0 is feature complete, and provides a stable API. A number of feature requests, received for version 0.9, are already incorporated.