EsoRex

CPL Title Graphic

EsoRex is the ESO Recipe Execution Tool. It can list, configure and execute CPL-based recipes from the command line.

Introduction

One of the features provided by the CPL is the ability to create data-reduction algorithms that run as plugins (dynamic libraries). These are called recipes and are one of the main aspects of the CPL data-reduction development environment.

As these recipes are dynamic libraries, it is not possible to run them directly from the command line. However, ESO provides several tools to do this, thus saving recipe developers the need to write such an application themselves. One of these is GASGANO (a GUI-based tool) and the other is EsoRex (which runs from the command line) and is described here.

List of features

In addition to running recipes, EsoRex provides the following features:

  • List recipes available in the specified directories
  • Multiple recipe-path specification
  • Recursive recipe-path searching
  • Log file level-control and management
  • Ability to create links to recipe products
  • Recipe product naming control
  • Generation of recipe man pages
  • Permission control for recipe products
  • Command line and configuration file parameters
  • Can automatically generate configuration files
  • Able to measure and report recipe execution times
  • Optionally, print a report of memory-leaks in the recipe

Obtaining and Installing EsoRex

EsoRex may be obtained from the download page. Installation instructions are provided in the README file that is included in the EsoRex distrubution.

Using EsoRex

EsoRex takes a number of command parameters in order to determine the required operation.

In general the command-line of an EsoRex call has the following structure: 
    
    esorex [esorex-options] [recipe [recipe-options] [sof]]

Note that after the command itself, all the command-line arguments are grouped according to their function. First come the EsoRex options (a full list of these can be determined by typing esorex --help. Then, optionally, the recipe may be specified. Following this are any command line options for the recipe itself, and a sof (set of frames) file. (Note that it is possible to list several sof files, in which case EsoRex will treat them as if they were appended in a single file.)

The use of these options is described in the following sections.

Getting help on EsoRex

    
    $ esorex --help

This lists all the command-line options for the EsoRex application itself.

Using an EsoRex configuration file

EsoRex has quite a lot of input parameters, so it is often much more convenient to use a configuration file. It is even possible for EsoRex to create a configuration file for you (see below).

The configuration file contains the EsoRex options, less the -- switch, but prefixed with esorex.caller.. Blank lines are ignored and lines beginning with # are treated as comments.

Here is an example configuration file: 
    
    # Example EsoRex configuration file
    #
    esorex.caller.recipe-dir=/home/username/EsoRex/Plugins
    esorex.caller.log-dir=.
    esorex.caller.log-file=esorex.log
    esorex.caller.output-dir=.
    esorex.caller.output-prefix=out_

Then, to use this configuration file, use the --config command switch.

For example: 
    
    $ esorex --config=myesorex.rc

Note that the configuration file created here, contains the essential configuration options for normal EsoRex use. This configuration file will be assumed in subsequent examples.

Note also, that EsoRex will also search for a configuration file called esorex.rc in the $HOME/.esorex directory. Any values in this file will be used by default, however any value specified in an explicit configuration file (using the --config option), or directly on the command line will overwrite these values.

Using EsoRex to generate a configuration file

Rather than coding a configuration file by hand, it is possible for EsoRex to create its own configuration file, using sensible defaults, and over-riding them with any additional ones that you provide on the command line.

To do this, use: 
    
    $ esorex --create-config

This will create a file .esorex/esorex.rc in your home directory. If the directory .esorex doesn't exist, it will be created for you. EsoRex will fill in most of the parameters in the configuration file for itself, but if there are any for which sensible defaults can not be determined, then EsoRex will put them in blank but commented out, and write a warning message to the screen.

If an configuration file already exists, then the old one will be copied to .esorex/esorex.rc.bak. This feature can be disabled by using the --cleanup option.

Displaying EsoRex settings

    
    $ esorex --config=myesorex.rc --params

This will show the caller (i.e. EsoRex) parameters. However, unlike the --help which shows the parameters as command line options, with their description, --params will display the variable and its current value, whether that is from the command line or the configuration file.

Listing all available recipes

    
    $ esorex --config=myesorex.rc --recipes

This will search for recipes in the recipe directory (which is specified in either the configuration file, or using the --recipe-dir command line switch). The directory search is recursive. That is, EsoRex will look not only in the recipe directory itself, but in all sub-directories.

⚠
If dynamic libraries other than recipe module libraries exist in the recipe directory tree, they may (depending on contents) cause EsoRex to crash. This is due to a limitation in the ltdl library, and is a known problem. There is no immediate solution, other than to ensure that the recipe directory tree does not include any non-identifiable dynamic libraries, other than genuine recipes.

Getting help on a specific recipe

    
    $ esorex --config=myesorex.rc --help myrecipe

This will display information about the recipe myrecipe, including a list of the recipes options.

Displaying recipe parameters

    
    $ esorex --config=myesorex.rc --params myrecipe

This will show the caller (i.e. EsoRex) parameters, followed by the recipe parameters. It will show the value that would be used were the recipe actually executed. These values will be the latest specified from either the configuration file or the command line. If the parameter was not specified in either of these locations, the default will be displayed.

Setting recipe parameters

To specify parameters for a recipe, put the options after the recipe name.

For example: 
    
    $ esorex --config=myesorex.rc myrecipe --myoption

Of course, one can use the --params option to ensure that the values are correctly read.

For example: 
    
    $ esorex --config=myesorex.rc --params recipeone --op=subtract

Using a recipe configuration file

Like EsoRex itself, it is possible to also specify a configuration file for the recipe options. Note that this is still an EsoRex command, and so should go before the recipe name (only actual recipe options should go after the recipe name).

Thus: 
    
    $ esorex --config=myesorex.rc --params --recipe-config=myrecipe.rc myrecipe

would display the parameters that would be used, given the recipe configuration file myrecipe.rc.

The recipe parameters must be specified in the configuration file as their full names.

As an example, here is what a configuration file for the made up "ZIMOS" instrument might look like: 
    
    #
    # EsoRex configuration file for ZIMOS 'zmspcaldisp' CPL recipe. 
    #
    
    zimos.Parameters.bias.removing.method=Zmaster
    zimos.Parameters.extraction.fuzz=5
    zimos.Parameters.lamp.frames.validate=true
    zimos.Parameters.ids.refine=false
    zimos.Parameters.extraction.window=-1
    zimos.Parameters.flat.apply=false
    zimos.Parameters.badpixel.clean=false
    zimos.Parameters.cosmics.clean=false
    zimos.Parameters.BLA=Input
    zimos.Parameters.slit.model=false
    zimos.Parameters.slit.order=0

Note that like the esorex.rc configuration file, EsoRex will also search $HOME/.esorex for any recipe configuration files and use them as a default. It is also possible to use the --create-config option when a recipe name is also specified, and EsoRex will create a configuration file for that recipe. This can then be altered with a text editor.

Actually running a recipe

Without any --params or --help options, EsoRex will attempt to execute any specified recipe. 

    
    %  esorex --config=myesorex.rc --recipe-config=myrecipe.rc myrecipe data.sof

See the section on Sets of frames (below) for more details on how to actually specify the data.

Note, that EsoRex will create all temporary and intermediate files in the current working directory. Thus, you cannot run mutliple instances of EsoRex to execute several recipes in parallel in the same directory - internal data would be overwritten in a non-deterministic way.
As a solution move to different directories before starting EsoRex in parallel to an already running EsoRex module.

The command-line-option --output-dir only stores the final product in that directory, all temporary files are still put in the current working directory.

Set-of-frames files

A set-of-frames (sof) is a list of the input files which are passed to a recipe. This list of input data is passed to EsoRex in an sof file. A sof file is a simple text file, where each input file is specified with its associated classification and an optional group hint on a separate line. In almost all cases, the group hint is not needed (see below).

The general format of each line of an sof file is as follows: 
    
    full-path-to-file  classification  [RAW|CALIB]

If a line starts with a # it is treated as a comment and ignored when the passing the input files on to the specified recipe. The leading directory components of the path to an input file may be given by an environment variable. Note that only the use of a single environment variable at the beginning of each line is supported (see the following example).

An example file, for the made up "ZIMOS" instrument, might look like this: 
    
    /home/user/data/mos/ZIMOS.2003-12-26T01:05:06.233.fits  MOS_SCIENCE
    /home/user/data/mos/ZIMOS.2003-12-26T01:26:00.251.fits  MOS_SCIENCE
    /home/user/data/mos/ZIMOS.2003-12-26T01:47:04.050.fits  MOS_SCIENCE
    #/home/user/data/mos/ZIMOS.2003-12-26T02:01:37.129.fits  MOS_SCIENCE
    /home/user/data/cal/master_bias4.fits                   MASTER_BIAS
    /home/user/data/cal/grs_LR_red.3.tfits                  GRISM_TABLE
    /home/user/gasgano/extract_table2.fits                  EXTRACT_TABLE
    $DATADIR/cal/badpixel.3.tfits                           CCD_TABLE

For concrete examples for a specific instrument, please check the documentation for that instrument.

Optionally, a third column may be provided. Permitted values are either RAW or CALIB. This is for when a recipe does not identify the type of input file, but as all ESO recipes are required to do so, this column is typically not needed.

Hints for using EsoRex

File permissions

(EsoRex v1.x)
When a recipe is used with the --suppress-prefix option, and the --output-dir is set to the current working directory, then the first execution of a recipe will work correctly, but subsequent executions may fail. This is due to output products being given "read-only" permission (to avoid the potential inadvertant loss of products). The recipe itself is unable to modify the permissions, and thus it fails when attempting to create the file. The solution (other than using a different output directory or prefixes) is to change the permission of these output files or delete them prior to any subsequent execution of that recipe.
(EsoRex v2+)
This problem is less likely to occur in EsoRex v2+, due to the replacement of the --output-overwrite option with the --output-readonly (which is disabled by default). However, a determined user can still reach this situation, in which case the non-readable products must have their permissions changed, as described above.

Configuration files

When creating configuration files, if the the recipe is provided on the command line, then EsoRex will generate the configuration file for this recipe. If no recipe name is given, then EsoRex will generate a configuration file for EsoRex itself. All configuration files are written in the $HOME/.esorex/ directory.

Memory checking

It is possible to get EsoRex (v3.2+)to check for memory leaks in the recipe that it is running, by enabling the --mem-check option. Then, at the conclusion of the recipe execution, and after memory deallocation, a list of all remaining allocated memory will be printed to screen. If there are no memory leaks, then no addition output is displayed.