A release of the VISIR pipeline recipes is now available !

A release of the VISIR pipeline recipes is now available.

Together with the data processing modules, we also provide standalone applications for running them:

  • Reflex, a GUI which provides an easy and flexible way of runing pipeline execution workflows.
  • Gasgano, a GUI for data browsing and organisation, providing an intuitive and comprehensive interface to pipeline recipes that are based on the ESO Common Pipeline Library (CPL).
  • EsoRex, a command-line driven utility to launch pipeline recipes, which can be used either as an interactive tool, or can be embedded into scripts for automating some of the data reduction tasks.

Changes:

Version 4.3.1 (Public release)

  • limit low resolution spectroscopy QC parameters (e.g. sensitivity and mean background before correction) to the 8 um to 13 um range
  • added BACKGD SIGMA QC parameter to spectroscopy output files which computes the standard deviation on the image after background correction and undistortion
  • allow non-interactive use in workflows via esoreflex -n
  • fix interactive gui of imaging workflow using 100% cpu under wxWidgets 3
  • apply jitter correction also when object detection fails
  • increase default eccmax parameter of visir_img_reduce to 3.0 which fixes object detection on the larger aquarius detector in some cases.
  • write intermediate set of frames into the temporary directory if the environment variable VISIR_TEST_MODE is set
  • fixed missing associations of low resolution spectroscopy calibration data to science data in Reflex workflow

Version 4.3.0 (Public release)

  • Update low resolution dispersion relation. The old relation was very inaccurate especially above 11 µm, the new one will produce much better wavelength calibration but the accuracy is still in the order of 5 pixels or 0.05µm.
  • Disable application of the linearity correction to the spectroscopic data. Specially for low signal counts the correction could cause severe signal distortion. It is now only applied to the uncorrected skyframes during the wavelength calibration. The object spectrum is not anymore corrected. Imaging data was already not automatically linearity corrected.
  • The pipeline can now handle any jitter angle and width on both detectors. Previously the pipeline could fail for non-normal angles and width that are larger than the chop throw in some modes.
  • The static bad pixel masks for all the detectors have been merged into a single file. The file now contains four image extensions for the four supported detectors (DRS/AQU, IMG/SPC).
  • Fix imaging Reflex workflow not converting data to Jy units.
  • Fix object selector in imaging Reflex workflow only using half the selected window size.
  • Co-addition is now also performed on multiple cpu cores on Apple OSX.
  • Workaround issues with low file descriptor limit on Apple OSX 10.11.
  • Added classifications for acquisition frames to gasgano oca rules.

Version 4.2.0 (Public release)

  • Documentation updates

Version 4.1.8 (Paranal release)

  • Removed ncycles parameter from visir_img_reduce and visir_spc_reduce. It is still available for visir_util_repack. As it works per nod-cycle it is not very useful for the full reduction recipes.
  • Added planelimit parameter to limit the number planes per nod position which the pipeline reduces.
  • Limited number of planes reduced for imaging science to at most 2000 to avoid slowing down following shorter reductions.
  • Enabled parallel processing of reduction blocks in RBS. This allows e.g. shorter calibration blocks to not blocked by very long burst mode reductions.
  • Use linearity table in automatic spectroscopy reductions.
  • Spectroscopy data is now also jitter corrected which allows using the rr and rl parameters to better handle extended targets.
  • Increased number of reading IO threads to improve performance of reading compressed files.
  • Reduced memory usage of visir_util_repack when receiving very many input files.
  • Improved parallelization depth of coaddition process to better handle the case of very few very large input files.

Version 4.1.7 (Public release)

  • Adapt to new data format written since February 2016.
  • Allow reduction of now commissioned Aquarius Burst mode data.
  • Updated Reflex tutorial.
  • Fix too many open file issues on some platforms.
  • Updated Reflex data association rules.
  • Tune default Gasgano display columns to Aquarius data.
  • A few usuability improvements for Reflex Imaging GUI.

VISIR pipeline recipes public version:

The public version of the VISIR pipeline recipes is 4.3.1, and it is included in the pipeline distribution kit visir-kit-4.3.1-1.tar.gz together with the following packages:


VISIR pipeline documentation

User Manual: 1.7
Reflex Tutorial: 1.1

VISIR pipeline recipes System Requirements

The VISIR pipeline kit version 4.3.1 is verified and supported on the VLT target platform:
  • Scientific Linux 6.3 (x86_64), using gcc v. 4.4.6
Additionally the pipeline kit version is known to build and to produce correct output with esorex on these platforms:
  • Linux Ubuntu 16.04 (amd64), using gcc v. 5.3.1
  • Linux Ubuntu 14.04 (amd64), using gcc v. 4.8.2
  • Linux Ubuntu 10.04 (i386), using gcc v. 4.4.3
  • Mac OS X 10.6.3 (i386), using gcc 4.2.1

Installing and running the VISIR pipeline recipes 4.3.1

There are several ways to install the pipeline on your machine. The recommended installation procedure depends on whether you are working on a Mac OSX or on Linux.

Installation using RPM repositories

For Fedora 20/21/22/23, it is recommended to install the pipeline from our RPM repository. The RPM repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the Reflex workflow, demo data and the Reflex tool itself.

Please refer to the instructions provided here.

Installation using MacPorts repositories

For Apple Mac OS X 10.9 or newer, it is recommended to install the pipeline with MacPorts. The MacPorts repositories include not only the pipeline recipes but also the library dependencies (CPL, etc..), the Reflex workflow, demo data and the Reflex tool itself.

Please refer to the instructions provided here.

Installation using the Reflex installation procedure

Please refer to Reflex Software Prerequisites and Installation Instructions for detailed instructions

Installation using the public pipeline kit

To install the VISIR pipeline recipes 4.3.1 you will need to retrieve the pipeline distribution kit, unpack and install it.

Prerequisites:
  • One of the C++ compilers listed above, and standard GNU tools including make.
  • To install the latest graphical front-end Gasano version 2.4.8 and configure it properly the Java Developer Kit (JDK) version 1.7.0 must be available on your system. If it is not, you may download it from the official Sun Java page distribution, for example http://developer.apple.com/java/download/ for a Mac installation or http://java.sun.com/javase/downloads/ for Solaris and Linux platforms installations. On some modern Linux distributions like Ubuntu you may install the JDK simply by choosing it from the software installer. Furthermore, the related java executable must be in your PATH. You may install and run Gasgano under other operating systems or use a different JDK version, but ESO cannot guarantee that Gasgano will work correctly in such case. Gasgano is known to be resource-intensive: see details of memory and CPU usage in Appendix A of the Gasgano User's Manual.
Note:
In order to retrieve the distribution kit, you may need to configure your Web browser so that it knows how to deal with the files we distribute, i.e. gzipped tar files (.tar.gz filename extension). If you click on one of the links below, your browser should display a file selection panel; if you end up instead with a page full of strange characters, you may try clicking on the link while holding the Shift key down, or configure your browser.

Installation procedure:

  1. Change directory to where you want to retrieve the VISIR pipeline recipes 4.3.1 package. It can be any directory of your choice but not:
    • $HOME/gasgano
    • $HOME/.esorex
  2. Download the software and the static calibration data (line catalogs etc.) necessary for running the pipeline recipes:
  3. Unpack using for example the following command:     tar xzf visir-kit-4.3.1-1.tar.gz
  4. Install: after moving into the installation directory,     cd visir-kit-4.3.1-1 it is recommended to perform the installation using the supplied installer script:     ./install_pipeline (BEWARE: The installation may take several minutes). By default the script will install the VISIR recipes, Gasgano, EsoRex, all the necessary libraries, and the static calibration tables, into a directory tree rooted at $HOME. A different path may be specified as soon as the script is run. For instance (user input is boldfaced):
    $ ./install_pipeline
    I am about to install the following software packages:
    • cfitsio3360.tar.gz
    • cpl-7.0.tar.gz
    • esorex-3.12.3.tar.gz
    • gasgano-2.4.8.tar.gz
    • visir-kit-4.3.1-1.tar.gz
    with the following pipeline calibration file(s):
    • visir-calib-4.3.1.tar.gz
    The software installation will be organised in bin/, lib/ and include/ directories, while the calibration files will be installed in a calib/ directory. Where should I install the software packages ? [/home/userid] /home/dummy/pipelines
    Where should I install the pipeline calibration files ? [/home/dummy/pipelines] /home/dummy/calibrations
    In this case the software is installed under the directory /home/dummy/pipelines and the static calibration tables under /home/dummy/calibrations/visir The only exception to all this is the Gasgano tool, that will always be installed under the directory $HOME/gasgano Note that the installer will move an existing $HOME/gasgano directory to $HOME/gasgano.old before the new Gasgano version is installed.
    Important: the installation script would ensure that any existing Gasgano and EsoRex setup would be inherited into the newly installed configuration files (avoiding in this way any conflict with other installed instrument pipelines).
    Please follow the instructions shown at the end of a successful installation script: they will indicate how to set your PATH and CPLDIR variables in your machine in order to run the pipeline.
    Alternatively, it is possible to perform a manual installation (experienced users only): the README file located in the top installation directory contains more detailed information about a step-by-step installation.

Executing the pipeline 4.3.1

Using Reflex

  • To run the Reflex workflow, please refer to the VISIR workflow tutorial available in this table under column Documentation.

Using Gasgano

  1. To run the Gasgano GUI, just add $HOME/gasgano/bin to your PATH environment variable:     export PATH="$HOME/gasgano/bin:"$PATH You should also define an environment variable CPLDIR to point to the same path specified for the installation. Possible files to update are:     $HOME/.bashrc     $HOME/.profile Finally, enter the command     gasgano
  2. If the Gasgano main panel appears, then you have successfully installed Gasgano. Make sure, however, that the expected Gasgano 2.4.8 release is running: the version number is listed at the top of the main Gasgano panel.
  3. Refer to the Gasgano User's Manual to learn how to browse data and run pipeline recipes. Refer to the Pipeline User Manual(s) for detailed information about the recipes usage.

Using EsoRex

  1. The EsoRex executable should be available in the bin directory under your installation path (see above). The PATH environment variable should be upgraded accordingly (similarly as done for Gasgano).
  2. Refer to the EsoRex web page for details about the related features and options.

Documentation

The Reflex Users' Manual is availale for download

The GASGANO Users' Manual (~1.2 Mb, 66 pages) is available for download in the Gasgano web page.

On the EsoRex web page some online documentation about EsoRex can be found.

The CPL manuals are available on the CPL web pages.

In case of problems when opening the documents directly from your web browser, the files may be first saved on disk, and then opened with Acrobat Reader.


Bug Reports

If you experience an unexpected behavior of any component of the VISIR pipeline recipes package, please, first verify that you are using one of the above mentioned supported platforms and refer to the list of known problems and limitations in the pipeline manual of the current VISIR pipeline release.

For any other issues or requests, please, send a report to the ESO User Support Department, describing:        

  • the VISIR pipeline version (currently 4.3.1), and the version of other components (e.g., Gasgano, EsoRex, ...) you are using
  • the version of your OS and C compiler.
  • the exact sequence of actions that were performed before the problem occurred
  • what were precisely the symptoms and the possible error message(s)
  • whether the problem is repeatable