eclipse Frequently Asked Questions

FAQ Revised: Friday 23 September 2005 12:12:00

Table of Contents

1. General
2. Commands
3. Software
4. Tips

1. General

1.1. Who is supporting eclipse?
Yves Jung is currently maintaining the main library and instrument packages. Roeland Rengelink is taking care of the eclipse-python interfaces.

1.2. What can I do with eclipse?
eclipse was initially written having Adonis in mind, but was later enhanced (almost completely rewritten) to support more instruments. It is basically a set of tools to work directly on images in FITS format.

1.3. Why should I use eclipse?
eclipse is not IRAF. It is a data reduction engine written in C for portability and speed reasons. From this C library, the ISAAC pipeline team has built a number of Unix commands that are specialized for ISAAC imaging and spectroscopy data processing. We have put into it the know-how of the instrument scientist, and many other contributing people. We do believe that what is actually done by these routines is representative of what most people will want to do with the data.

You do not "learn" eclipse. The commands are more or less dedicated, one per template. What you have to learn is more or less similar to learning new IRAF commands (what to put in 'epar', how to launch it, learn what is happening in the algorithm, etc.). You do not have to learn a new language, because what it takes is launching a Unix command, not more.

1.4. Is it safe to base my software developments on eclipse?
It depends on which part of eclipse you intend to use. The C library offers a number of services for astronomical image processing, but we do not guarantee the stability of the C API. In other words, some functions might disappear or be completely re-engineered from one version to another without warning, function prototypes are constantly modified, data structures are evolving, etc.

If you want to base a C development on this library, you are not advised to do so. Say you want to go ahead and develop C or C++ software based on eclipse. You will download the latest version today and start working on it. Next version comes, and you realize that an object you have been using all along has changed types and is now offered in an incompatible way. You will have to either re-write your code completely to make use of the new structure (tedious and unsafe), or you have to un-hook your development from the eclipse mainstream, forking from the main project. It might be good to be independent, but you will also miss all bug corrections and new enhancements to the library.

On the other hand, we have developped interfaces to scripting languages (Python and Lua) for which we guarantee a certain level of stability. Functions and objects might change in the underlying C library, but we try to keep the upper layer as stable as possible. The Python and Lua cubes offer very simple operators, and they are sufficiently high-level not to be concerned with the details of the underlying C library. This efficiently shields script writers from changes in the low-level library.

So: if you want to base a C development on libeclipse.a and eclipse.h, it is highly not recommended unless you really want to fork from the mainstream development, or are sufficiently confident that you will be able to support the changes arising in the later versions. Using scripting languages is the way to go for most developments. There is of course no absolute rule, you are the best placed to consider what should be done in your software project.

1.5. How do I quote eclipse in my papers?
References to eclipse should be made by quoting:
N. Devillard, "The eclipse software", The messenger No 87 - March 1997

2. Commands

2.1. What is a pipeline recipe exactly?
This question has never been raised directly, but many misunderstandings could have been cleared earlier if it had been raised in time.

ESO is trying to build and validate a number of automatic data reduction tools to provide calibrated and reduced data in the shortest time, without any intervention from the user. The fundamental principle behind pipelines is precisely that they run without anybody to specify which calibration data to use, which recipe to use, or tell the software what to do in case of failure of one component or another.

A pipeline recipe is a data reduction program that reduces a given set of frames, possibly using provided calibration frames. This program takes in input a list of relevant frames and produces one or more products. Recipes are launched by the pipeline infrastructure.

Since recipes do not interact with a user, there are many choices that they cannot do like discarding a frame that would happen to have a satellite trail, and if they have to make choices, there can be no guarantee that it will always be correct. Other restrictions imposed by this lack of interactivity are simply a switch to another algorithm that might be more efficient for a given data set. No automatic software can be expected to do that in a reliable way. Pipeline products are usually a good indication of what will be obtained after proper (hand) data reduction, but they are rarely a final product.

If you reduce your data using eclipse recipes for the supported instruments, you will find handy that with little preparation work you can usually get a result in no time (assuming everything goes fine), without having to type anything more than the configuration files. But your task should not stop there: validate what can be validated against catalogs and previous measurements, re-run the procedure with slightly different parameters, apply the same reduction with another algorithm or software, etc. Keep in mind that an automatic software must be robust and will not outsmart you. Do not blindly trust what comes out.

2.2. What kind of FITS files are supported?
The latest version now supports data files with 2 or 3 dimensions (images and cubes), and list of FITS files declared in ASCII lists. The latest version of the FITS handling routines now offer support for FITS files with extensions and both binary and ASCII tables, but this does not show up yet in the associated Unix commands. Only some ISAAC recipes are offering access to FITS tables or producing them, and no command for the moment is offering any functionality on data found in extensions. That will be added as the need arises.

2.3. What are maximum data sizes accepted by eclipse?
This has been increased in the latest versions. Initially, it was set that no image bigger than 512x512 and no cube having more than 128 frames could be fed into any eclipse routine... But that was eons ago (1995), and since then detector sizes have grown a lot bigger, and instruments like GraF are producing cubes with thousands of (small) frames. In version 3.3, eclipse supports image sizes up to 40k pixels in both directions, and up to 10k frames per cube. Do not even try to input a cube of size 40k/40k/10k, though, that could mean more than the fatal limit of 4 gigabytes of data, limit intrisically set on 32-bit machines (2^32 is usually used to represent 4 gigabytes).

If you need to increase some of these parameters, the limits have been intentionally set in a deeply rooted include file, which means you have to modify a basic eclipse source file and recompile everything from scratch (having taken care of removing all binary executables, libraries and object files before recompiling). The file to edit is eclipse/include/cube_defs.h and the constants to modify are: 

#define MAX_COLUMN_NUMBER               (40000)
#define MAX_LINE_NUMBER                 (40000)
#define MAX_IMAGE_NUMBER                (10240)

Do not forget in any case that eclipse always makes the assumption that any single input image fits into memory (be it real or virtual). Nowhere in eclipse has been introduced a stripe-based processing of data which could allow simple operations to be executed on huge images. This is again done on purpose, because eclipse is more targeted towards executing image processing tasks on many reasonably-sized images, than applying dumb operations to enormous data sets. If you happen to process data files containing really large images, I'd recommend either to cut down the images to smaller ones, or to go to other data processing packages.

This being said, if there are requests for that kind of processing it is very possible to add up such functionalities in the library, but that effort simply has not been identified yet.

2.4. Where can I find a list of all commands?
Try 'man eclipse-commands'.

2.5. Where is the help message for each command?
Previous versions of eclipse used to deliver a help message when launched with no arguments. Another convention has been taken, namely that launching a command without arguments would start it with default arguments whenever possible. To get the help message, you should use the --help option. See the following example for jitter: 

% jitter



********** jitter
********** part of eclipse library. (c) ESO 1996-2001


purpose: isaac/sofi jitter imaging reduction


parsing configuration file...
error: cannot find ini file [jitter.ini]: aborting
error: in parsing jitter.ini: aborting jitter


% jitter --help



********** jitter
********** part of eclipse library. (c) ESO 1996-2001


purpose: isaac/sofi jitter imaging reduction



use : jitter [flags] [options]
flags are :


        -g or --generate : generate a .ini file
        -t or --time     : estimate used CPU time
        --offset         : get help about the offset file format
        -h or --help     : get this help


options are :


        -f  or --file 
        to specify which .ini file to work on (default: jitter.ini)


        -w or --gui
        to launch the GUI


following options are only valid with -g or --generate:
        -i or --in      provide input file name
        -o or --out     provide output file name
        -c or --calib   provide calibration file name


2.6. What are the options common to all commands?
Some options are supported by most eclipse commands. Only the stand-alone sources do not support them all. The options are: 

Short    Long          Action


-L       --license     Display license and exit
         --help        Display short help and exit
         --version     Display current version and exit

Notice that there were previously (eclipse 3.3 and before) options to set the verbose and debug mode, and an option to setup memory requirements. These are not settable through command-line options any more, but through environment variables. See the INSTALL file in the base eclipse directory in any distribution (3.4 and later).

2.7. Why is there no deconvolution routine?
Because eclipse is dedicated to data pre-processing and simple calibrations. People who want to build pipelines reducing data quickly without user interactions are interested in eclipse because it offers convenient functionalities to implement that kind of thing once the algorithms are frozen. Deconvolution is definitely something which badly needs user interactions. People doing deconvolution would not believe a blackbox doing the job without more than text messages, better turn to MIDAS or IDL for that kind of job, where an interactive session is really supported.

This being said, it is true that eclipse has a very fast processing engine which may be ideally suited to implement CPU-intensive algorithms as all deconvolution algorithms are. You are free to write your own scripts making use of the FFT and arithmetic routines of eclipse to implement quickly standard deconvolution algorithms. Support for classical deconvolution stuff is not yet planned into eclipse, though.

2.8. ccube does not recognize my FITS files
ccube recognizes FITS files just like any other eclipse routine. It is more likely that you have typed an ambiguous arithmetic expression. The syntax for ccube arithmetic expressions is token-based. A token is either an operator or an operand. Operators are usual arithmetic signs: + - / * ^ e l ('e' stands for exponentation and 'l' for logarithm). Operands are either numbers in standard floating point notation or FITS files. A token is identified as as FITS file by the following criteria:

  • It has a .fits or .FITS suffix.
  • It is prefixed by an arobas @.

For example, if you want to subtract two FITS images named object and skybg, they will not be recognized by ccube as FITS files unless you prefix their name in the expression with an arobas. 

--- Incorrect ---
% ccube "object skybg -" bgsub
error: unrecognized token: [object]
error: in arithmetic expression: exiting


--- Correct ---
% ccube "@object @skysub -" bgsub


2.9. ccube does not handle the standard arithmetic notation correctly
Well, let's face it: with file names such as 32 or image1-image2, it becomes really hard to tell what is an operation and what is an operand. Example: 

% ls
  image1
  image2
  image1-image2
  2
% ccube -s "image1-image2/2"

Simple case: there are files named 'image1', 'image2', 'image1-image2', and '2' in the current directory. What should be used to do what?? Also: how do you make the difference between the slash as the operator for the division, and the slash as a separator for path names?

To remove ambiguities, go to Polish Reverse Notation and separate all your arguments by blanks. The above example could then be written, depending on what was intended: 

Subtract file 'image2' from file 'image1' and divide by 2:
% ccube "@image1 @image2 - 2 /"


Subtract file 'image2' from file 'image1' and divide by file '2':
% ccube "@image1 @image2 - @2 /"


Divide the file 'image1-image2' by 2:
% ccube "@image1-image2 2 /"


Divide the file 'image1-image2' by file '2':
% ccube "@image1-image2 @2 /"


Subtract file '2' in directory 'image2' from file 'image1':
% ccube "@image1 @image2/2 -"

The case is quite common, actually with VLT archive file names such as: ONTT.1998-12-03T00:45:04.069.fits it becomes hard to distinguish the file name from a bunch of numerical arguments, unless we restrict FITS file names in input to a pre-defined list, but that would be endless and not so useful. Better go to Polish Reverse Notation.

3. Software

3.1. How can I tell which eclipse version I have?
Launch an eclipse command with a single option on command-line: --version (don't forget the double dash sign, for long options). With most eclipse commands, it prints out the version number of the eclipse library that was used to compile that command. Only stand-alone C sources do not support this option, all others do. Try for example: 

% ccube --version
eclipse version: 3.6-11


3.2. What is the numbering scheme for versions?
The revision number is made of a major number, a minor number, and a micro number as: major.minor.micro. The major number is usually increased when lots of new functionalities are added for the user, usually support has been added for a new instrument or a major instrumental mode. The minor number increases every time we think the release has reached a stable level and can be released on the Web. The micro level indicates more or less a patchlevel. It is incremented very often, every time a non-negligible amount of work has been brought to the current version.

3.3. eclipse does not compile on my station!
eclipse is supported on the following platforms:

  • Solaris 2.5 and 2.6
  • HPUX 10.20 and 11
  • Linux 2.X

The actual requirements to compile eclipse are that the system must be POSIX-compliant, have the mmap() system call (POSIX.1b compatibility), and... have an ANSI C compiler.

See the INSTALL file in the base eclipse directory for troubleshooting the installation.

3.4. Is it Ok to compile eclipse on a machine and run the binaries on another?
In general no, you should avoid doing so.

The problem with cross-compilation (i.e. compiling the binaries on one machine and running them on another) is that there are a number of parameters which are determined at compile-time. These parameters are then assumed constant by all binaries derived from the eclipse libraries. If they happen to change because the machine on which you are running eclipse has undergone drastic changes (switch from 32-bit to 64-bit, processor or OS upgrade, etc.), or because you are running the binaries on a secondary platform, you are running the risk of getting unexplained core dumps at best, file corruption without warning at worst.

It is possible to move some parameter checks from compile-time to run-time, but that would bring serious performance issues and is not a good solution. A compiler usually optimizes the compiled code for the platform it is running on. If you really want to do cross-compilation, you should use dedicated cross-compilers or specific switches which are meant to do that.

The main recommendation would be: if you want to run eclipse on a given machine, compile it locally. The source code is distributed for that purpose.

3.5. I get some warnings during compilation
Please just ignore any kind of warning during compilation. They are mostly due to differences between compiler tolerances about what each of them considers as ANSI C. They are harmless. Real bugs are usually detected by more sophisticated tools!

If you compile eclipse using gcc -Wall you are likely to get a family of warnings for which we can do nothing about (functions defined and never used, etc.). No need to report these warnings, we get them too and try to minimize the number of such messages.

3.6. The *.fits issue
The same problem seems to be reported over and over again about eclipse commands that accept jokers on the command-line, like: 

dfits *.fits
is_ghost *.fits
ado_refits *.fits
stcube *.fits

Or even worse: 

dfits */*.fits
is_ghost */*.fits

The problem is not really in the eclipse command, but in the shell itself. Whenever you type '*.fits' on the command-line, the shell expands it to a list of files and feeds it into the command. Example:

You have a.fits, b.fits and c.fits in the current directory. Typing: 

is_ghost *.fits

will actually be sent to the command as if you had typed: 

is_ghost a.fits b.fits c.fits

Problem is: the maximum character length for a command-line is limited to 512 characters (or something like that, but a fixed length) on many Unix systems, which means that if the list of FITS files is longer than that, the list will be truncated. To be convinced, try out the following: replace the eclipse command name by 'echo' and have a look at what you get: 

echo *.fits
echo */*.fits

The result is that the list does not contain all the file names. This usually makes eclipse command barf with a stupid error message.

A solution is to provide the file names one by one by making use of the 'find' command (present on all Unix platforms). Example: 

find . -name \*.fits -exec is_ghost {} \;

Use of the backslashes is important, it prevents the shell from expanding *.fits to the actual list.

You can also use the shell 'foreach' command under csh/tcsh: 

foreach i (*.fits)
foreach? is_ghost $i
foreach? end

There are similar mechanisms under bash, sh, ksh, and other Unix shells. Have a look at your Unix shell documentation to learn how to loop on a large number of file names.

4. Tips

4.1. grepfits
If you are through typing:
% dfits *.fits | fitsort keyword1 keyword2 ...  keywordn


You can set the following alias in csh or tcsh:


alias grepfits 'dfits *.fits | fitsort \!*'


Type the above line exactly as it appears here. Now you can do:


% grepfits keyword1 keyword2 ... keywordn


4.2. Sorting out files
It is possible to sort out the files in the output of fitsort to make them appear by date. 

To make them appear by Unix file creation date (most recent first):
% dfits `ls -1t *.fits` | fitsort keyword1 ... keywordn


To make them appear by FITS date (assuming in is found in the DATE keyword):
% difts *.fits | fitsort DATE keyword1 ... keywordn | sort -k 2


4.3. Using eclipse commands as Unix filters
It is possible to redirect the output of most eclipse commands to the stdout stream, making them efficient Unix filters. On most Unix systems, there is a device called /dev/stdout that is bound to your local terminal's stdout. It is enough to declare to an eclipse command that the name of the output file is /dev/stdout to redirect all data output to this stream.

Example:

% ccube "a.fits 2 *" /dev/stdout > b.fits

This trick can be used to process data on one machine and output the results to another one (e.g. on a Beowulf cluster). If you have the correct settings for rsh, you can try:

Assuming machine1 and machine2 are two machines on which you have accounts, and they trust each other (your .rhosts file or similar must be properly configured):

machine1% ccube "a.fits 2 *" /dev/stdout | rsh machine2 "cat > b.fits"

This command loads the file a.fits, multiplies all its pixels by 2, then sends the results to its stdout, they are caught by the pipe and redirected to an rsh command. This remote command running on machine2 uses cat to catch all data coming from its stdin and re-direct it to a file called b.fits on machine2.

Unfortunately, it looks like the /dev/stdout device does not exist on all Unixes. If that is the case for you, you can still declare the ouput file name to be STDOUT. eclipse will recognize this name and dump all outputs to the stdout steam instead of saving to a file.

Overview

Software

Documentation