* Frequently Asked Questions about eclipse

© European Southern Observatory


FAQ Revised: Tue Oct 24 10:27:42 2000


Table of Contents

1. General
2. Software
3. Tips

1. General

1.1. Who is supporting eclipse?
Currently, the team is made of Nicolas Devillard, Yves Jung and Thomas Rogon. We are all working at ESO in the ISAAC/CONICA pipeline development team. Our first customer is the VLT pipeline for these two instruments, which does not prevent us from trying to make versatile commands that can be used with other data. But the main concern is of course focused on data reduction for ISAAC and CONICA in the context of the DataFlow environment.

1.2. What can I do with eclipse?
eclipse was initially written having Adonis in mind, but was later enhanced to support more instruments. It is basically a set of tools to work directly on images in FITS format. A list of Unix commands provided by eclipse is available from the eclipse-commands manual page:

eclipse commands manual page

1.3. Why should I use eclipse?
eclipse is not IRAF. It is a data reduction engine written in C for portability and speed reasons. From this C library, the ISAAC pipeline team has built a number of Unix commands that are specialized for ISAAC imaging and spectroscopy data processing. We have put into it the know-how of the instrument scientist, and many other contributing people. We do believe that what is actually done by these routines is representative of what most people will want to do with the data.

You do not "learn" eclipse. The commands are more or less dedicated, one per template. What you have to learn is more or less similar to learning new IRAF commands (what to put in 'epar', how to launch it, learn what is happening in the algorithm, etc.). You do not have to learn a new language, because what it takes is launching a Unix command, not more.

Now, if you think you have better routines, IRAF scripts or whatever, you will probably be more efficient with tools you have written, you know where they fail and where they work. If you have no clue about ISAAC data reduction or if what you want to do is already performed by an eclipse command, you will be faster and more efficient using it. If you want to do something non-trivial that is not supported in the eclipse commands, or if you need any kind of interactive algorithm, you have no choice but use interactive environments like IRAF. Anyway, past some point you will need something like IRAF to interact with your data to see what is in there. eclipse only covers the first stages of data reduction, until an astronomical knowledge of the data is required to go further. Then it is the astronomer's job and no software can do it in your place.

1.4. What is a pipeline recipe exactly?
This question has never been raised directly, but many misunderstandings could have been cleared earlier if it had been raised in time.

ESO is trying to build and validate a number of automatic data reduction tools to provide calibrated and reduced data in the shortest time, without any intervention from the user. The fundamental principle behind pipelines is precisely that they run without anybody to specify which calibration data to use, which recipe to use, or tell the software what to do in case of failure of one component or another.

A pipeline recipe is a data reduction program that reduces a set of given frames, possibly using provided calibration frames. This program takes in input a list of relevant frames and produces one or more products. Recipes are launched by the pipeline infrastructure.

Since recipes do not interact with a user, there are many choices that they cannot do like discarding a frame that would happen to have a satellite trail, and if they have to make choices, there can be no guarantee that it will never be wrong. Other restrictions imposed by this lack of interactivity are simply a switch to another algorithm that might be more efficient for a given data set. No automatic software can be expected to do that in a reliable way. Pipeline products are usually a good indication of what will be obtained after proper (hand) data reduction, but they are rarely a final product.

If you reduce your data using eclipse recipes for the supported instruments, you will find handy that with little preparation work you can usually get a result in no time (assuming everything goes fine), without having to type anything past the configuration files. But your task should not stop there: validate what can be validated against catalogs and previous measurements, re-run the procedure with slightly different parameters, apply the same reduction with another algorithm or software, etc. Keep in mind that an automatic software must be robust but will not outsmart you. Do not blindly trust what comes out.

1.5. What kind of FITS files are supported?
The latest version now supports data files with 2 or 3 dimensions (images and cubes), and list of FITS files declared in ASCII lists. The latest version of the FITS handling routines now offer support for FITS files with extensions and both binary and ASCII tables, but this does not show up yet in the associated Unix commands. Only some ISAAC recipes are offering access to FITS tables or producing them, and no command for the moment is offering any functionality on data found in extensions. That will be added as the need arises.

1.6. What are maximum data sizes accepted by eclipse?
This has been increased in the latest versions. Initially, it was set that no image bigger than 512x512 and no cube having more than 128 frames could be fed into any eclipse routine... But that was eons ago (1995), and since then detector sizes have grown a lot bigger, and instruments like GraF are producing cubes with thousands of (small) frames. In version 3.3, eclipse supports image sizes up to 40k pixels in both directions, and up to 10k frames per cube. Do not even try to input a cube of size 40k/40k/10k, though, that could mean more than the fatal limit of 4 gigabytes of data, limit intrisically set on 32-bit machines (2^32 cannot represent more than 4 gigabytes).

If you need to increase some of these parameters, the limits have been intentionally set in a deeply rooted include file, which means you have to modify a basic eclipse source file and recompile everything from scratch (having taken care of removing all binary executables, libraries and object files before recompiling). The file to edit is eclipse/include/cube_defs.h and the constants to modify are:

#define MAX_COLUMN_NUMBER               (40000)
#define MAX_LINE_NUMBER                 (40000)
#define MAX_IMAGE_NUMBER                (10240)

Do not forget in any case that eclipse always makes the assumption that any single input image fits into memory (be it real or virtual). Nowhere in eclipse has been introduced a stripe-based processing of data which could allow simple operations to be executed on huge images. This is again done on purpose, because eclipse is more targeted towards executing image processing tasks on many reasonably-sized images, than applying dumb operations to enormous data sets. If you happen to process data files containing really large images, I'd recommend either to cut down the images to smaller ones, or to go to other data processing packages.

This being said, if there are requests for that kind of processing it is very possible to add up such functionalities in the library, but that effort simply has not been identified yet.

1.7. What is specific to Adonis?
The following commands are targetted for Adonis only:

Notice that some commands have default values for the ESO 3.6m telescope in la Silla and some contain Adonis specific features for default behaviour (strehl for example). David Le Mignant has put up a series of scripts to reduce Adonis data with eclipse, have a look at the following page:

http://sc6.sc.eso.org/~dlemigna/eclipse/

The wdat command is a tool written to read Adonis DAT tapes, it can only be useful to Adonis users. wdat has been written and is supported by Francois Lacombe at the Paris-Meudon observatory. Contact him in case of trouble with the command. You can also have a look at the wdat home page which you can find from the main eclipse home page.

Since version 3.1.1, all Adonis specific commands can be found in the base directory eclipse/src/adonis.

Since version 3.8, all Fabry-Perot commands have disappeared from the eclipse distribution. A quick survey has shown that nobody was actually using them at all. They are kept in the repository for future reference but not distributed any more.

1.8. What is specific to ISAAC/SOFI?
Anything under the base directory src/isaac has been written with ISAAC/SOFI in mind. See the directories below eclipse/src/isaac.

1.9. Where can I find a list of all commands?
Try 'man eclipse-commands'. You can also browse the list from the eclipse commands manual page on-line.

1.10. Where is the help message for each command?
Previous versions of eclipse used to deliver a help message when launched with no arguments. Another convention has been taken, namely that launching a command without arguments would start it with default arguments whenever possible. To get the help message, you should use the --help option. See the following example for jitter:

% jitter
           jitter
           part of eclipse library. (c) ESO 1996-2000

purpose: isaac/sofi jitter imaging reduction

parsing configuration file... error: cannot find ini file [jitter.ini]: aborting error: in parsing jitter.ini: aborting jitter

% jitter --help

jitter part of eclipse library. (c) ESO 1996-2000

purpose: isaac/sofi jitter imaging reduction

use : jitter [flags] [options] flags are :

-d or --display : display information stored in a .ini file -g or --generate : generate a .ini file -v or --verbose : enable verbose mode -D or --debug : enable debug mode -t or --time : estimate used CPU time --test : generate test data (see doc)

options are:

-f or --file to specify which .ini file to work on (default: jitter.ini) -h or --help to get this help



1.11. What are the options common to all commands?
Some options are supported by most eclipse commands. Only the stand-alone sources do not support them all. The options are:

Short    Long          Action

-L --license Display license and exit --help Display short help and exit --version Display current version and exit

Notice that there were previously (eclipse 3.3 and before) options to set the verbose and debug mode, and an option to setup memory requirements. These are not settable through command-line options any more, but through environment variables. See the INSTALL file in the base eclipse directory in any distribution (3.4 and later).

1.12. Why is there no deconvolution routine?
Because eclipse is dedicated to data pre-processing and simple calibrations. People who want to build pipelines reducing data quickly without user interactions are interested in eclipse because it offers convenient functionalities to implement that kind of thing once the algorithms are frozen. Deconvolution is definitely something which badly needs user interactions. People doing deconvolution would not believe a blackbox doing the job without more than text messages, better turn to MIDAS or IDL for that kind of job, where an interactive session is really supported.

This being said, it is true that eclipse has a very fast processing engine which may be ideally suited to implement CPU-intensive algorithms as all deconvolution algorithms are. You are free to write your own scripts making use of the FFT and arithmetic routines of eclipse to implement quickly standard deconvolution algorithms. Support for classical deconvolution stuff is not yet planned into eclipse, though.

1.13. ccube does not recognize my FITS files
ccube recognizes FITS files, just like any other eclipse routine. It is more likely that you have mistyped an arithmetic expression. The syntax for ccube arithmetic expressions is token-based. A token is either an operator or an operand. Operators are usual arithmetic signs: + - / * ^ e l ('e' stands for exponentation and 'l' for logarithm). Operands are either numbers in standard floating point notation or FITS files. A token is identified as as FITS file by the following criteria:

For example, if you want to subtract two FITS images named object and skybg, they will not be recognized by ccube as FITS files unless you prefix their name in the expression with an arobas.

--- Incorrect ---
% ccube "object skybg -" bgsub
error: unrecognized token: [object]
error: in arithmetic expression: exiting

--- Correct --- % ccube "@object @skysub -" bgsub

All this (and more) is explained in details in the ccube manual page.

1.14. ccube does not handle the standard arithmetic notation correctly
Well, let's face it: with file names such as 32 or image1-image2, it becomes really hard to tell what is an operation and what is an operand. Example:

% ls
  image1
  image2
  image1-image2
  2
% ccube -s "image1-image2/2"

Simple case: there are files named 'image1', 'image2', 'image1-image2', and '2' in the current directory. What should be used to do what?? Also: how do you make the difference between the slash as the operator for the division, and the slash as a separator for path names?

To remove ambiguities, go to Polish Reverse Notation and separate all your arguments by blanks. The above example could then be written, depending on what was intended:

Subtract file 'image2' from file 'image1' and divide by 2:
% ccube "@image1 @image2 - 2 /"

Subtract file 'image2' from file 'image1' and divide by file '2': % ccube "@image1 @image2 - @2 /"

Divide the file 'image1-image2' by 2: % ccube "@image1-image2 2 /"

Divide the file 'image1-image2' by file '2': % ccube "@image1-image2 @2 /"

Subtract file '2' in directory 'image2' from file 'image1': % ccube "@image1 @image2/2 -"

The case is quite common, actually with VLT archive file names such as: ONTT.1998-12-03T00:45:04.069.fits it becomes hard to distinguish the file name from a bunch of numerical arguments, unless we restrict FITS file names in input to a pre-defined list, but that would be endless and not so useful. Better go to Polish Reverse Notation.

1.15. Where is cleandead gone?
cleandead has been included in the deadpix command which can now be invoked in one of the two following modes: clean or detect. Basically, calling deadpix --clean is equivalent to the previous call to cleandead.

You can have a look at deadpix manual page.

1.16. How do I quote eclipse in my papers?
References to eclipse should be made by quoting:
N. Devillard, "The eclipse software", The messenger No 87 - March 1997


2. Software

2.1. How can I tell which eclipse version I have?
Launch an eclipse command with a single option on command-line: --version (don't forget the double dash sign, for long options). With most eclipse commands, it prints out the version number of the eclipse library that was used to compile that command. Only stand-alone C sources do not support this option, all others do. Try for example:

% ccube --version
eclipse version: 3.6-11


2.2. Where are versions 3.3 and 3.5?
Version 3.3 was never distributed on the net, but only internally in ESO during the ISAAC pipeline development. To keep consistency within ESO, it has been decided to upgrade the version before releasing on the Web.

Same applies to version 3.5.

2.3. What is the numbering scheme for versions?
The revision number is made of a major number, a minor number, and a micro number as: major.minor.micro. The major number is usually increased when lots of new functionalities are added for the user, usually support has been added for a new instrument or a major instrumental mode. The minor number increases every time we think the release has reached a stable level and can be released on the Web. The micro level indicates more or less a patchlevel. It is incremented very often, every time a non-negligible amount of work has been brought to the current version.

2.4. eclipse does not compile on my station!
eclipse is supported on the following platforms:

The requirements to compile eclipse are that the system must be POSIX-compliant, have the mmap() system call (POSIX.1b compatibility), and... have an ANSI C compiler.

The eclipse Makefile knows how to handle the gcc and egcs compilers, so if you want to use one of those rather than the default set for your machine, you can do so by requesting a different OSTYPE:

% make OSTYPE=gcc
% make OSTYPE=egcs
Sometimes, you also need to define by hand the variable CCNAME to gcc to compile. In that case, you would type:

% setenv CCNAME gcc
% make OSTYPE=gcc

gcc and egcs bugs

Be aware that there have been enormous problems reported lately with gcc and egcs, on Linux mostly:

If you are using Linux with any of the problematic compiler versions, good luck!

If you got eclipse to compile on any other system, please drop me a mail explaining the problems you faced and the options on your local system to compile ANSI C.

2.5. I get some warnings during compilation
Please just ignore any kind of warning during compilation. They are mostly due to differences between compiler tolerances about what each of them considers as ANSI C. They are harmless. Real bugs are usually detected by more sophisticated tools!

For HPUX users: there is a warning about a MAXINT macro multiply defined. We cannot do anything about that, it is due to an internal HPUX inconsistency.

2.6. Why is eclipse running slower on my Intel PC with Linux than on an UltraSparc with Solaris?
Complain to the people who designed the FITS format! Unfortunately, binary FITS data are stored as big-endian values (no other choice), and Intel machines work in little-endian format. This means that for every FITS file you read into memory, you need to perform conversion on the fly, which needs of course CPU cycles. Also: with Intel machines you cannot map any FITS file into memory as it is, because the file format does not correspond to the internal machine format. In the jitter command, advantage is taken of this memory mapping to speed up the process, but this is no more possible on PCs. As a result, the same process runs in roughly the same CPU time on a Pentium II and an UltraSparc, but there is about 20 seconds overhead due to disk I/O on UltraSparc, and about 40 minutes on a Pentium!

2.7. The *.fits issue
The same problem seems to be reported over and over again about eclipse commands that accept jokers on the command-line, like:

dfits *.fits
is_ghost *.fits
ado_refits *.fits
stcube *.fits

Or even worse:

dfits */*.fits
is_ghost */*.fits

The problem is not really in the eclipse command, but in the shell itself. Whenever you type '*.fits' on the command-line, the shell expands it to a list of files and feeds it into the command. Example:

You have a.fits, b.fits and c.fits in the current directory. Typing:

is_ghost *.fits

will actually be sent to the command as if you had typed:

is_ghost a.fits b.fits c.fits

Problem is: the maximum character length for a command-line is limited to 512 characters (or something like that, but a fixed length) on many Unix systems, which means that if the list of FITS files is longer than that, the list will be truncated. To be convinced, try out the following: replace the eclipse command name by 'echo' and have a look at what you get:

echo *.fits
echo */*.fits

The result is that the list does not contain all the file names. This usually makes eclipse command barf with a stupid error message. I will try to remedy to that in future versions, giving an intelligible error message instead.

In the mean time, you can also provide the file names one by one by making use of the 'find' command (present on all Unix platforms). Example:

find . -name \*.fits -exec is_ghost {} \;

Use of the backslashes is important, it prevents the shell from expanding *.fits to the actual list.

You can also use the shell 'foreach' command under csh/tcsh:

foreach i (*.fits)
foreach? is_ghost $i
foreach? end

Or similarly:

foreach i (*/*.fits)
foreach? is_ghost $i
foreach? end

There are similar mechanisms under bash, sh, ksh, and other Unix shells.


3. Tips

3.1. grepfits
If you are through typing:
% dfits *.fits | fitsort keyword1 keyword2 ...  keywordn

You can set the following alias in csh or tcsh:

alias grepfits 'dfits *.fits | fitsort \!*'

Type the above line exactly as it appears here. Now you can do:

% grepfits keyword1 keyword2 ... keywordn



3.2. Sorting out files
It is possible to sort out the files in the output of fitsort to make them appear by date.

To make them appear by Unix file creation date (most recent first):
% dfits `ls -1t *.fits` | fitsort keyword1 ... keywordn

To make them appear by FITS date (assuming in is found in the DATE keyword): % difts *.fits | fitsort DATE keyword1 ... keywordn | sort -k 2



3.3. Using eclipse commands as Unix filters
It is possible to redirect the output of most eclipse commands to the stdout stream, making them efficient Unix filters. On most Unix systems, there is a device called /dev/stdout that is bound to your local terminal's stdout. It is enough to declare to an eclipse command that the name of the output file is /dev/stdout to redirect all data output to this stream.

Example:

% ccube "a.fits 2 *" /dev/stdout > b.fits

This trick can be used to process data on one machine and output the results to another one (e.g. on a Beowulf cluster). If you have the correct settings for rsh, you can try:

Assuming machine1 and machine2 are two machines on which you have accounts, and they trust each other (your .rhosts file or similar must be properly configured):

machine1% ccube "a.fits 2 *" /dev/stdout | rsh machine2 "cat > b.fits"

This command loads the file a.fits, multiplies all its pixels by 2, then sends the results to its stdout, they are caught by the pipe and redirected to an rsh command. This remote command running on machine2 uses cat to catch all data coming from its stdin and re-direct it to a file called b.fits on machine2.

Unfortunately, it looks like the /dev/stdout device does not exist on all Unixes. If that is the case for you, expect support for stdout output in the next eclipse release. The convention will be: if the name of an output file is STDOUT all output data are sent to stdout.


Back to eclipse home page

This page is maintained by nDevil@eso.org