* Frequently Asked Questions about jitter

© European Southern Observatory


FAQ Revised: Tue Oct 24 10:27:43 2000


Table of Contents

1. General
2. Input and configuration files
3. Dark, Flatfield, Bad pixel map
4. Sky estimation/subtraction
5. Shift and Add
6. Registration
7. Frame average
8. Software-related questions
9. Output FITS file format

1. General

1.1. What is jitter doing exactly?
Read the manual! jitter comes with a spartan manual page describing the basics, and a full algorithmic documentation.

Links:



1.2. What is a pipeline recipe exactly?
This question is answered in the general eclipse FAQ.
See also question below: "I heard some horrible things about jitter, what is true?".

1.3. May I use it for optical data?
You can of course use this software on optical data. Actually, number of people have been using it as a shift-and-add tool for images taken in the visible bands. The most sensible thing to do in this case would be to disable the sky estimation in the ini file, but otherwise algorithms are unchanged.

1.4. I heard some horrible things about jitter, what is true?
jitter is a very complex tool that is aimed at performing the data reduction steps with as little user interaction as possible. In an ideal future, it should be able to modify itself its internal parameters depending on the input data, to get the best possible results. But that is still a dream! Today you need to first read the documentation carefully to be sure you understand correctly what the input parameters are. Then you need to take the time to study your own case and define a correct set of parameters for your problem. Then you can run jitter and see if the results you get are consistent. The best data reduction ever is almost never achieved on the first attempt. jitter is not (yet!) a magical tool that does everything for you.

jitter is trying to be a good tool, not a good worker. There is no replacement so far for the attention an astronomer can have on a data set, and the scientific appreciation of the results. Consider this software as a black-box that automates many tasks for you but has to be tailored to your data sets to produce the most efficient results.


2. Input and configuration files

2.1. Are there examples of jitter.ini files?
Use the -g or --generate option. It will produce a default jitter.ini file in the current directory.

2.2. What are the default parameters valid for?
Section by section:



2.3. How much memory do you recommend to have?
It depends on the amount of data you have to process. A typical SOFI or ISAAC batch of jitter files is usually made of about 60 frames. That translates to roughly 256 megs in input, thus about 700 to 800 megs total memory consumption during the process. If you can afford a gigabyte of RAM for such data sets, be my guest. If you have less memory, just declare it in the memory configuration parameters and jitter will swap pages as necessary to make the best use of what you have.

For your information: jitter has been reported to reduce such SOFI batches on a Solaris station with only 32 megs of RAM (and tons of disk space), without any problem. The process is much slower, but it runs fine. I would say that 32 megs in RAM is the minimal amount to get a decent working jitter process. If you have a slow machine or little RAM, launch your processes at night with an at or cron command and get the results in the morning.

2.4. How do I create the list of input files?
The simplest way is to use 'ls'.

% ls -1 *.fits > framelist.ascii

Replace *.fits by the wildcards you need to describe the files you want to process, and framelist.ascii by any name you want, provided it is declared correctly in jitter.ini.

Notice that ls is used with the option -1 (minus one), which requests a list of file names on a single column output.

Beware!

If your ls command is aliased to some exotic combination of options (most people alias ls to ls -F), the results of the previous commands may append some funny characters to the file names. Simple example: if your files have executable permission, and you aliased ls -F, all file names will have asterisks (*) appended, which will prevent jitter from finding the files. Notice that you can de-alias a command by preceding it with a backslash, i.e.

% \ls -1 *.fits > myframelist

If you want to do jitter+offset reductions, you will need to declare in second column the file type for each input frame (see next question). Hint: use dfits and fitsort to create this column output. For ISAAC, the simplest is to do:

    % dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii


2.5. How do I declare OBJECT and SKY frames?
The input list format is the following: provide the input file names in first column, and optionally the file type in second column. Both columns are separated by blanks or tabs. The file type in second column identifies whether the frame is an OBJECT or a SKY. The definition is: if the word sky (case-insensitive) can be found in the character string identifying the file type, then the frame is a sky. Otherwise it is an object.

Examples

The following declares only objects:
(no second column -> no sky string -> no SKY plane)

--- begin frame1.fits frame2.fits frame3.fits frame4.fits --- end

The following declares frame2.fits and frame4.fits as skies: (their second column contains a sky string)

--- begin frame1.fits frame2.fits sky frame3.fits frame4.fits sky --- end

Same for the following:

--- begin frame1.fits OBJECT_FRAME frame2.fits SKY_FRAME frame3.fits OBJECT_FRAME frame4.fits SKY_FRAME --- end

Same for the following (notice that the second column is case-insensitive to identify 'sky')

--- begin frame1.fits NGC9999 frame2.fits NGC_SKY frame3.fits NGC9999 frame4.fits skyFrame --- end

With SOFI/ISAAC frames, you can get away with a simple:

    % dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii



3. Dark, Flatfield, Bad pixel map

3.1. What about flat-field division?
They did not work in previous versions but they do now (3.1.1 and later). Either upgrade to the latest eclipse release or use ccube to do plane arithmetic Example:

To subtract dark.fits and divide by ff.fits all planes named isaac*.fits
in the current directory (with tcsh), and output them to files named
isaac*.fits.sub:

% foreach i (isaac*.fits) % foreach? ccube "$i dark.fits - ff.fits /" $i.sub % foreach? end




4. Sky estimation/subtraction

4.1. Why all these dark crowns in output around extended objects or crowded regions?
During sky background estimation, the aim is to remove all objects from the sky background, to get something which is as close as possible to the real sky value. If this rejection does not work too well, there will be star signal left in the background. This background is then subtracted from all input frames, which means a signal which is a little bit negative in case some object signal was left in the background.

This will happen for example if you have crowded regions. The jitter movements are not sufficient (most will end up on object signal). For some pixels, most observed signal will be object signal, not sky. In that case it becomes impossible to filter reliably the sky out from this signal.

Warning!

It is a terrible idea to observe a very crowded field or an extended object with a simple jitter pattern. Use instead jitter+offset, it has been designed for that purpose. Of course, you are then using precious observation time to observe a blank sky, but it is the only reliable way to get rid of this strong signal in the infrared regime.

There have been many cases already of observers who did not want to loose any time observing such useless signals as the sky, and were totally unable to reduce their data. If you have not calibrated the sky background, you are left out with atmosphere models to estimate it, without any guarantee of result. You are advised NOT to use the 'jitter' command in such cases.

4.2. How good is the sky-subtraction?
The sky-subtraction algorithm has been carefully studied by Angela Iovino. Her results are presented in a report available from the eclipse Web site in Garching only (not at the Grenoble mirror).

Have a look at jitter and photometry report

4.3. How do I submit new sky-subtraction methods?
On average, we receive one new suggestion about sky filtering methods per week. On average, there is also one valid suggestion every thousand. If you really think you have the ultimate sky-filtering algorithm, it is recommended to go to your favourite programming environment and try out your method. If you can prove on a reasonable set of data that your method is better than the current one, pack everything into a decently written report and send it to us.

An algorithm does not have to be believed but demonstrated. Just as you would not publish a scientific paper where you only quote your opinions, you want to prove that your method is working in a given domain with some restrictions and specified parameters. If you just want to make suggestions, keep them for yourself until they are properly demonstrated. The next step is then to try to convince us of your method.

There are so many suggestions about this topic that we cannot spend our whole time looking at them into details. Most of them have turned out in the end to be useless. Best is probably to make a complete and convincing study of what you suggest before you contact us.


5. Shift and Add

5.1. How do I create an offset list?
Everything you need to know about offset lists can be found by launching jitter with the --offset option.

Here is one way to create simple offset lists. The easiest is to create it from header information. Using dfits, fitsort and awk you could do something like:

% dfits *.fits | fitsort -d SEQ.CUMOFFSETX SEQ.CUMOFFSETY

That prints out the values of SEQ.CUMOFFSETX and SEQ.CUMOFFSETY for all FITS files in the current directory. Now if you want to create an offset list from this, you also need to add the plane number, which is usually just sequential from 1 to NPlanes. Using awk, you can easily add that by doing:

% [...] | awk '{print NR-1,$2,$3}' > offsets.in

[...] stands for the previous command, combination of dfits and fitsort.

5.2. jitter fails to find any object for cross-correlation. What should I do?
Until version 3.1 patchlevel 3 (eclipse-3.1.3), the peak finder algorithm (in charge of finding the right place for cross-correlation) used a threshold method based on mean and sigma estimates. Now, if your input signal is really noisy, mean and sigma cannot be estimated by just summing up pixel values in the image. Also: if you have lots of stars in the input image, it might be impossible to rely on such estimations. One way to robustify the detection has been to change the algorithm to look for the median and average absolute deviation to the median. This was found to estimate noise and signal levels somewhat better for well-behaving but noisy or crowded signals.

A second very simple way to enhance detectability was to use the difference between the first two raw frames as a reference image for cross-correlation, instead of using the first sky-subtracted frame. This modification produced spectacular results in detection efficiency.

The peak finder algorithm can now be described as:

    Given an input image, find out:
	m = median value of the input image
	d = average absolute deviation to the median
	Threshold the image at m+K.d, K being a user-provided
	parameter (default is 2.0). Apply a morphological binary dilation,
	return a list of barycenters of all connected regions.

Some new strategies have been included to do all what's possible to find reliable objects for cross-correlation. The strategies are given hereafter. For each step, if no object was found the algorithm proceeds to next step and so on until suitable objects have been found.

People interested in strategies should have a look at the implementation in C, it is fairly readable. Have a look in cubes/correlation.c, the function to look at is called get_interest_center().

If, despite the algorithm modifications, jitter still cannot find any object in the first frame, try the following tricks.

Any other ideas on this subject are welcome!

Last resort: if 'jitter' is still unable to find anything worth cross-correlating in your images, of if the objects it picks are not suitable for alignment (e.g. moving objects), you can provide your own list of objects in an ASCII file. See the corresponding sections in the jitter.ini file.

5.3. What if I am missing the first frame?
The first frame in the batch is considered the best centered on the object, at offset (0,0). If the offset provided for the first frame is not (0,0) but say (dx,dy), 'jitter' will subtract (dx,dy) from all consecutive pairs of offsets, to make them relative to the first frame. You will get a warning notifying that offsets have been made relative to the first frame.

What that means is: if you lost the first frame in the batch for any reason, you can still take your batch from image 2 to image N and feed it into 'jitter' directly. By subtracting the offsets in the first image, you will implicitly use frame number 2 as the reference frame for alignment.

Now it might be the case that frame number 2 is not the best centered frame, and you want to use, say frame number 11. Easy: just declare frame 11 as the first frame in your framelist, it will be used as reference. If its offsets are non-zero, they will again be subtracted from all consecutive offsets anyway.

Which frame should you use in case you are missing the first one? If you used an AutoJitter template, the best centered frames are those falling on a multiple of the Poisson homogeneity (see the HIERARCH ESO SEQ POISSON keyword). So if HIERARCH ESO SEQ POISSON is 10, and you are missing the first frame in the batch, you can use frame number 11, 21, 31, etc.

5.4. What is the meaning of the returned value in parentheses, after the offset values?
The typical output from the offset matching routine is something like:

from plane 1 to plane 18:       [-4.24  -54.42] (25.42)
from plane 1 to plane 19:       [-57.44 -28.21] (33.38)
from plane 1 to plane 20:       [ 1.96  45.27]  (1396.27)
from plane 1 to plane 21:       [-18.35 -27.00] (35.13)
from plane 1 to plane 22:       [-37.79 -44.23] (1396.86)
from plane 1 to plane 23:       [37.00  41.88]  (25.61)

The 'jitter' command determines offsets between frames by using a cross-correlation criterion. This measurement is searching for an offset between the reference and the candidate frame, that minimizes the sum of squared differences between the frames. The value printed out in parentheses along with the offset measurements is the minimal difference value found in cross-correlation. The smaller the value, the better the match between frames. The theoretical limit for this value is zero (perfect match) in the case of a frame that is shifted by an integer translation vector. There is no absolute reference to measure the "goodness of match" with this value, since it will depend on:

To have a visual indication of what this value is, you can have a look at the jitter algorithmic manual: http://www.eso.org/projects/dfs/papers/jitter99/node11.html

The figure shows what a cross-correlation matrix looks like. The value returned in parentheses by the 'jitter' command is the lowest value of this matrix (the minimum of difference).

As a rule of thumb, for standard SOFI/ISAAC images, your can roughly say the following:

If the distance:

That tends to work if the objects chosen for cross-correlation are normal stars. That will not work if for example the signal which is cross-correlated is a bright galaxy with a high background signal. In that case, these empiric thresholds should not apply.

A better way to determine the "goodness of match" is to look at the returned offsets. If the search domain was set to say +/-50 pixels in X and Y, and the returned offset indicates a value of 50 or -50 (exactly) in either direction, it means that the minimal difference was found on the border of the search domain. Usually, this is an indication of failure. The chosen search domain should always be a little bigger than the expected offset throw.

Starting from eclipse 3.6, a test is done on the offsets returned by the cross-correlation procedure. If the returned offsets are returned equal to the size of the search domain, the frame is declared bad for correlation and removed from the stack of frames.


6. Registration

6.1. What is an interpolation kernel?
People coming from a signal processing background usually refer to interpolation as a kind of filtering in Fourier space. The filtering functions which are used in this space are localized functions named kernels, because they all more or less look like a "door" function.

There is an article about image resampling, containing a fair description of what an interpolation kernel is. You should have a look at the ESO Messenger (issue 100 - July 2000). The article is called: Astronomical image resampling. You can download the July 2000 issue of the Messenger from the ESO Web site, at:

http://www.eso.org/gen-fac/pubs/messenger/


7. Frame average

7.1. What are the 3 average options?
Linear is a simple, straight linear average. Each pixel along time is summed, then divided by the number of planes. This method is bad because all bad pixels and cosmics will end up in the final image.

Median is a median 3d average. Take a line of pixels along time, extract the median: it is the output pixel value for this position. This method is also bad, because it does not sum up faint signals.

Filtered is somewhere in between: take a line of pixels along time, sort it out by increasing pixel value, remove the highest and lowest ones (outliers) and average the remaning pixels. This is the output pixel value for this detector position. This is probably the best method, because it rejects outliers but still sums faint signals.


8. Software-related questions

8.1. jitter uses more RAM than I have!
If anyone knows a way to extend physical RAM chips in a workstation by software means, I want to learn about it! :-)

Since version 3.0, eclipse uses a new (hopefully portable) memory allocation scheme. Without going into too many details, the library opens its own swap files in a temporary directory and tells the Operating System to use it as virtual memory.

Some process monitoring tools such as top will report these temporary files (of litterally unlimited size) as memory use, thus providing a false information. Do not be alarmed by the quantities announced on top on systems such as Solaris and Linux, they are just bugged. jitter has been reported to show a memory use of more than 2 gigabytes (on a 32-bit system, that's not bad) with no more than half a gig of memory space (RAM chips + swap space)... Do not always believe blindly your memory monitoring tools!

8.2. Is there a way to save intermediate data in case of machine crashes?
Not really. The ideal would be an interruptible process which would be able to stop completely and restart later on from configuration files. This is nowhere near to be implemented.

On the other hand, if you run the complete process of sky estimation and shift-and-add, it is possible to save the sky-subtracted planes after sky estimation to a single FITS cube on the disk. To re-enter the jitter process you will have to modify the list of input files to a single file (the intermediate sky-subtracted cube) and disable sky estimation to jump directly to shift-and-add.

8.3. jitter quits unexpectedly when I try to reduce more than 50 frames
Increase the number of descriptors (= files) you can open per process. On most systems, this is done with a built-in shell command named 'limit' (on csh, tcsh) or 'ulimit' (on sh, ksh, bash). On Solaris with tcsh it reads:

% limit
cputime         unlimited
filesize        unlimited
datasize        2097148 kbytes
stacksize       8192 kbytes
coredumpsize    0 kbytes
vmemoryuse      unlimited
descriptors     64 

Numbers will actually vary with the platform, these are the default values for mine. The parameter you want to change is descriptors. Move it to unlimited by typing:

% limit descriptors unlimited
% limit
cputime         unlimited
filesize        unlimited
datasize        2097148 kbytes
stacksize       8192 kbytes
coredumpsize    0 kbytes
vmemoryuse      unlimited
descriptors     1024 

While you're at it, check out that the values for cputime, filesize, datasize and vmemoryuse are unlimited (or as high as they can), and that coredumpsize is zero, unless you want to flood your disk with core files at the first crash.

8.4. What are the byproducts of this software?
If in [SkyEngine] you activated the OutputDiff flag, you will create a data cube in the current directory, containing all sky-subtracted input planes.

If sky background computation is activated, section [SkyBackground], variable Computation, you will get an ASCII file in output containing what has been found about the sky background, for each input frame. If you could provide DIT, pixel scale, pixel gain and zero point, the outputs will also be shown in magnitudes per square arcsec.

If in the section [Detpeak] you ticked the option OutputObjects = yes, a file containing all objects used from cross-correlation will be produced in the current directory.

If in the [Offsets] section you activated the flag [OffsetFileOutput], the offsets used for shift-and-add will be written to an ASCII file.

Last, if you ticked in the section [PostProcessing] the ProduceStatusReport variable, a complete report status will be generated in the current directory, containing all of the above information (except the sky-subtracted input planes).


9. Output FITS file format

9.1. What is the meaning of the EXPTIME keyword?
According to the dictionary definition for ESO keywords:

EXPTIME provides the total integration time in seconds; it may have decimals. When the exposure is made of several periods, EXPTIME time is the sum of the exposure periods, and not simply the difference between end and start of exposure. Subintegrations, i.e. multiple exposures before a readout of the detector are described by the DIT and NDIT.

The reference is The Data Interface Control Board document.

This being said, the EXPTIME written in the output header by 'jitter' is the EXPTIME found in the first input header (NOT the sum of all EXPTIME values from all input files). This has been done upon request from the instrument scientist.

9.2. What are these HIERARCH ESO PRO keywords?
They are added to identify the frame as a pipeline product, in the ESO DataFlow environment. The only keywords you might be interested in are:

PRO CATG         product category, identifies the frame as COADDED_IMG
PRO DATE         date the jitter process was executed
PRO DATANCOM     how many frames were combined to produce this one



Back to eclipse home page

This page is maintained by nDevil@eso.org