© European Southern Observatory
FAQ Revised: Tue Oct 24 10:27:43 2000
Links:
jitter is trying to be a good tool, not a good worker. There is
no replacement so far for the attention an astronomer can have on a data
set, and the scientific appreciation of the results. Consider this
software as a black-box that automates many tasks for you but has to be
tailored to your data sets to produce the most efficient results.
Memory and swap defaults are set for a powerful machine that has lots of available disk space. You most certainly want to modify these before starting any jitter process! Notice that whatever you put in the jitter.ini file as values for MaximumMemory and MaximumSwap, has precedence over the values given through environment variables. Beware that the default generated jitter.ini declares a maximum memory allocation capability of 512 megs, which most likely will crash on stations with less memory, especially on Linux stations which do not handle memory overflows correctly.
Default parameters are set for a slowly varying sky, i.e. the sky can be considered constant in time over a range of 2*RejectHalfWidth+1 frames. If you are observing in J or H with a total exposure time of 1 second per frame, it is reasonable to set the half-width of the running filter to a high value. If you are observing in K or Ks and have a long exposure time per frame, do not use values above 2 or 3 as half-width, the assumption of a constant sky in time is not valid anymore.
Default behaviour is to look for cross-correlating spots automatically, locate a small number of spots around the center of the first frame, and refine the inputs given by the FITS headers. Frame shifts are by default performed by resampling using a hyperbolic tangent kernel for interpolation. The default average method is filtered, rejecting a small percentage of lowest and highest pixel values on each time line.
Default is to apply subtraction of the median from each row, to take care of saturation effects. No image viewer is launched at the end by default.
For your information: jitter has been reported to reduce such
SOFI batches on a Solaris station with only 32 megs of RAM (and tons of
disk space), without any problem. The process is much slower, but it
runs fine. I would say that 32 megs in RAM is the minimal amount to get
a decent working jitter process. If you have a slow machine or
little RAM, launch your processes at night with an at or
cron command and get the results in the morning.
% ls -1 *.fits > framelist.ascii
Replace *.fits by the wildcards you need to describe the files you want to process, and framelist.ascii by any name you want, provided it is declared correctly in jitter.ini.
Notice that ls is used with the option -1 (minus one), which requests a list of file names on a single column output.
Beware!
If your ls command is aliased to some exotic combination of options (most people alias ls to ls -F), the results of the previous commands may append some funny characters to the file names. Simple example: if your files have executable permission, and you aliased ls -F, all file names will have asterisks (*) appended, which will prevent jitter from finding the files. Notice that you can de-alias a command by preceding it with a backslash, i.e.
% \ls -1 *.fits > myframelist
If you want to do jitter+offset reductions, you will need to declare in second column the file type for each input frame (see next question). Hint: use dfits and fitsort to create this column output. For ISAAC, the simplest is to do:
% dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii
Examples
The following declares only objects: (no second column -> no sky string -> no SKY plane)--- begin frame1.fits frame2.fits frame3.fits frame4.fits --- end
The following declares frame2.fits and frame4.fits as skies: (their second column contains a sky string)
--- begin frame1.fits frame2.fits sky frame3.fits frame4.fits sky --- end
Same for the following:
--- begin frame1.fits OBJECT_FRAME frame2.fits SKY_FRAME frame3.fits OBJECT_FRAME frame4.fits SKY_FRAME --- end
Same for the following (notice that the second column is case-insensitive to identify 'sky')
--- begin frame1.fits NGC9999 frame2.fits NGC_SKY frame3.fits NGC9999 frame4.fits skyFrame --- end
With SOFI/ISAAC frames, you can get away with a simple:
% dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii
To subtract dark.fits and divide by ff.fits all planes named isaac*.fits in the current directory (with tcsh), and output them to files named isaac*.fits.sub:% foreach i (isaac*.fits) % foreach? ccube "$i dark.fits - ff.fits /" $i.sub % foreach? end
This will happen for example if you have crowded regions. The jitter movements are not sufficient (most will end up on object signal). For some pixels, most observed signal will be object signal, not sky. In that case it becomes impossible to filter reliably the sky out from this signal.
Warning!
It is a terrible idea to observe a very crowded field or an extended object with a simple jitter pattern. Use instead jitter+offset, it has been designed for that purpose. Of course, you are then using precious observation time to observe a blank sky, but it is the only reliable way to get rid of this strong signal in the infrared regime.
There have been many cases already of observers who did not want to
loose any time observing such useless signals as the sky, and were
totally unable to reduce their data. If you have not calibrated the sky
background, you are left out with atmosphere models to estimate it,
without any guarantee of result. You are advised NOT to use the 'jitter'
command in such cases.
Have a look at
jitter and photometry report
An algorithm does not have to be believed but demonstrated. Just as you would not publish a scientific paper where you only quote your opinions, you want to prove that your method is working in a given domain with some restrictions and specified parameters. If you just want to make suggestions, keep them for yourself until they are properly demonstrated. The next step is then to try to convince us of your method.
There are so many suggestions about this topic that we cannot spend our
whole time looking at them into details. Most of them have turned out in
the end to be useless. Best is probably to make a complete and convincing
study of what you suggest before you contact us.
Here is one way to create simple offset lists. The easiest is to create it from header information. Using dfits, fitsort and awk you could do something like:
% dfits *.fits | fitsort -d SEQ.CUMOFFSETX SEQ.CUMOFFSETY
That prints out the values of SEQ.CUMOFFSETX and SEQ.CUMOFFSETY for all FITS files in the current directory. Now if you want to create an offset list from this, you also need to add the plane number, which is usually just sequential from 1 to NPlanes. Using awk, you can easily add that by doing:
% [...] | awk '{print NR-1,$2,$3}' > offsets.in
[...] stands for the previous command, combination of dfits and
fitsort.
A second very simple way to enhance detectability was to use the difference between the first two raw frames as a reference image for cross-correlation, instead of using the first sky-subtracted frame. This modification produced spectacular results in detection efficiency.
The peak finder algorithm can now be described as:
Given an input image, find out:
m = median value of the input image
d = average absolute deviation to the median
Threshold the image at m+K.d, K being a user-provided
parameter (default is 2.0). Apply a morphological binary dilation,
return a list of barycenters of all connected regions.
Some new strategies have been included to do all what's possible to find reliable objects for cross-correlation. The strategies are given hereafter. For each step, if no object was found the algorithm proceeds to next step and so on until suitable objects have been found.
People interested in strategies should have a look at the implementation in C, it is fairly readable. Have a look in cubes/correlation.c, the function to look at is called get_interest_center().
If, despite the algorithm modifications, jitter still cannot find any object in the first frame, try the following tricks.
If your offsets are coming from a file, decrease the same variables in the [FileOffsets] section. If you are using the automatic mode, it may be very dangerous to decrease these values since you might miss an object. The best would probably be to try to find the offsets by another method first (e.g. by eye!), and provide them through a file, asking then for offset refining.
Any other ideas on this subject are welcome!
Last resort: if 'jitter' is still unable to find anything worth
cross-correlating in your images, of if the objects it picks are not
suitable for alignment (e.g. moving objects), you can provide your own
list of objects in an ASCII file. See the corresponding sections in the
jitter.ini file.
What that means is: if you lost the first frame in the batch for any reason, you can still take your batch from image 2 to image N and feed it into 'jitter' directly. By subtracting the offsets in the first image, you will implicitly use frame number 2 as the reference frame for alignment.
Now it might be the case that frame number 2 is not the best centered frame, and you want to use, say frame number 11. Easy: just declare frame 11 as the first frame in your framelist, it will be used as reference. If its offsets are non-zero, they will again be subtracted from all consecutive offsets anyway.
Which frame should you use in case you are missing the first one? If you
used an AutoJitter template, the best centered frames are those falling
on a multiple of the Poisson homogeneity (see the HIERARCH ESO SEQ
POISSON keyword). So if HIERARCH ESO SEQ POISSON is 10, and you are
missing the first frame in the batch, you can use frame number 11, 21,
31, etc.
from plane 1 to plane 18: [-4.24 -54.42] (25.42) from plane 1 to plane 19: [-57.44 -28.21] (33.38) from plane 1 to plane 20: [ 1.96 45.27] (1396.27) from plane 1 to plane 21: [-18.35 -27.00] (35.13) from plane 1 to plane 22: [-37.79 -44.23] (1396.86) from plane 1 to plane 23: [37.00 41.88] (25.61)
The 'jitter' command determines offsets between frames by using a cross-correlation criterion. This measurement is searching for an offset between the reference and the candidate frame, that minimizes the sum of squared differences between the frames. The value printed out in parentheses along with the offset measurements is the minimal difference value found in cross-correlation. The smaller the value, the better the match between frames. The theoretical limit for this value is zero (perfect match) in the case of a frame that is shifted by an integer translation vector. There is no absolute reference to measure the "goodness of match" with this value, since it will depend on:
To have a visual indication of what this value is, you can have a look at the jitter algorithmic manual: http://www.eso.org/projects/dfs/papers/jitter99/node11.html
The figure shows what a cross-correlation matrix looks like. The value returned in parentheses by the 'jitter' command is the lowest value of this matrix (the minimum of difference).
As a rule of thumb, for standard SOFI/ISAAC images, your can roughly say the following:
If the distance:
That tends to work if the objects chosen for cross-correlation are normal stars. That will not work if for example the signal which is cross-correlated is a bright galaxy with a high background signal. In that case, these empiric thresholds should not apply.
A better way to determine the "goodness of match" is to look at the returned offsets. If the search domain was set to say +/-50 pixels in X and Y, and the returned offset indicates a value of 50 or -50 (exactly) in either direction, it means that the minimal difference was found on the border of the search domain. Usually, this is an indication of failure. The chosen search domain should always be a little bigger than the expected offset throw.
Starting from eclipse 3.6, a test is done on the offsets returned by
the cross-correlation procedure. If the returned offsets are returned
equal to the size of the search domain, the frame is declared bad for
correlation and removed from the stack of frames.
There is an article about image resampling, containing a fair description of what an interpolation kernel is. You should have a look at the ESO Messenger (issue 100 - July 2000). The article is called: Astronomical image resampling. You can download the July 2000 issue of the Messenger from the ESO Web site, at:
Median is a median 3d average. Take a line of pixels along time, extract the median: it is the output pixel value for this position. This method is also bad, because it does not sum up faint signals.
Filtered is somewhere in between: take a line of pixels along
time, sort it out by increasing pixel value, remove the highest and
lowest ones (outliers) and average the remaning pixels. This is the
output pixel value for this detector position. This is probably the
best method, because it rejects outliers but still sums faint signals.
Since version 3.0, eclipse uses a new (hopefully portable) memory allocation scheme. Without going into too many details, the library opens its own swap files in a temporary directory and tells the Operating System to use it as virtual memory.
Some process monitoring tools such as top will report these
temporary files (of litterally unlimited size) as memory use, thus
providing a false information. Do not be alarmed by the quantities
announced on top on systems such as Solaris and Linux, they
are just bugged. jitter has been reported to show a memory use
of more than 2 gigabytes (on a 32-bit system, that's not bad) with no
more than half a gig of memory space (RAM chips + swap space)... Do
not always believe blindly your memory monitoring tools!
On the other hand, if you run the complete process of sky estimation
and shift-and-add, it is possible to save the sky-subtracted planes
after sky estimation to a single FITS cube on the disk. To re-enter the
jitter process you will have to modify the list of input files to a
single file (the intermediate sky-subtracted cube) and disable sky
estimation to jump directly to shift-and-add.
% limit cputime unlimited filesize unlimited datasize 2097148 kbytes stacksize 8192 kbytes coredumpsize 0 kbytes vmemoryuse unlimited descriptors 64
Numbers will actually vary with the platform, these are the default values for mine. The parameter you want to change is descriptors. Move it to unlimited by typing:
% limit descriptors unlimited % limit cputime unlimited filesize unlimited datasize 2097148 kbytes stacksize 8192 kbytes coredumpsize 0 kbytes vmemoryuse unlimited descriptors 1024
While you're at it, check out that the values for cputime,
filesize, datasize and vmemoryuse are
unlimited (or as high as they can), and that coredumpsize is
zero, unless you want to flood your disk with core files at the first
crash.
If sky background computation is activated, section [SkyBackground], variable Computation, you will get an ASCII file in output containing what has been found about the sky background, for each input frame. If you could provide DIT, pixel scale, pixel gain and zero point, the outputs will also be shown in magnitudes per square arcsec.
If in the section [Detpeak] you ticked the option OutputObjects = yes, a file containing all objects used from cross-correlation will be produced in the current directory.
If in the [Offsets] section you activated the flag [OffsetFileOutput], the offsets used for shift-and-add will be written to an ASCII file.
Last, if you ticked in the section [PostProcessing] the
ProduceStatusReport variable, a complete report status will be
generated in the current directory, containing all of the above
information (except the sky-subtracted input planes).
EXPTIME provides the total integration time in seconds; it may have decimals. When the exposure is made of several periods, EXPTIME time is the sum of the exposure periods, and not simply the difference between end and start of exposure. Subintegrations, i.e. multiple exposures before a readout of the detector are described by the DIT and NDIT.
The reference is The Data Interface Control Board document.
This being said, the EXPTIME written in the output header by 'jitter' is
the EXPTIME found in the first input header (NOT the sum of all
EXPTIME values from all input files). This has been done upon
request from the instrument scientist.
PRO CATG product category, identifies the frame as COADDED_IMG PRO DATE date the jitter process was executed PRO DATANCOM how many frames were combined to produce this one