Frequently Asked Questions about jitter
Frequently Asked Questions about jitter
© European Southern Observatory
FAQ Revised: Monday 11 March 2002 13:01:21
Long answer: the eclipse team is dedicated to provide pipeline tools to the community to help them reduce their data. We are by no means an absolute reference about infrared data reduction.
If you happen to find a bug or have comments about the methods offered by the pipeline, you are welcome to contact us: eclipse-team AT eso DOT org.
If you have problems with your data, you are invited to contact the User Support Group in Garching to clarify points related to data acquisition.
If you have problems understanding what should be done on your data,
you need expert assistance. The eclipse team can unfortunately
not provide that level of service. The User Support Group might give
you hints. Reading all ISAAC documentation might be a good idea too.
See the ISAAC Data Reduction Guide (from the ISAAC or eclipse
home page).
Links:
jitter is trying to be a good tool, not a good worker. There is
no replacement so far for the attention an astronomer can have on a data
set, and the scientific appreciation of the results. Consider this
software as a transparent box that automates many tasks for you but has to
be tailored to your data sets to produce the most efficient results.
Default parameters are set for a slowly varying sky, i.e. the sky can be considered constant in time over a range of 2*RejectHalfWidth+1 frames. If you are observing in J or H with a total exposure time of 1 second per frame, it is reasonable to set the half-width of the running filter to a high value. If you are observing in K or Ks and have a long exposure time per frame, do not use values above 2 or 3 as half-width, the assumption of a constant sky in time is not valid anymore.
Default behaviour is to look for cross-correlating objects automatically, locate a small number of objects around the center of the first frame, and refine the inputs given by the FITS headers. Frame shifts are by default performed by resampling using a hyperbolic tangent kernel for interpolation. The only average method is filtered, rejecting a user-specified number of min and max pixels on each time line before averaging.
Default is to apply subtraction of the median from each row, to take care of saturation effects. No image viewer is launched at the end by default.
For your information: jitter has been reported to reduce such SOFI batches on a Solaris station with only 32 megs of RAM (and loads of disk space), without any problem. The process is considerably slower, but it runs fine. I would say that 32 megs in RAM is the minimal amount to get a decent working jitter process. If you have a slow machine or little RAM, launch your processes at night with an at or cron command and get the results in the morning.
Do not push it too far, though. The memory handling mechanism in eclipse is able to deal with far more memory than you truly have (RAM chips + swap space on disk) because of this internal swap page mechanism. But it cannot allocate infinite resources. You might still run out of memory in the following cases:
There are many things that might fail if you are trying to launch
processing of huge data sets, and there is nothing that can be done to
prevent that. If you really need to run jitter on very large
numbers of frames, you have something wrong anyway. See below,
question about combining several data sets together.
% ls -1 *.fits > framelist.ascii
Replace *.fits by the wildcards you need to describe the files you want to process, and framelist.ascii by any name you want, provided it is declared correctly in jitter.ini.
Notice that ls is used with the option -1 (minus one), which requests a list of file names on a single column output.
Beware!
If your ls command is aliased to some exotic combination of options (most people alias ls to ls -F), the results of the previous commands may append some funny characters to the file names. Simple example: if your files have executable permission, and you aliased ls -F, all file names will have stars (*) appended, which will prevent jitter from finding the files. Notice that you can de-alias a command by preceding it with a backslash, i.e.
% \ls -1 *.fits > myframelist
If you want to do jitter+offset reductions, you will need to declare in second column the file type for each input frame (see next question). Hint: use dfits and fitsort -d to create this column output. For ISAAC, the simplest is to do:
% dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii
Long answer:
Here is an example: you have observed the same field during several nights, getting in the end N data sets, each of them containing typically 30 to 60 frames. Since you want to get an image of the field that includes all of your observations, you are wondering if you could give all these files to jitter to magically re-combine them to a single image.
Well... this is not possible. Several reasons:
Agreed, this means that your frames will get interpolated a second time, but this should not be an issue. First, each reduced frame has already a quite high signal to noise ratio, which ensures that re-sampling will not act randomly. Second, there are by definition no very high frequencies in astronomical images because the acquisition system has been designed so. Re-sampling is not likely to cause artefacts with well-behaving signals like these.
If you are really worried about re-sampling the frames a second time,
you can always work on the offsets between all frames to make them
consistent relative to the first one in the batch. You will have to
find a way of subtracting the sky in each batch to produce
correctly sky-subtracted frames. And then you can try pouring all your
frames into the same framelist and hope that your machine does not
crash due to memory/CPU load. Compare what you will get in the end to
the doubly-interpolated image and try to determine how much you have
lost in photometry precision, compared to the time you have lost doing
all of that. ;->
Examples
The following declares only objects: (no second column -> no sky string -> no SKY plane)--- begin frame1.fits frame2.fits frame3.fits frame4.fits --- end
The following declares frame2.fits and frame4.fits as skies: (their second column contains a sky string)
--- begin frame1.fits frame2.fits sky frame3.fits frame4.fits sky --- end
Same for the following:
--- begin frame1.fits OBJECT_FRAME frame2.fits SKY_FRAME frame3.fits OBJECT_FRAME frame4.fits SKY_FRAME --- end
Same for the following (notice that the second column is case-insensitive to identify 'sky')
--- begin frame1.fits NGC9999 frame2.fits NGC_SKY frame3.fits NGC9999 frame4.fits skyFrame --- end
With SOFI/ISAAC frames, you can get away with a simple:
% dfits *.fits | fitsort -d DPR.TYPE > framelist.ascii
This will happen for example if you have crowded regions. The jitter movements are not sufficient (most will end up on object signal). For some pixels, most observed signal will be object signal, not sky. In that case it becomes impossible to filter reliably the sky out from this signal.
Warning!
It is a terrible idea to observe a very crowded field or an extended object with a simple jitter pattern. Use instead jitter+offset, it has been designed for that purpose. Of course, you are then spending observation time to observe a blank sky, but it is the only reliable way to get rid of this strong signal in the infrared regime.
There have been many cases already of observers who did not want to
loose any time observing such useless signals as the sky, and were
totally unable to reduce their data. If you have not calibrated the
sky background, you are left out with atmosphere models to estimate
it, without any guarantee of result. You are advised NOT to use the
'jitter' command in such cases.
Have a look at
jitter and photometry report
An algorithm does not have to be believed but demonstrated. Just as you would not publish a scientific paper where you only quote your opinions, you want to prove that your method is working in a given domain with some restrictions and specified parameters. If you just want to make suggestions, make sure they are properly documented and the result of a scientific study, not just your own belief.
The next step is then to try to convince us of your method. There are
so many suggestions about this topic that we cannot spend our whole
time looking at them into details.
This method is Ok to provide a quick-look on the data set (which is what the pipeline is meant to do), but insufficient in many cases. In K band, the sky sometimes varies too fast for this method to provide a reliable sky plane. You might want to produce several sky planes in this case, combining them with whatever 3d filter seems appropriate to get a good estimate. It is definitely recommended to measure sky background variations on the sky frames before deciding on a way to filter them.
Once you have correctly subtracted out the sky, you can give the frames to
jitter as you would with raw frames, disabling of course the sky
estimation feature.
Here is one way to create simple offset lists. The easiest is to create it from header information. Using dfits, fitsort and awk you could do something like:
% dfits *.fits | fitsort -d seq.cumoffsetx seq.cumoffsety
That prints out the values of SEQ.CUMOFFSETX and SEQ.CUMOFFSETY for all FITS files in the current directory. Now if you want to create an offset list from this, you also need to add the plane number, which is usually just sequential from 1 to NPlanes. Using awk, you can easily add that by doing:
% [...] | awk '{print NR-1,$2,$3}' > offsets.in
[...] stands for the previous command, combination of dfits and
fitsort.
Last resort: if 'jitter' is still unable to find anything worth
cross-correlating in your images, of if the objects it picks are not
suitable for alignment (e.g. moving objects), you can provide your own
list of objects in an ASCII file. See the corresponding sections in the
jitter.ini file.
from plane 1 to plane 18: [-4.24 -54.42] (25.42) from plane 1 to plane 19: [-57.44 -28.21] (33.38) from plane 1 to plane 20: [ 1.96 45.27] (1396.27) from plane 1 to plane 21: [-18.35 -27.00] (35.13) from plane 1 to plane 22: [-37.79 -44.23] (1396.86) from plane 1 to plane 23: [37.00 41.88] (25.61)
The 'jitter' command determines offsets between frames by using a cross-correlation criterion. This measurement is searching for an offset between the reference and the candidate frame, that minimizes the sum of squared differences between the frames. The value printed out in parentheses along with the offset measurements is the minimal difference value found in cross-correlation. The smaller the value, the better the match between frames. The theoretical limit for this value is zero (perfect match) in the case of a frame that is shifted by an integer translation vector. There is no absolute reference to measure the "goodness of match" with this value, since it will depend on:
To have a visual indication of what this value is, you can have a look at the jitter algorithmic manual: http://www.eso.org/projects/dfs/papers/jitter99/node11.html
The figure shows what a cross-correlation matrix looks like. The value returned in parentheses by the 'jitter' command is the lowest value of this matrix (the minimum of difference).
As a rule of thumb, the values you get should all be more or less
consistent. The lower the better, but the most important is to get
variations in this distance measurement as small as possible.
Cross-correlation is failing on a number of signals. If your image exhibits a strong periodical pattern, any replica in the pattern might be a good candidate for cross-correlation, introducing large matching errors in the output. A solution is to increase the size of the measurement area until the reference zone does not exhibit any periodical pattern.
Another source of errors could be that the object used for cross-correlation reference is elongated (e.g. a galaxy). In the direction of elongation, any position is as likely to be picked as another. This would introduce systematic errors like an elongation of all stars in the final image. If you happen to be in such a case, try picking yourself the main object used for correlation, e.g. a bright star that can be seen on all frames. You could also increase the number of objects used in cross-correlation to compensate errors.
Resampling is not completely harmless. A new signal is derived from the input signal, using some assumptions in the process like signal smoothness. If you happen to have very high frequencies (typically bad pixels) this is likely to disturb the neighbouring pixels and introduce some errors in final pixel values. If your signal behaves correctly and smooth enough, you should not have introduced any noise at all through resampling, though.
The final stacking of images is also applying a 3d filter to remove the
highest and lowest pixel values before summing up. This will systematically
remove faint signals that appear on few frames, which might not be what you
want to do. If you are interesting in keeping all input signal in the
output, you should try linear averaging for the stacking and filter out bad
pixels by yourself. Be aware that removing pixels systematically might be a
cause of bias if your signal behaves irregularly.
There is an article about image resampling, containing a fair description of what the interpolation scheme is. You should have a look at the ESO Messenger (issue 100 - July 2000). The article is called: Astronomical image resampling. You can download the July 2000 issue of the Messenger from the ESO Web site, at:
The first method maintains a consistent signal-to-noise ratio over the
frame, the second one maintains a consistent photometry for objects
located on the edges. Version 3 and earlier of eclipse offered only
the first method. Version 4 currently only offers the second method,
an option might be added later on if the need arises.
Since version 3.0, eclipse uses a new (hopefully portable) memory allocation scheme. Without going into too many details, the library opens its own swap files in a temporary directory and tells the Operating System to use it as virtual memory.
Some process monitoring tools such as top will report these
temporary files (of litterally unlimited size) as memory use, thus
providing a false information. Do not be alarmed by the quantities
announced on top on systems such as Solaris and Linux, they
are just bugged. jitter has been reported to show a memory use
of more than 2 gigabytes (on a 32-bit system, that's not bad) with no
more than half a gig of memory space (RAM chips + swap space)... Do
not always believe blindly your memory monitoring tools!
On the other hand, if you run the complete process of sky estimation
and shift-and-add, it is possible to save the sky-subtracted planes
after sky estimation to a single FITS cube on the disk. To re-enter the
jitter process you will have to modify the list of input files to a
single file (the intermediate sky-subtracted cube) and disable sky
estimation to jump directly to shift-and-add.
EXPTIME provides the total integration time in seconds; it may have decimals. When the exposure is made of several periods, EXPTIME time is the sum of the exposure periods, and not simply the difference between end and start of exposure. Subintegrations, i.e. multiple exposures before a readout of the detector are described by the DIT and NDIT.
The reference is The Data Interface Control Board document.
This being said, the EXPTIME written in the output header by 'jitter' is
the EXPTIME found in the first input header (NOT the sum of all
EXPTIME values from all input files). This has been done upon
request from the instrument scientist.
PRO CATG product category, identifies the frame as COADDED_IMG PRO DATE date the jitter process was executed PRO DATANCOM how many frames were combined to produce this one