|
CR2RE Pipeline Reference Manual 1.6.2
|
The European Southern Observatory (ESO) provides pipelines to reduce data for almost all Very Large Telescopes (VLT) instruments. In order to reduce the cost of development, verification, and maintenance of ESO pipelines, and at the same time to improve the scientific quality of pipelines data products, ESO develops a limited set of versatile and instrument-independent high-level scientific functions to be used in the pipelines. These routines are provided by the High-level Data Reduction Library (HDRL). This Doxygen based documentation provides in the following sections information on the HDRL Releases, on its Usage, what are the external packages the library depends on, and a few other useful links.
There is no fixed release cycle for the HDRL library (as e.g. for CPL), but new releases are feature-driven, i.e. if there are new functionality/algorithms available and carefully tested a new release will be announced and the pipeline developer can change the svn::external to this new release. This has the advantage that the pipeline developer has more freedom to decide when to update the pipeline. On the other hand it allows the developer to incorporate the new HDRL release to his pipeline on a very short timescale and prepare a new Paranal/public release. This is only possible as the library is not installed in the Data Flow System environment in Paranal (like CPL) but delivered inside each pipeline.
The HDRL release version 1.5.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.5.0
In this release we updated and/or added the following algorithms:
The HDRL release version 1.4.0 can be included from http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.4.0
In this release we updated and/or added the following algorithms:

To the statistical estimators we added the mode of a distribution, i.e. the following algorithms are now supported for collapsing imagelists or deriving statistics on images:

Please note that all methods but the mode are doing error propagation. The mode method is special in this case as it calculates the error from the data. The error estimation can either be done analytically or based on a bootstrap Montecarlo simulation. In this case the input data are perturbed with the bootstrap technique and the mode is calculated N times (controlled with a parameter). From this N modes the standard deviation is calculated and returned as error. See the manual for detailed information on the mode algorithm.
The HDRL release version 1.3.0 can be included from http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.3.0
In this release we updated and/or added the following algorithms:
The HDRL release version 1.2.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.2.0
In this release we updated and/or added the following algorithms:
m1_radiusand
m2_radiuswas corrected. The code was correct.
The HDRL release version 1.1.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.1.0
In this release we added five new algorithms:
The HDRL release version 1.0.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.0.0
In order to provide astrometric and photometric calibration information, the HDRL implements in this release a functionality to generate a catalogue of detected objects (i.e. stars, galaxies).
A high-level summary of the implemented data reduction sequence is:
The HDRL release version 0.3.0b1 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.3.0b1
In this release we added an algorithm to do fringe correction. In a first step the algorithm creates a master-fringe image using a Gaussian mixture model. A properly scaled version of the master-fringe image is then used to remove the fringes from the single images.
The HDRL release version 0.2.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.2.0
In this release we added two algorithms to derive a master flatfield and one algorithm to compute the Strehl ratio
The HDRL release version 0.1.5 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.1.5
The sigma clipping algorithm has been changed. It now uses a scaled Median Absolute Deviation (MAD) to derive a robust RMS for the clipping and not anymore the interquartile range (IQR). The MAD gives better results for the case for low number statistics and a high fraction of pixels affected by e.g. cosmic ray hits. Furthermore, the library integration in the pipeline slightly changed.
The HDRL release version 0.1.0 can be included from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.1.0
Various methods for bad pixel detection are added in this release.
Relationship with CPL and other libraries:
The latest hdrl library depends on
Th HDRL library should be included into the pipeline library during the svn checkout by using the svn::external concept. The external should point to a given library release in the tags and not to the trunk, i.e. to a subdirectory of
http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl
. In order to build the library one has to add the hdrl.m4 from the common m4macros repository (Pipelines/common/m4macros in SVN) to the pipeline sources.
Then one has to modify the configure.ac and the top-level Makefile.am as follows:
# mark hdrl folder as containing a configurable package AC_CONFIG_SUBDIRS([hdrl]) # check and define all required variables to build and # link hdrl external located in the hdrl folder HDRL_CHECK([hdrl])
SUBDIRS = hdrl ...
Moreover, the variable must also be added to the link variable, to the linker flag variable, and to the AM_CPPFLAGS variable in the Makefile.am of any folder making use of HDRL. As HDRL is currently a static library it also needs has to be added as a dependency of objects using it so these are relinked when HDRL changes. For example:
hdrldemo_bias_la_LDFLAGS = $(HDRL_LDFLAGS) ... hdrldemo_bias_la_LIBADD = $(HDRL_LIBS) ... hdrldemo_bias_la_DEPENDENCY = $(LIBHDRL) ...
In the source files, the only include needed is:
#include <hdrl.h>
In order to test the various algorithms the HDRL team has also written the hdrldemo pipeline. The pipeline follows the version numbers of the HDRL release and can be downloaded from a sub-directory of
http://svnhq2.hq.eso.org/p2/tags/Pipelines/hdrldemo/
For example version 1.2.0 can be downloaded from
http://svnhq2.hq.eso.org/p2/tags/Pipelines/hdrldemo/hdrldemo-1.2.0/hdrldemop
Please note that the pipeline is only meant to test the HDRL library and the recipes are in most of the cases not ready for operations, i.e. the HDRL team spends very little resources on the hdrldemo pipeline.