High-Level Data Reduction Library: HDRL Introduction

High-Level Data Reduction Library 1.6.0 High-Level data reduction routines for ESO pipelines

Loading...
Searching...
No Matches
HDRL Introduction

The European Southern Observatory (ESO) provides pipelines to reduce data for almost all Very Large Telescopes (VLT) instruments. In order to reduce the cost of development, verification, and maintenance of ESO pipelines, and at the same time to improve the scientific quality of pipelines data products, ESO develops a limited set of versatile and instrument-independent high-level scientific functions to be used in the pipelines. These routines are provided by the High-level Data Reduction Library (HDRL). This Doxygen based documentation provides in the following sections information on the HDRL Releases, on its Usage, what are the external packages the library depends on, and a few other useful links.

# Releases

There is no fixed release cycle for the HDRL library (as e.g. for CPL), but new releases are feature-driven, i.e. if there are new functionality/algorithms available and carefully tested a new release will be announced and the pipeline developer can change the svn::external to this new release. This has the advantage that the pipeline developer has more freedom to decide when to update the pipeline. On the other hand it allows the developer to incorporate the new HDRL release to his pipeline on a very short timescale and prepare a new Paranal/public release. This is only possible as the library is not installed in the Data Flow System environment in Paranal (like CPL) but delivered inside each pipeline.

## Release version 1.6.0

The HDRL release version 1.6.0 can be obtained from the HDRL Git repository

https://gitlab.eso.org/pipelines/hdrl.git (tag hdrl-1.6.0)

In this release we updated and/or added the following algorithms:

- The HDRL catalogue has been updated to follow ESO Phase 3 conventions. This ensures compatibility with ESO's Phase 3 data products and standards.
- The build system has been enhanced to support HDRL as a standalone third-party library. While HDRL remains usable as a convenience library as before, it can now be installed as a standalone library using the `--enable-standalone` configure option (false by default). This provides greater flexibility for integration into different build environments.
- Resampling: Fixed PIPE-12792 bug where rotated WCS (with significant off-diagonal CD matrix terms) caused unreasonably large output grid dimensions and memory allocation failures. The fix automatically corrects pixel scale values (delta_ra/delta_dec) when they are incorrectly extracted using only diagonal CD matrix elements, ensuring proper computation accounting for rotation.
- Various quality control and bug-fix improvements have been included (see ChangeLog for detailed information).
- Licensing and documentation: clarified GPLv2-or-later licensing by homogenising headers and FSF address, replacing LICENSE with a canonical COPYING file, and adding GNU-style distribution files (README, INSTALL, AUTHORS, ChangeLog, NEWS) that are shipped in the tarball and installed under docdir for standalone builds.

## Release version 1.5.0

The HDRL release version 1.5.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.5.0

In this release we updated and/or added the following algorithms:

- The computation of the barycenric correction, i.e. the wavelength shift to apply to a spectrum to compensate for the motion of the observer with respect to the barycenter of the solar system. The implemented algorithm derives the barycentric correction of an observation by using the ERFA (Essential Routines for Fundamental Astronomy) library. ERFA is a C library containing key algorithms for astronomy, and is based on the SOFA library published by the International Astronomical Union (IAU). See the manual for detailed information.
- The build system has been modified to include GSL as direct dependency to the hdrl unit-test as direct GSL calls are performed when unit-testing the limiting magnitude module.
- For the barycentric correction algorithm two additional dependencies were added to HDRL.
  - A dependency to the ERFA (Essential Routines for Fundamental Astronomy) library.
  - A dependency to the libcurl (The multiprotocol file transfer library) library

## Release version 1.4.0

The HDRL release version 1.4.0 can be included from http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.4.0

In this release we updated and/or added the following algorithms:

- The computation of the limiting magnitude of an image as defined in the ESO Phase 3 Standard. The limiting magnitude characterizes the depth of an observation and is defined as the magnitude of a unresolved source whose flux is 5 times the noise background, i.e. the magnitude of a point like source detected with . See the manual for detailed information.

- To the statistical estimators we added the mode of a distribution, i.e. the following algorithms are now supported for collapsing imagelists or deriving statistics on images:

  - Mean
  - Weighted mean
  - Min-max rejected mean
  - clipped mean
  - Median
  - Mode

  Please note that all methods but the mode are doing error propagation. The mode method is special in this case as it calculates the error from the data. The error estimation can either be done analytically or based on a bootstrap Montecarlo simulation. In this case the input data are perturbed with the bootstrap technique and the mode is calculated N times (controlled with a parameter). From this N modes the standard deviation is calculated and returned as error. See the manual for detailed information on the mode algorithm.

- Due to the addition of the mode, the functions hdrl_overscan_parameter_create_parlist() and hdrl_collapse_parameter_create_parlist() have changed. The two functions now require an additional default mode hdrl parameter. See the doxygen information of the two functions for more details.

## Release version 1.3.0

The HDRL release version 1.3.0 can be included from http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.3.0

In this release we updated and/or added the following algorithms:

- Resampling of 2-dimensional images and 3-dimensional cubes. A common problem in astronomy is the resampling of images (or cubes) onto a common grid. Ideally, this is done only once in the data reduction workflow as each sub pixel resampling redistributes the flux and leads to correlations. The algorithm provided by the HDRL is doing the 2D and 3D interpolation in 2-dimensional and 3-dimensional spaces, respectively. Currently there are six different interpolation methods implemented:
  - Nearest: Nearest neighbour resampling
  - Linear: Weighted resampling using an inverse distance weighting function
  - Quadratic: Weighted resampling using a quadratic inverse distance weighting function
  - Renka: Weighted resampling using a Renka weighting function
  - Drizzle: Weighted resampling using a drizzle-like weighting scheme
  - Lanczos: Weighted resampling using a lanczos-like restricted sinc as weighting function
- The object catalogue generation code has been updated. In previous releases, pixels with a value of exactly 0 where automatically added to the confidence map as zero and excluded in all further computations. This has been removed.

## Release version 1.2.0

The HDRL release version 1.2.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.2.0

In this release we updated and/or added the following algorithms:

- Detection of fixed pattern noise. A classical example is pick noise, i.e. low-amplitude, quasi-periodical patterns super-imposed on the normal read-noise. It is due to electronic interference and might show up or disappear on short timescales (days or hours). The algorithms tries to identify it by the usage of the power spectrum.
- An error in the documentation of the strehl ratio variable

        m1_radius

  and

      m2_radius

  was corrected. The code was correct.
- In the spectral efficiency computation a sign error in the atmospheric correction was corrected in the documentation as well as in the code.

## Release version 1.1.0

The HDRL release version 1.1.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.1.0

In this release we added five new algorithms:

- Computation of the Strehl ratio. The Strehl ratio is defined as the ratio of the peak image intensity from a point source compared to the maximum attainable intensity using an ideal optical system limited only by diffraction over the telescope aperture. The Strehl ratio is very frequently used to perform the quality control of the scientific data obtained with the AO assisted instrumentation.
- Computation of the spectral efficiency as a function of wavelength: The efficiency is used to monitor the system performance and health. It is calculated from observing flux standard stars (in photometric conditions). Then, the observed 1D spectrum is compared with the reference spectrum, as it would be observed outside the Earth's atmosphere. The reference spectrum is provided by the user, usually via a catalog of standard stars.
- Computation of the spectral response as a function of wavelength: The algorithm is divided in two parts: Telluric correction and Response calculation. In the provided implementation the Telluric correction is optional and can be disabled by the user.
- Computation of the Differential Atmospheric Refraction as a function on wavelength. The differential atmospheric refraction is calculated according to the algorithm from Filippenko (1982, PASP, 94, 715) using the Owens formula which converts relative humidity in water vapor pressure.
- Computation of the effective air mass of an observation.

## Release version 1.0.0

The HDRL release version 1.0.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-1.0.0

In order to provide astrometric and photometric calibration information, the HDRL implements in this release a functionality to generate a catalogue of detected objects (i.e. stars, galaxies).

A high-level summary of the implemented data reduction sequence is:

- estimate the local sky background over the image and track any variations at adequate resolution to eventually remove them,
- detect objects/blends of objects and keep a list of pixels belonging to each blend for further analysis
- parametrise the detected objects, i.e. perform astrometry, photometry and a shape analysis.

## Release version 0.3.0b1

The HDRL release version 0.3.0b1 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.3.0b1

In this release we added an algorithm to do fringe correction. In a first step the algorithm creates a master-fringe image using a Gaussian mixture model. A properly scaled version of the master-fringe image is then used to remove the fringes from the single images.

## Release version 0.2.0

The HDRL release version 0.2.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.2.0

In this release we added two algorithms to derive a master flatfield and one algorithm to compute the Strehl ratio

## Release version 0.1.5

The HDRL release version 0.1.5 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.1.5

The sigma clipping algorithm has been changed. It now uses a scaled Median Absolute Deviation (MAD) to derive a robust RMS for the clipping and not anymore the interquartile range (IQR). The MAD gives better results for the case for low number statistics and a high fraction of pixels affected by e.g. cosmic ray hits. Furthermore, the library integration in the pipeline slightly changed.

## Release version 0.1.0

The HDRL release version 0.1.0 can be included from

http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl/hdrl-0.1.0

Various methods for bad pixel detection are added in this release.

# Dependencies

Relationship with CPL and other libraries:

The latest hdrl library depends on

- The Common Pipeline Library (CPL) version 7.0 or higher. Please note that CPL must be compiled with wcs functionality available
- The GSL - GNU Scientific Library version 1.16 or higher.
- The ERFA (Essential Routines for Fundamental Astronomy) library.
- The libcurl (The multiprotocol file transfer library) library

# SVN usage and library integration

Th HDRL library should be included into the pipeline library during the svn checkout by using the svn::external concept. The external should point to a given library release in the tags and not to the trunk, i.e. to a subdirectory of

    http://svnhq2.hq.eso.org/p2/tags/Pipelines/common/hdrl

. In order to build the library one has to add the hdrl.m4 from the common m4macros repository (Pipelines/common/m4macros in SVN) to the pipeline sources.

Then one has to modify the configure.ac and the top-level Makefile.am as follows:

- configure.ac:

      # mark hdrl folder as containing a configurable package
      AC_CONFIG_SUBDIRS([hdrl])

      # check and define all required variables to build and
      # link hdrl external located in the hdrl folder
      HDRL_CHECK([hdrl])

- Makefile.am:

      SUBDIRS = hdrl ...

Moreover, the variable must also be added to the link variable, to the linker flag variable, and to the AM_CPPFLAGS variable in the Makefile.am of any folder making use of HDRL. As HDRL is currently a static library it also needs has to be added as a dependency of objects using it so these are relinked when HDRL changes. For example:

    hdrldemo_bias_la_LDFLAGS = $(HDRL_LDFLAGS) ...
    hdrldemo_bias_la_LIBADD = $(HDRL_LIBS) ...
    hdrldemo_bias_la_DEPENDENCY = $(LIBHDRL) ...

In the source files, the only include needed is:

    #include <hdrl.h>

# The hdrldemo pipeline

In order to test the various algorithms the HDRL team has also written the hdrldemo pipeline. The pipeline follows the version numbers of the HDRL release and can be downloaded from a sub-directory of

    http://svnhq2.hq.eso.org/p2/tags/Pipelines/hdrldemo/

For example version 1.2.0 can be downloaded from

    http://svnhq2.hq.eso.org/p2/tags/Pipelines/hdrldemo/hdrldemo-1.2.0/hdrldemop

Please note that the pipeline is only meant to test the HDRL library and the recipes are in most of the cases not ready for operations, i.e. the HDRL team spends very little resources on the hdrldemo pipeline.