Advanced Features
=================

.. _userguide.quickstart.complex-data-products:

Creating complex data products
------------------------------

In the previous example :ref:`userguide.quickstart.complete-recipe` the
utility functions of the ``cpl.dfs`` module were used to create a ESO
standards compliant image data product. While there are also utilities to
create image list and table data products, it may happen that these utilities
are not enough. For instance there is no utility function available which
would create a data product which has several extensions (e.g. data, variance
and quality information). How such products can still, easily be created is
shown in this example.

The example is again the ``basic_science`` recipe example. However here, the way
the product file is created has changed. Instead of creating a FITS product file
where the image data is stored in the FITS primary data unit, a FITS file with
an ESO standard compliant primary FITS header and an empty FITS primary data unit
is created first. The result image is then added to this product FITS file as an
image extension:

.. include:: examples/pycpl_recipe_basic_science_extensions.py
  :code: python

The primary FITS header, is written to a file using the utility functions from
the ``cpl.dfs`` module to make sure that the primary FITS header of the output
file meets the formal requirements of a standard ESO data product. Because an
extension header is just a normal FITS header, the basic *PyCPL* saving functions
can be used to add extensions to the product file.

Note that an extension header should still follow the general ESO guidelines for
data products or may need to have additional keywords to be compliant with the
ESO Science Data Producs standard. For instance the extension header should
have keywords like ``EXTNAME`` added, or other keywords characterizing and
describing the data in this extension. This has been skipped here deliberately
to illustrate the situation where no additional information should be passed,
i.e. how an empty header (except for the information needed for a valid FITS
image extension) can be created.

Error handling
--------------

Errors from PyCPL
.................

In Python errors which occur in the code at runtime are reported through
exceptions, which can be raised, catched and handled, or re-raised. In order
to fit within this environment returning error codes, which is the way
C libraries usually deal with errors, is not the best solution.

For this reason *PyCPL* provides custom exceptions which seamlessly integrate
with the standard Python exceptions. These custom *PyCPL* exceptions are also
representing the errors of the CPL C library which are translated into
exceptions. Because these translated errors have become Python exceptions,
they can be used in exactly the same way as any other Python exception, i.e.
they can be raised, caught, handled and re-raised.

Python code using PyCPL exceptions
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Python code can raise any of the custom *PyCPL* exceptions. This is
illustrated in the following.

The recipe expects to find input frames of type ``OBJECT`` in the input
set-of-frames file. Therefore feeding ``basic_science`` with inappropriate
input data, for instance the BIAS frames of the ``gimasterbias`` recipe,
will raise a custom *PyCPL* exception, ``cpl.core.DataNotFoundError``.

Because the exceptions is not handled the recipe will terminate showing
the familiar Python traceback output.

.. code:: python

  >>> p = Pyesorex()
  [ INFO  ] pyesorex: This is PyEsoRex, version 1.0.1.
  >>> p.recipe = "basic_science"
  [ INFO  ] pyesorex: Loaded recipe 'basic_science'.
  >>> p.sof_location = "bias.sof"
  >>> p.run()
  [ INFO  ] pyesorex: Running recipe 'basic_science'...
  [WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits' with unexpected tag 'BIAS', ignoring.
  [WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits' with unexpected tag 'BIAS', ignoring.
  [WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits' with unexpected tag 'BIAS', ignoring.
  [WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T10:03:05.335.fits' with unexpected tag 'BIAS', ignoring.
  [WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T10:08:15.239.fits' with unexpected tag 'BIAS', ignoring.
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/home/dummy/VirtualEnvs/pycpl/lib/python3.12/site-packages/pyesorex/pyesorex.py", line 1231, in run
      result = self.recipe.run(self.sof, self.recipe_parameters.as_dict())
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/dummy/test/pyrecipes/pycpl_recipe_basic_science.py", line 71, in run
      raise core.DataNotFoundError("No raw frames in frameset.")
  cpl.core.DataNotFoundError: Most recent error last:
    File "/home/dummy/test/pyrecipes/pycpl_recipe_basic_science.py", line 71, in run
  No raw frames in frameset.

Errors from CPL
^^^^^^^^^^^^^^^

When calling a *PyCPL* function results in an error in the underlying CPL function,
the error from the CPL library is translated into a Python exception. It can be used
like any other exception in Python.

For example, trying to create a *PyCPL* ``cpl.core.Mask`` object with illegal,
negative dimensions, results in an ``cpl.core.IllegalInputError``:

.. code:: python

  >>> from cpl.core import Mask, IllegalInputError, Msg
  >>> mask = Mask(2, 2)
  >>> print(mask)
  #----- mask: 1 <= x <= 2, 1 <= y <= 2 -----
          X       Y       value
          1       1       0
          1       2       0
          2       1       0
          2       2       0

  >>> mask - Mask(2, -2)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<string>", line 87, in __new__
  cpl.core.IllegalInputError: Most recent error last:
    File "../../../cplcore/cpl_mask.c", line 798, in cpl_mask_new
  Illegal input

Compared to the previous example where a *PyCPL* exception was raised directly
from Python code, the exception here is actually triggered by an error reported
by the underlying CPL function. This is visible in the resulting Python traceback,
which shows the CPL source file, line number and function where the problem
was detected.

Because CPL errors become ordinary Python exceptions they can be used like
those, i.e. they can be raised, caught, handled or re-raised:

.. code:: python

  >>> def mask_maker(*args):
  ...     try:
  ...         mask = Mask(*args)
  ...     except IllegalInputError as err:
  ...         Msg.error("mask_maker", f"Tried to create Mask with illegal dimensions: {args!r}")
  ...         raise err
  ...     return mask
  ... 
  >>> Msg.set_config(show_domain=False, show_component=True)
  >>> mask_maker(1, 3)
  #----- mask: 1 <= x <= 1, 1 <= y <= 3 -----
          X       Y       value
          1       1       0
          1       2       0
          1       3       0

  >>> mask_maker(0, 3)
  [ ERROR ] mask_maker: Tried to create Mask with illegal dimensions: (0, 3)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "<stdin>", line 6, in mask_maker
    File "<stdin>", line 3, in mask_maker
    File "<string>", line 87, in __new__
  cpl.core.IllegalInputError: Most recent error last:
    File "../../../cplcore/cpl_mask.c", line 797, in cpl_mask_new
  Illegal input

Errors occurring in native CPL Recipes
......................................

When an application runs a native CPL recipe as a plug-in and an error
occurs inside the CPL recipe causing it to terminate for instance with
a segmentation fault it will also terminate the the calling application
because the plug-in runs as part of this process.

This is not a desirable behaviour if this happens to a user's Python session
while processing data using ESO instrument pipelines and *PyCPL*. In order to
prevent a user's Python session being torn down by a faulty native CPL recipe,
*PyCPL* executes native CPL recipes in a separate child process. A failing CPL
recipe will therefore only affect the child process, but not the Python
interpreter that was used to call the recipe. Within the Python interpreter a
forcibly terminated CPL recipe is therefore simply reported as an error.

Running Recipes from Python using Low-Level PyCPL Interfaces
------------------------------------------------------------

TBD

Debugging Native CPL Recipes when using PyCPL
---------------------------------------------

TBD
