.. _VLT Instrument Pipelines: https://www.eso.org/sci/software/pipelines

.. _userguide.quickstart.basic-usage:

Getting Started
===============

This section will show you the basic usage of *PyCPL* and *PyEsoRex* looking at
a number of use cases. It will give you an easy entry to using *PyCPL* for
executing recipes and it will also get you started with developing recipes using
*PyCPL*.

.. note:: The following examples assume that the Python virtual environment which
   was used to build and install *PyCPL* and *PyEsoRex* is still active, and that
   the environment variables which were set at the time of the installation are
   still in effect (see :ref:`userguide.installation.installation-of-pycpl`)!
  
   It is also assumed that a source kit installation of the GIRAFFE pipeline
   (available from the `VLT Instrument Pipelines`_ page) is present and that
   the pipeline recipes are located in the directory
   ``$HOME/pipelines/lib/esopipes-plugins``.

Running Recipes Using PyEsoRex
------------------------------

For convenience we define the environment variable :literal:`PYESOREX_PLUGIN_DIR`
to point to the directory where the recipes are installed to avoid hard to read
command examples in the following.  

.. code:: shell

  (pycpl)$ export PYESOREX_PLUGIN_DIR=$HOME/pipelines/lib/esopipes-plugins

Instead of setting the environment variable it would of course be possible to
use the corresponding *PyEsoRex* command line option to specify the recipes
search path.

.. code:: shell

  (pycpl)$ pyesorex --recipe-dir=$HOME/pipelines/lib/esopipes-plugins ...

.. _pyesorex-cli:

Using PyEsoRex as Command Line Tool
...................................

If you are used to run pipeline recipes using *EsoRex*, the easiest way to run
pipeline recipes using *PyCPL* is to use *PyEsoRex* as a drop-in replacement of
*EsoRex*. The look-and-feel of *PyEsoRex* is essentially the same as for
*EsoRex*:

Getting usage information from *PyEsoRex*
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: shell

  (pycpl)$ pyesorex --help
  
Output:

.. include:: outputs/pyesorex_output_help.txt
  :literal:

Show the list of available recipes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: shell
  
  (pycpl)$ pyesorex --recipes

Output:

.. include:: outputs/pyesorex_output_recipes.txt
  :literal:

Running a recipe
^^^^^^^^^^^^^^^^

The following command will execute the GIRAFFE pipeline recipe
:literal:`gimasterbias`, which processes the data provided by the
input set-of-frames :literal:`bias.sof`:

.. code:: shell

  (pycpl)$ pyesorex gimasterbias bias.sof

Output:

.. include:: outputs/pyesorex_output_gimasterbias.txt
  :literal:

The products created by the recipe are by *default* written to the current
working directory together with the pyesorex log file ``pyesorex.log``.

.. code:: shell
  
  (pycpl)$ ls 
  bad_pixel_map.fits  bias.sof    master_bias.fits  pyesorex.log    qc0000.paf

A different, existing output directory can be chosen with the ``output-dir``
command line option

.. code:: shell

  (pycpl)$ pyesorex --output-dir products gimasterbias bias.sof

which will put the recipe products into the subdirectory ``products``.

Running a Recipe from Python using PyEsoRex
...........................................

Instead of using Pyesorex as a command line tool, it is also possible
to use it from Python through its Python API. This is demonstrated in
this section.

Create a PyEsoRex instance:
^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

  >>> from pyesorex.pyesorex import Pyesorex
  >>> p = Pyesorex()
  [ INFO  ] pyesorex: This is PyEsoRex, version 1.0.1.

Inspect PyEsoRex settings
^^^^^^^^^^^^^^^^^^^^^^^^^

.. code:: python

  >>> print(p.get_params_text())
  Pyesorex parameters:

  help                  : False
  config                : 
  create-config         : False
  display-import-errors : False
  link-dir              : /tmp
  log-dir               : .
  log-file              : pyesorex.log
  log-level             : info
  man-page              : False
  msg-level             : info
  no-datamd5            : False
  no-checksum           : False
  output-dir            : .
  output-mkdir          : False
  output-prefix         : out
  params                : False
  products-sof          : 
  products-sof-abspaths : False
  recipes               : False
  recipe-config         : 
  recipe-dir            : /home/dummy/pipelines/lib/esopipes-plugins
  suppress-link         : True
  suppress-prefix       : True
  time                  : True

Note that the value shown for :literal:`recipe_dirs` depends on the
value of the environment variable :literal:`PYESOREX_PLUGIN_DIR`. If
:literal:`PYESOREX_PLUGIN_DIR` is not defined the *PyEsoRex* default
value :literal:`/usr/local/lib/esopipes-plugins` would be shown
instead.

Listing all available recipes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The list of available recipes contains both native CPL recipes
implemented in C and PyCPL recipes implemented in Python if they
are found in the recipe search path. Both types of recipes
are shown in the same format.


.. code:: python

  >>> print(p.get_recipes_text())
  List of available recipes:

  giframestack          : Creates a stacked image from a set of raw images.
  gimasterbias          : Creates a master bias image from a set of raw biases.
  gimasterdark          : Creates a master dark image from a set of raw dark frames.
  gimasterflat          : Create the fiber master flat field and the localization mask.
  giscience             : Process a science observation.
  gistandard            : Process a spectro-photometric standard star observation and compute the instrument response curve.
  giwavecalibration     : Compute dispersion solution from an arc-lamp spectrum.

Loading a recipe
^^^^^^^^^^^^^^^^

We use the same recipe :literal:`gimasterbias`, that was already used already
here :ref:`pyesorex-cli`, which happens to be a native CPL recipe. *PyCPL*
Python recipes are loaded exactly in the same way.

.. code:: python

  >>> p.recipe = "gimasterbias"
  [ INFO  ] pyesorex: Loaded recipe 'gimasterbias'.
  >>> p.recipe
  <cpl.ui.Recipe gimasterbias>

Setting the SOF location
^^^^^^^^^^^^^^^^^^^^^^^^

Telling *PyEsoRex* which set-of-frames to use when running the recipe is a simple
assignment:

.. code:: python

  >>> p.sof_location = "bias.sof"

To view the contents of the input set-of-frames one can create a *PyCPL* Frameset object

.. code:: python

  >>> from cpl.ui import Frame, FrameSet
  >>> sof_frames = FrameSet("bias.sof")
  >>> for frame in sof_frames:
  ...     print(frame)
  ... 
  (file=/home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits, tag=BIAS, group=FrameGroup.NONE, level=FrameLevel.NONE, type=FrameType.NONE)
  (file=/home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits, tag=BIAS, group=FrameGroup.NONE, level=FrameLevel.NONE, type=FrameType.NONE)
  (file=/home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits, tag=BIAS, group=FrameGroup.NONE, level=FrameLevel.NONE, type=FrameType.NONE)
  (file=/home/dummy/test/bias/GIRAF.2019-06-25T10:03:05.335.fits, tag=BIAS, group=FrameGroup.NONE, level=FrameLevel.NONE, type=FrameType.NONE)
  (file=/home/dummy/test/bias/GIRAF.2019-06-25T10:08:15.239.fits, tag=BIAS, group=FrameGroup.NONE, level=FrameLevel.NONE, type=FrameType.NONE)

This shows the attributes, like file name and tag, of each frame in the Frameset object.
As an alternative, the following variant will show some technical details of each frame in addition:

.. code:: python

  >>> for frame in sof_frames:
  ...     print(repr(frame))
  ... 
  cpl.ui.Frame('/home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits', 'BIAS', cpl.ui.Frame.FrameGroup.NONE, cpl.ui.Frame.FrameLevel.NONE, cpl.ui.Frame.FrameType.NONE)
  cpl.ui.Frame('/home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits', 'BIAS', cpl.ui.Frame.FrameGroup.NONE, cpl.ui.Frame.FrameLevel.NONE, cpl.ui.Frame.FrameType.NONE)
  cpl.ui.Frame('/home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits', 'BIAS', cpl.ui.Frame.FrameGroup.NONE, cpl.ui.Frame.FrameLevel.NONE, cpl.ui.Frame.FrameType.NONE)
  cpl.ui.Frame('/home/dummy/test/bias/GIRAF.2019-06-25T10:03:05.335.fits', 'BIAS', cpl.ui.Frame.FrameGroup.NONE, cpl.ui.Frame.FrameLevel.NONE, cpl.ui.Frame.FrameType.NONE)
  cpl.ui.Frame('/home/dummy/test/bias/GIRAF.2019-06-25T10:08:15.239.fits', 'BIAS', cpl.ui.Frame.FrameGroup.NONE, cpl.ui.Frame.FrameLevel.NONE, cpl.ui.Frame.FrameType.NONE)

Running the recipe
^^^^^^^^^^^^^^^^^^

Running the recipe is simply calling the :literal:`run()` method of the
*PyEsoRex* instance. The result of the recipe invokation is a *PyCPL*
Frameset which can be inspected as priviously shown.

.. code:: python

  >>> result = p.run()
  [ INFO  ] pyesorex: Running recipe 'gimasterbias'...
  [ INFO  ] gimasterbias: Creating master bias from 5 bias frames ...
  [ INFO  ] gimasterbias: Combination method is Average
  [ INFO  ] gimasterbias: Averaging 5 images

  [ INFO  ] gimasterbias: Writing master bias image ...
  [ INFO  ] gimasterbias: Creating bad pixel map from master bias frame ...
  [ INFO  ] gimasterbias: Using local median +/- 3.7759 [ADU] as valid pixel value range
  [ INFO  ] gimasterbias: Computing QC1 parameters ...
  [ INFO  ] gimasterbias: Processing product frame 'master_bias.fits' (MASTER_BIAS)
  [ INFO  ] pyesorex: Recipe 'gimasterbias' complete.
  [ INFO  ] pyesorex: Calculating product checksums
  [ INFO  ] pyesorex: Created product master_bias.fits (in place)
  [ INFO  ] pyesorex: Created product bad_pixel_map.fits (in place)
  [ INFO  ] pyesorex: 2 products created
  [ INFO  ] pyesorex: Recipe operation(s) took 2.29 seconds to complete.
  [ INFO  ] pyesorex: Total size of 5 raw input frames = 88.96 MB
  [ INFO  ] pyesorex: => processing rate of 38.87 MB/s
  >>> for frame in result:
  ...     print(frame)
  ... 
  (file=master_bias.fits, tag=MASTER_BIAS, group=FrameGroup.PRODUCT, level=FrameLevel.FINAL, type=FrameType.IMAGE)
  (file=bad_pixel_map.fits, tag=BAD_PIXEL_MAP, group=FrameGroup.PRODUCT, level=FrameLevel.FINAL, type=FrameType.IMAGE)

The output Frameset object will only contain recipe products which are FITS files!

Inspecting Recipe Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Whether a recipe is a native CPL recipe or a PyCPL recipe, they usually have
parameters a user may want to inspect and eventually modify to customize the
processing of the input data set. The parameters of a recipe can be listed
as shown here

.. code:: python

  >>> for parameter in p.recipe_parameters: print(f"{parameter.name}={parameter.value}")
  ... 
  giraffe.stacking.method=average
  giraffe.stacking.ksigma.low=5.0
  giraffe.stacking.ksigma.high=5.0
  giraffe.stacking.minmax.minimum=1
  giraffe.stacking.minmax.maximum=1
  giraffe.masterbias.overscan.remove=False
  giraffe.masterbias.badpixel.clean=False
  giraffe.masterbias.bpm.create=True
  giraffe.masterbias.bpm.factor=5.0
  giraffe.masterbias.bpm.fraction=0.15

Changing Recipe Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^

The following example changes the method used for stacking the input images
from :literal:`average` to from :literal:`median`

.. code:: python

  >>> p.recipe_parameters.update({"giraffe.stacking.method": "median"})
  >>> for parameter in p.recipe_parameters: print(f"{parameter.name}={parameter.value}")
  ... 
  giraffe.stacking.method=median
  giraffe.stacking.ksigma.low=5.0
  giraffe.stacking.ksigma.high=5.0
  giraffe.stacking.minmax.minimum=1
  giraffe.stacking.minmax.maximum=1
  giraffe.masterbias.overscan.remove=False
  giraffe.masterbias.badpixel.clean=False
  giraffe.masterbias.bpm.create=True
  giraffe.masterbias.bpm.factor=5.0
  giraffe.masterbias.bpm.fraction=0.15

Executing the recipe again with the changed parameters

.. code:: python
  
  >>> result = p.run()
  [ INFO  ] pyesorex: Running recipe 'gimasterbias'...
  [ INFO  ] gimasterbias: Creating master bias from 5 bias frames ...
  [ INFO  ] gimasterbias: Combination method is Median
  [ INFO  ] gimasterbias: Finding median of 5 images
  [ INFO  ] gimasterbias: Writing master bias image ...
  [ INFO  ] gimasterbias: Creating bad pixel map from master bias frame ...
  [ INFO  ] gimasterbias: Using local median +/- 3.9626 [ADU] as valid pixel value range
  [ INFO  ] gimasterbias: Computing QC1 parameters ...
  [ INFO  ] gimasterbias: Processing product frame 'master_bias.fits' (MASTER_BIAS)
  [ INFO  ] pyesorex: Recipe 'gimasterbias' complete.
  [ INFO  ] pyesorex: Calculating product checksums
  [ INFO  ] pyesorex: Created product master_bias.fits (in place)
  [ INFO  ] pyesorex: Created product bad_pixel_map.fits (in place)
  [ INFO  ] pyesorex: 2 products created
  [ INFO  ] pyesorex: Recipe operation(s) took 2.96 seconds to complete.
  [ INFO  ] pyesorex: Total size of 5 raw input frames = 88.96 MB
  [ INFO  ] pyesorex: => processing rate of 30.08 MB/s

Working with Recipe Config Files
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In order to keep a changed set of recipe parameters for reusing it later the
recipe parameters can be saved to a configuration file:

.. code:: python

  >>> p.write_recipe_config("gimasterbias.rc")
  [ INFO  ] pyesorex: Wrote config file '/home/dummy/test/gimasterbias.rc'.

The parameter values can now also be edited in the configuration file,
and once finished the updated parameters can be loaded again:

.. code:: python
  
  >>> p.read_recipe_config("gimasterbias.rc")
  [ INFO  ] pyesorex: Read config file '/home/dummy/test/gimasterbias.rc'.

The recipe configuration file which was created calling the
:literal:`write_recipe_config` method is shown here as an example:

.. include:: outputs/pyesorex_config_rcfile.txt
  :literal:


Writing a Simple Recipe in Python
---------------------------------

This section will demonstrate how a simple recipe is implemented with *PyCPL*.
This section can, by no means, be comprehensive and cover the complete *PyCPL*
API, but it will show what is required in order to create a *PyCPL* recipe which
can be executed with *PyEsoRex*.

A Minimal PyCPL Recipe
......................

There is actually not much that is really required. In order to create a 
*PyCPL* recipe one needs to define a class which inherits from ``cpl.ui.PyRecipe``
and this class must at least implement the method ``run``. The method ``run``
has two arguments, the input ``cpl.ui.Frameset`` which contains the input data
files the recipe should process, and a dictionary containing the recipe parameters.
When ``run`` finishes it must return a ``cpl.ui.Frameset``. It will contain the
created products, if there are any.

The only other thing that needs to be done is to provide the necessary
information about the recipe itself. This information is:

  - the recipe name
  - the recipe version
  - the author name
  - the author's contact email address
  - the copyright information
  - a verbose description of the recipe: what its purpose is, how it works, etc.
  - a short, one line description of the recipe's purpose.

The most important recipe information in the previous list is the recipe name.
It is this name that is used to distiguish different recipes and it should therefore
be unique. The name of the file where the recipe is defined does not matter in this
context. Actually, a single Python source file may define more than one recipe.

The first, minimal example is a *PyCPL* version of the usual "Hello, world!".
As Python code this will look like this:

.. include:: examples/pycpl_recipe_helloworld.py
  :code: python

How is this recipe now executed? Assume that the recipe
we just implemented is stored in the following location:

.. code:: shell

  (pycpl)$ ls /home/dummy/test/pyrecipes
  pycpl_recipe_helloworld.py

Executing this recipe is done in exactly the same way as previously shown
for native CPL recipes:

.. code:: python

  >>> from pyesorex.pyesorex import Pyesorex
  >>> p = Pyesorex()
  [ INFO  ] pyesorex: This is PyEsoRex, version 1.0.1.
  >>> print(p.get_params_text())
  Pyesorex parameters:

  help                  : False
  config                : 
  create-config         : False
  display-import-errors : False
  link-dir              : /tmp
  log-dir               : .
  log-file              : pyesorex.log
  log-level             : info
  man-page              : False
  msg-level             : info
  no-datamd5            : False
  no-checksum           : False
  output-dir            : .
  output-mkdir          : False
  output-prefix         : out
  params                : False
  products-sof          : 
  products-sof-abspaths : False
  recipes               : False
  recipe-config         : 
  recipe-dir            : /home/dummy/pipelines/lib/esopipes-plugins
  suppress-link         : True
  suppress-prefix       : True
  time                  : True

In order to see the new PyCPL recipe, which is not stored in the current
recipe search path of *PyEsoRex*, we have to update the ``recipe_dirs``
parameter of *PyEsoRex* to include the directory where
:file:`pycpl_recipe_helloworld.py` is located:

.. code:: python

  >>> p.parameters.update({"recipe_dirs": "/home/dummy/pipelines/lib/esopipes-plugins:/home/dummy/test/pyrecipes"})
  >>> print(p.get_recipes_text())
  List of available recipes:

    giframestack          : Creates a stacked image from a set of raw images.
    gimasterbias          : Creates a master bias image from a set of raw biases.
    gimasterdark          : Creates a master dark image from a set of raw dark frames.
    gimasterflat          : Create the fiber master flat field and the localization mask.
    giscience             : Process a science observation.
    gistandard            : Process a spectro-photometric standard star observation and compute the instrument response curve.
    giwavecalibration     : Compute dispersion solution from an arc-lamp spectrum.
    helloworld            : PyCPL version of 'hello, world!'

Now the *PyCPL* recipe is listed among all other available recipes in the
search path. To actually run the recipe the same steps as previously shown
have to be executed: loading the recipe, set the input set-of-frames and
eventually call the recipe. 

.. code:: python

  >>> p.recipe = "helloworld"
  [ INFO  ] pyesorex: Loaded recipe 'helloworld'.
  [WARNING] pyesorex: Recipe 'helloworld' has no parameters attribute, cannot access recipe settings or use recipe config files.
  >>> p.recipe
  <cpl.ui.Recipe helloworld>
  >>> p.sof_location = "bias.sof"
  >>> result = p.run()
  [ INFO  ] pyesorex: Running recipe 'helloworld'...
  Hello, /home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits!
  Hello, /home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits!
  Hello, /home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits!
  Hello, /home/dummy/test/bias/GIRAF.2019-06-25T10:03:05.335.fits!
  Hello, /home/dummy/test/bias/GIRAF.2019-06-25T10:08:15.239.fits!
  [ INFO  ] pyesorex: Recipe 'helloworld' complete.
  [ INFO  ] pyesorex: Calculating product checksums
  [ INFO  ] pyesorex: 0 products created
  [ INFO  ] pyesorex: Recipe operation(s) took 0.00 seconds to complete.
  [ INFO  ] pyesorex: Total size of 5 raw input frames = 88.96 MB
  [ INFO  ] pyesorex: => processing rate of 35945.34 MB/s

The warning can be ignored, since we did not add any parameters to this
recipe. Also, since the recipe does not create any product files the
returned set-of-frames ``result`` is of course empty. How this can be
changed is shown in the following.

A Recipe with Parameters
........................

A recipe which cannot be configured by the user, by either passing a command
line option to the *PyEsoRex* command line tool, or setting a parameter value
through the *PyEsoRex* API, is hardly useful. Therefore, in this next step, we
will add parameters to our recipe.

This example will greet the user with the name the user has specified. The
message that is printed will also show a reaction that can be set by the user,
and the user can select the number of input files being listed.

.. include:: examples/pycpl_recipe_hellouser.py
  :code: python

Again, parameters of *PyCPL* work in the same way as for native CPL recipes:

.. code:: python

  >>> p.recipe = "hellouser"
  [ INFO  ] pyesorex: Loaded recipe 'hellouser'.
  >>> for parameter in p.recipe_parameters: print(f"{parameter.name}={parameter.value}")
  ... 
  hellouser.username=Unknown user
  hellouser.nfiles=3
  hellouser.reaction=nice

  >>> p.recipe_parameters.update({"hellouser.username": "World"})
  >>> for parameter in p.recipe_parameters: print(f"{parameter.name}={parameter.value}")
  ... 
  hellouser.username=World
  hellouser.nfiles=3
  hellouser.reaction=nice

  >>> result = p.run()
  [ INFO  ] pyesorex: Running recipe 'hellouser'...
  [ INFO  ] hellouser: Hello, World, nice to see you.
  [ INFO  ] hellouser: The first 3 input files are:
  [ INFO  ] hellouser: /home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits
  [ INFO  ] hellouser: /home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits
  [ INFO  ] hellouser: /home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits
  [ INFO  ] pyesorex: Recipe 'hellouser' complete.
  [ INFO  ] pyesorex: Calculating product checksums
  [ INFO  ] pyesorex: 0 products created
  [ INFO  ] pyesorex: Recipe operation(s) took 0.00 seconds to complete.
  [ INFO  ] pyesorex: Total size of 5 raw input frames = 88.96 MB
  [ INFO  ] pyesorex: => processing rate of 30217.63 MB/s

.. _userguide.quickstart.complete-recipe:

A Complete Recipe
.................

While the first two examples are basically showing the anatomy of a *PyCPL*
recipe, this example will showcase a bit of actual data processing using the
*PyCPL* API. While still being basic, this recipe example contains everything
that is needed to do data processing with *PyCPL*.

While this example demonstrates some error handling, please note that it is
neither complete, nor should it be used as a template. It is heavily
simplified and its purpose is only to showcase different parts of the
*PyCPL* API.

.. include:: examples/pycpl_recipe_basic_science.py
  :code: python

Running the recipe will produce an output file ``OBJECT_REDUCED.fits`` in the
current working directory:

.. code:: python

  >>> p.recipe = "basic_science"
  [ INFO  ] pyesorex: Loaded recipe 'basic_science'.
  >>> p.sof_location = "object.sof"
  >>> result = p.run()
  [ INFO  ] pyesorex: Running recipe 'basic_science'...
  [ INFO  ] basic_science: Loaded bias frame '/home/dummy/test/fors/FORS1.2003-05-28T11:12:17.762.fits'.
  [ INFO  ] basic_science: Loaded flat frame '/home/dummy/test/fors/FORS1.2003-05-28T10:53:44.489.fits'.
  [ INFO  ] basic_science: Preparing flat field
  [ INFO  ] basic_science: Processing '/home/dummy/test/fors/FORS1.2003-06-02T07:26:47.725.fits'...
  [ INFO  ] basic_science: Processing '/home/dummy/test/fors/FORS1.2003-06-02T07:26:47.725.fits'...
  [ INFO  ] basic_science: Processing '/home/dummy/test/fors/FORS1.2003-06-02T07:26:47.725.fits'...
  [ INFO  ] basic_science: Combining images using method 'add'
  [ INFO  ] basic_science: Saving product file as 'OBJECT_REDUCED.fits'.
  [ INFO  ] basic_science: Writing FITS image product(OBJECT_REDUCED): OBJECT_REDUCED.fits
  [ INFO  ] pyesorex: Recipe 'basic_science' complete.
  [ INFO  ] pyesorex: Calculating product checksums
  [ INFO  ] pyesorex: Created product OBJECT_REDUCED.fits (in place)
  [ INFO  ] pyesorex: 1 products created
  [ INFO  ] pyesorex: Recipe operation(s) took 0.49 seconds to complete.
  [ INFO  ] pyesorex: Total size of 5 raw input frames = 42.74 MB
  [ INFO  ] pyesorex: => processing rate of 87.17 MB/s

This example shows how simple data products, in this case a single image data
product, can be created. A modification of this recipe demonstrating how
complex data products can be assembled can be found in the section
:ref:`userguide.quickstart.complex-data-products`.

A few things to be aware of
...........................

- When writing recipes in Python, please be aware that when you modify your code in an editor and save
  it the modified code is not executed unless you reload the recipe (which will compile the modified code).
  There is no reload command or function in *PyCPL*. To reload a python recipe you simply have to reassign
  it to the ``recipe`` property of your ``Pyesorex`` instance.

  .. code:: python

    >>> from pyesorex.pyesorex import Pyesorex
    >>> p = Pyesorex()
    >>> p.recipe = "helloworld" # initial recipe code is loaded and compiled
    >>> # Code changes happen here!
    >>> p.recipe = "helloworld" # modified code is loaded and compiled

Using PyCPL with Other Packages
-------------------------------

Visualizing Data
................

*PyCPL* ``Image`` objects can be passed to functions that expect numpy arrays,
e.g. matplotlib plotting functions. As an example the following displays the
raw image used for the previous run of our ``basic_science`` recipe:

.. code:: python
  
  >>> import numpy as np
  >>> from matplotlib import pyplot as plt
  >>> from astropy.visualization import PercentileInterval, ImageNormalize
  >>> from cpl.ui import FrameSet
  >>> from cpl.core import Image
  >>> sof_frames = FrameSet("object.sof")
  >>> raw_frame = sof_frames[0]
  >>> raw_image = Image.load(raw_frame.file)
  >>> plt.rcParams['figure.figsize'] = 12, 8
  >>> norm = ImageNormalize(raw_image, interval=PercentileInterval(90))
  >>> plt.imshow(raw_image, interpolation='nearest', cmap='hot', norm=norm)
  <matplotlib.image.AxesImage object at 0x7fb18c329b70>
  >>> plt.title(raw_frame.file)
  Text(0.5, 1.0, '/home/dummy/test/fors/FORS1.2003-06-02T07:26:47.725.fits')
  >>> plt.colorbar()
  <matplotlib.colorbar.Colorbar object at 0x7fb18c329f00>
  >>> plt.show()

.. image:: images/raw_image.png
  :width: 1200
  :align: center
  :alt: Raw data image

And the same for the created result image. Since the recipe returned the
``cpl.ui.FrameSet`` object containing the processed image frame, one can
just use it directly:

.. code:: python

  >>> result_frame = result[0]
  >>> result_image = Image.load(result_frame.file)
  >>> norm = ImageNormalize(result_image, interval=PercentileInterval(90))
  >>> plt.imshow(result_image, interpolation='nearest', cmap='hot', norm=norm)
  <matplotlib.image.AxesImage object at 0x7fb18c3a4a90>
  >>> plt.title(result_frame.file)
  Text(0.5, 1.0, '/home/dummy/test/OBJECT_REDUCED.fits')
  >>> plt.colorbar()
  <matplotlib.colorbar.Colorbar object at 0x29c64b1c0>
  >>> plt.show()

.. image:: images/result_image.png
  :width: 1200
  :align: center
  :alt: Processed image

Exchanging data with astropy
............................

Converting Frames to an HDU list
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``cpl.ui.Frame`` objects can open the corresponding FITS file as an ``astropy.io.fits.HDUList`` object.

.. code:: python

  >>> sof_frames = FrameSet("object.sof")
  >>> raw_hdulist = sof_frames[0].as_hdulist()
  >>> raw_hdulist
  [<astropy.io.fits.hdu.image.PrimaryHDU object at 0x134d2a9c0>]
  >>> raw_hdulist[0].header

.. include:: outputs/pycpl_astropy_hdulist.txt
  :code: shell

.. code:: python

  >>> raw_hdulist[0].data
  array([[249, 245, 244, ..., 236, 235, 236],
        [240, 241, 248, ..., 235, 235, 236],
        [242, 249, 245, ..., 233, 241, 237],
        ...,
        [248, 248, 250, ..., 253, 249, 248],
        [243, 246, 243, ..., 251, 248, 246],
        [244, 247, 250, ..., 250, 246, 251]], dtype=uint16)

Converting the Frames to CCDData
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

``cpl.ui.Frame`` can also open the corresponding FITS file as an ``astropy.nddata.CCDData object``.

.. code:: python

  >>> raw_ccddata = sof_frames[0].as_ccddata(unit="adu")
  WARNING: FITSFixedWarning: RADECSYS= 'FK5 ' / Coordinate reference frame 
  the RADECSYS keyword is deprecated, use RADESYSa. [astropy.wcs.wcs]
  WARNING: FITSFixedWarning: PC001001= 1.00000 / Translation matrix element 
  this form of the PCi_ja keyword is deprecated, use PCi_ja. [astropy.wcs.wcs]
  WARNING: FITSFixedWarning: PC001002= 0.00000 / Translation matrix element 
  this form of the PCi_ja keyword is deprecated, use PCi_ja. [astropy.wcs.wcs]
  WARNING: FITSFixedWarning: PC002001= -0.00000 / Translation matrix element 
  this form of the PCi_ja keyword is deprecated, use PCi_ja. [astropy.wcs.wcs]
  WARNING: FITSFixedWarning: PC002002= 1.00000 / Translation matrix element 
  this form of the PCi_ja keyword is deprecated, use PCi_ja. [astropy.wcs.wcs]
  >>> raw_ccddata
  CCDData([[249, 245, 244, ..., 236, 235, 236],
          [240, 241, 248, ..., 235, 235, 236],
          [242, 249, 245, ..., 233, 241, 237],
          ...,
          [248, 248, 250, ..., 253, 249, 248],
          [243, 246, 243, ..., 251, 248, 246],
          [244, 247, 250, ..., 250, 246, 251]], unit='adu')
