:tocdepth: 3

.. _VLT Instrument Pipelines: https://www.eso.org/sci/software/pipelines

.. _userguide.installation.installation-of-pycpl:

Installing PyCPL
================


.. _userguide.installation.installation-from-prebuild-packages:

Installation from Pre-Built Packages
------------------------------------

Pre-built, RPM packages for Fedora 40 -- 43 and MacPorts packages for
macOS 12 and later are available from the respective *ESO repositories*.

.. note:: 
    This installation method aims at using *PyCPL* and *PyEsoRex* in a production
    environment, i.e. using it for data processing. For developing *PyCPL* pipeline
    recipes, additional software tools may be required. For example, support for
    unit testing and generating documentation. These need to be installed separately
    depending on the requirements and conventions of the individual development project.


.. _userguide.installation.installation-from-rpm:

RPM Package Installation 
........................

Installing *PyCPL* and *PyEsoRex* using the RPM packages will utilize the default
Python 3 interpreter of the system. In addition to *PyCPL* and *PyEsoRex* itself all
dependencies needed to run pipeline recipes will be installed automatically too.

*Step 1:*
  In case the *ESO RPM repository* is not yet configured on the machine where *PyCPL* and
  *PyEsoRex* will be installed, then please follow the
  `ESO RPM repository configuration instructions <https://www.eso.org/sci/software/pipelines/installation/rpm.html>`__ to set
  up the repository.

*Step 2:*
  With the *ESO repository* configured, a system wide installation of *PyCPL* and *PyEsoRex*
  can be set up running the following command (requires sudo or root privileges) in a terminal:

  .. code:: shell

    $ sudo dnf install python3-pycpl python3-pyesorex


.. _userguide.installation.installation-from-macports:

MacPorts Package Installation 
.............................

Installing *PyCPL* and *PyEsoRex* using MacPorts will use a suitable Python 3 interpreter from
MacPorts. Both packages are available for a variety of Python 3 versions which can be installed
explicitly. Alternativley, if no specific version is requested, the active default Python 3
version will be used. Installing *PyCPL* and *PyEsoRex* from MacPorts will also install all
required dependencies.

*Step 1:*
  In case the *ESO MacPorts repository* is not yet configured on the machine where *PyCPL* and
  *PyEsoRex* will be installed, then please follow the 
  `ESO MacPorts repository configuration instructions <https://www.eso.org/sci/software/pipelines/installation/macports.html>`__ to set
  up the repository.

*Step 2:*
  With the *ESO Repository* configured, a MacPorts installation of *PyCPL* and *PyEsoRex* can be
  set up running the following command (requires sudo privileges) in a terminal:

  .. code:: shell

    $ sudo port install py312-pyesorex

  This will install *PyCPL* and *PyEsoRex* for Python 3.12. It is possible to install *PyCPL* and
  *PyEsoRex* for multiple Python 3 versions if needed.

  An installation for the active default Python 3 version is done by installing the generic package, 
  which does not have the encoded Python version in its name:
  
  .. code:: shell

    $ sudo port install py-pyesorex

  This will install the packages either for the MacPorts default Python 3 version, or for the
  Python version which was previously selected as the default using the MacPorts :literal:`port select`
  command.

*Step 3: (optional)*
  After *Step 2* is finished, *PyEsoRex* can for example be called using the :literal:`pyesorex-3.12`
  command. Of course, the version suffix may vary depending on the Python 3 version *PyEsoRex* was
  actually installed for.
  
  To be able to call *PyEsoRex* without a version suffix simply as :literal:`pyesorex`,
  an installed version of *PyEsoRex* can be set as default. For example, to make the Python
  3.12 version of *PyEsoRex* the default, run:

  .. code:: shell

    $ sudo port select --set pyesorex py312-pyesorex
 
  
.. _userguide.installation.installation-from-sources:

Installation from Sources
-------------------------

While a system wide installation of *PyCPL* is supported, the following
installation instructions will show an installation using a Python virtual
environment. It is expected that this is a more common approach followed by
individual users when installing python packages.

It is however assumed that the dependencies needed to build and use *PyCPL* can be
installed using the system's package management system or by installing an ESO
instrument pipeline source kit.


.. _userguide.installation.software-prerequisites:

Software Prerequisites
......................

In order to build, install and use *PyCPL* the following software packages, with the
given *minimum* version (if indicated), is needed:

    - C++ compiler supporting C++17 (GCC 9.5, Clang 5)
    
    - CMake 3.12
    
    - Python 3.10 (including the development package, i.e. headers and libraries)

    - Python packages:

      - setuptools 45.0.0
    
      - pybind11 2.8.0
    
      - astropy
    
      - pandas (optional)
    
      - Sphinx (optional, for generating the API documentation)
    
      - pytest (optional, for running the testsuite)

      - scipy (optional, for running the testsuite)

    - CPL 7.4

Depending on whether *PyCPL* will be installed system wide, local to the user, or in
a Python virtual environment (or alike) the required packages must be available in
the respective environment. While the compiler, Python itself and CPL are generally 
expected to be provided by the system, or possibly the user's local environment. CMake
and the required Python packages however may be installed into the target virtual
environment only, in case a virtual environment approach is chosen.


.. _userguide.installation.getting-the-source-code:

Getting the Source Code
.......................

The most recent version of the *PyCPL* and *PyEsoRex* source distributions is available here:

- `PyCPL <https://ftp.eso.org/pub/dfs/pipelines/libraries/pycpl/pycpl-1.0.4.tar.gz>`__ :bdg-link-secondary-line:`SHA-256 <https://ftp.eso.org/pub/dfs/pipelines/libraries/pycpl/pycpl-1.0.4.sha256>`
- `PyEsoRex <https://ftp.eso.org/pub/dfs/pipelines/libraries/pyesorex/pyesorex-1.0.4.tar.gz>`__ :bdg-link-secondary-line:`SHA-256 <https://ftp.eso.org/pub/dfs/pipelines/libraries/pyesorex/pyesorex-1.0.4.sha256>`

Older releases of *PyCPL* and *PyEsoRex* can be found in the
`PyCPL archive <https://ftp.eso.org/pub/dfs/pipelines/libraries/pycpl>`_ and
the `PyEsoRex archive <https://ftp.eso.org/pub/dfs/pipelines/libraries/pyesorex>`_.

Users with access to the ESO Gitlab may also clone the *PyCPL* and *PyEsoRex*
repositories directly (**ESO internal access only**):

    .. code:: shell

        $ git clone https://gitlab.eso.org/pipelines/pycpl
        $ git clone https://gitlab.eso.org/pipelines/pyesorex

.. _userguide.installation.installing-tools:

Installing Development Tools 
............................

The following shows how to install the standard dependencies for an ESO pipeline source
installation. These dependencies are also needed to build *PyCPL*. These dependencies
can be installed using the package manager of the different systems. Please skip to the
section that applies to your environment.

**Fedora 36 and newer:**
  .. code:: shell

    $ sudo dnf install gcc gcc-c++ zlib-devel make gzip bzip2 tar perl gawk sed grep coreutils pkgconfig
    $ sudo dnf install cmake python3-devel

**Ubuntu 20.04 and newer:**
  .. code:: shell

    $ sudo apt install gcc g++ zlib1g-dev make gzip bzip2 tar perl gawk sed grep coreutils pkg-config
    $ sudo apt install cmake python3-dev python3-venv

**macOS 12 and newer:**
  .. code:: shell
 
    $ sudo port install pkgconfig autoconf automake libtool bzip2
    $ sudo port install cmake python312

  To set Python 3.12 as the default Python version to use on macOS, run

  .. code:: shell

    $ sudo port select --set python python312
    $ sudo port select --set python3 python312


.. _userguide.installation.installing-cpl:

Installing CPL
..............

There are several ways to install CPL. Depending on the use case one may choose
one way or the other. If CPL is already installed on your system, because you have
already installed an ESO instrument pipeline for instance, it is recommended that
you use this CPL installation for building *PyCPL*, provided that the existing
CPL installation includes the CPL header files, and meets the *PyCPL* requirements.

.. attention:: Regardless of which of the following methods you use to install CPL,
  the root directory of the CPL installation will later be needed to correctly
  build *PyCPL*.


.. _userguide.installation.installing-cpl-from-repositories:

Installing CPL from the ESO repositories
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Binary packages of CPL are available from the ESO repositories for a number
of different platforms. The information for which platforms binary packages
of CPL are currently available can be found on the `VLT Instrument Pipelines`_
page.

Detailed instruction on how to configure and use the ESO repositories are
available at the pipeline webpage for
`Linux (RPM) <http://www.eso.org/sci/software/pipelines/installation/rpm.html>`__
and `macOS <http://www.eso.org/sci/software/pipelines/installation/macports.html>`__
systems.

With the repositories configured you will have to install the CPL packages:

**Fedora 36 and newer:**

.. code:: shell

  $ sudo dnf install cpl-devel

**macOS 12 and newer:**

.. code:: shell

  $ sudo port install cpl

This will install CPL, including the required CPL headers, into ``/usr``
on Linux systems, and into ``/opt/local`` on macOS systems.

All pipeline packages which are available from these repositories can be used
together with *PyCPL*.


.. _userguide.installation.installing-cpl-from-kit:

Installing CPL using a Pipeline Source Kit
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When installing an ESO instrument pipeline using one of the pipeline source
kits which are available from the `VLT Instrument Pipelines`_ page, the CPL
which is part of the source kit will be automatically installed.

For use with *PyCPL* the source kit maybe be installed using either the `pipeline
installer script <https://www.eso.org/sci/software/pipelines/install_pipelinekit>`__,
or the `Reflex installer script <https://ftp.eso.org/pub/dfs/reflex/install_esoreflex>`__.
Instructions on how to install an ESO pipeline source kit using any of the two
installer scripts can be found on the `VLT Instrument Pipelines`_ page.

The list of available pipeline source kits can be found `here
<https://www.eso.org/sci/software/pipelines/#pipelines_table>`__. All kits
shown in this table provide a CPL which meets the *PyCPL* requirements.

Using this method CPL will be installed into the directory that is given when
the installer scripts ask for an installation directory.

.. _userguide.installation.installing-cpl-from-sources:

Installing CPL from Sources
^^^^^^^^^^^^^^^^^^^^^^^^^^^

It is also possible to install CPL using the CPL source distribution. This is the
recommended way for developers working on an instrument pipeline project, or 
expert users who are familiar with the CPL build system and the needed developer
tools. Using this method should hardly be necessary for a normal user.

To install CPL, and possibly its dependencies, using their respective source
distributions, please refer to the installation instructions coming with the
individual source packages. The installation instructions of CPL are included in
the source package, which is available from the `CPL download page
<http://www.eso.org/sci/software/cpl/download.html>`__

Choosing this method to install CPL gives you complete freedom where CPL, and possibly
its dependencies, will be installed. It is however up to you to know where to point the
*PyCPL* build system so that it finds the CPL installation. 


.. _userguide.installation.create-python-environment:

Creating a Python Virtual Environment
.....................................

To prepare for building and installing *PyCPL* a Python virtual environment
may be created. To do so choose a location where the virtual environment is
going to be installed. If you have used virtual environments before, you may
have such a location already and it may be used to host the Python virtual
environment for *PyCPL* as well.

In the following ``<venv-root-dir>`` will have to be replaced by the path you have chosen
as the installation directory of the virtual environment.

.. code:: shell

  $ python3 -m venv <venv-root-dir>/pycpl


.. _userguide.installation.installing-pycpl-pyesorex:

Installing PyCPL and PyEsoRex
.............................

Once the required dependencies are installed the *PyCPL* and the *PyEsoRex*
source distributions can be installed. 

.. _userguide.installation.preparing-the-environment:

Preparing the Environment
^^^^^^^^^^^^^^^^^^^^^^^^^

The first step is to setup the environment so that *PyCPL* uses the correct CPL
installation. In order to point the *PyCPL* build system to the appropriate
CPL installation the environmet variable ``CPLDIR`` has to be set to the
root directory of the CPL installation. For instance, if the CPL libraries
are installed in the directory ``/usr/lib``, then ``CPLDIR`` must be set to
``/usr``. Hints on where CPL might be installed are given in the Section
:ref:`userguide.installation.installing-cpl`.

.. code:: shell

  $ export CPLDIR=<cpl-root-dir>

.. note:: If in doubt where CPL is installed, try to find the file ``libcplcore.so``
  or ``libcplcore.dylib`` depending whether you use a Linux system or a macOS system,
  respectively.

Make sure the libraries from the CPL installation located in ``CPLDIR`` are preferred
when looking for libraries. This is particularly important if you have more than one CPL
installation on your system, but it is also recommended if there is only a single CPL
installation. 

This is done by adding the CPL library directory to the beginning of the library search
path. The environment variable names differ, depending on the OS you are using:

**Linux**:

.. code:: shell
  
  $ export LD_LIBRARY_PATH=$CPLDIR/lib64:$CPLDIR/lib:$LD_LIBRARY_PATH

**macOS**:

.. code:: shell

    $ export DYLD_LIBRARY_PATH=$CPLDIR/lib:$DYLD_LIBRARY_PATH

.. note:: If you exit from your shell session and start a new one later you will
   have to set these previously defined environment variables again before you can
   continue with the installtion.
   
   The same applies if you are using *PyCPL* or *PyEsoRex* in a newly started shell
   environment.

The next step is to activate the Python virtual environment:

.. code:: shell

  $ . <venv-root-dir>/pycpl/bin/activate

At this point it is recommended to update *pip*, *setuptools*, *wheel*
and *build*, in particular if your virtual environment was created from
a somewhat older Python installation:

.. code:: shell

  (pycpl)$ python3 -m pip install --upgrade pip setuptools build wheel

As a last step before actually building *PyCPL* it may be necessary to change the
default *PyCPL* recipe search path. By default *PyCPL* expects recipes to be found
in a subdirectory of the CPL installation. This default may however not be suitable,
for instance if CPL is installed into a system location where a normal user is usually
not able to put his own recipes.

.. attention:: If the default recipe directory does not exist, a warning will be
  issued when *PyCPL* is built and when the Python module is imported. Usually
  this default recipe location does **not** exist if CPL is installed stand-alone,
  i.e. not as part of a pipeline source kit installation.

This default location can be changed when building *PyCPL* using the environment
variable ``PYCPL_RECIPE_DIR`` and point it to an appropriate, colon separated list
of directories.

Typical locations where recipes are installed may depend on the method used
to install pipeline packages, and the operating system. For instance, using the
package managers to install pipeline packages from the ESO repositories will
install recipes into ``/usr/lib64/esopipes-plugins`` and
``/opt/local/lib/esopipes-plugins`` on 64-bit Fedora systems and macOS, respectively.
If pipeline packages are installed using a source kit the recipe location will be
``<prefix>/lib/esopipes-plugins``, where ``prefix`` is the root directory of the
installation given by the user during the installation process. 

.. note:: It is of course always possible to change this default recipe
  location later, at runtime by changing the recipe search path from the
  Python interpreter prompt, a script, or through a *PyEsoRex* command
  line option. Setting ``PYCPL_RECIPE_DIR`` will only change the built-in
  default. However, this path will persist as a default setting until
  *PyCPL* is eventually re-compiled. Thus, reasonable locations should
  be chosen.

Assuming that all pipeline recipes will be installed locally in a user account
into the **existing** location ``$HOME/pipelines/lib/esopipes-plugins`` the
default *PyCPL* recipe location can be changed setting the variable like this:

.. code:: shell

  (pycpl)$ export PYCPL_RECIPE_DIR=$HOME/pipelines/lib/esopipes-plugins

The environment variable ``PYCPL_RECIPE_DIR`` is only used for building *PyCPL*. 
Once the installation of *PyCPL* is finished, the variable may be unset again.

.. _userguide.installation.building-pycpl-pyesorex:

Building PyCPL and PyEsoRex
^^^^^^^^^^^^^^^^^^^^^^^^^^^

With the environment set *PyCPL* and *PyEsoRex* can be build and installed from
the source distribution packages in the one of the following ways:

*Option 1:*
  The easiest way to install the latest available version of *PyCPL* and *PyEsoRex*, without the
  need for downloading the source packages first, you can use the following command:

  .. code:: shell

    (pycpl)$ python3 -m pip install --extra-index-url https://ftp.eso.org/pub/dfs/pipelines/libraries pycpl pyesorex

*Option 2:*
  To install *PyCPL* and *PyEsoRex* after getting the source distributions from the locations
  given in :ref:`userguide.installation.getting-the-source-code` and placing them into a
  temporary directory ``<tmpdir>``, use the command:

  .. code:: shell

    (pycpl)$ python3 -m pip install <tmpdir>/pycpl-1.0.4.tar.gz <tmpdir>/pyesorex-1.0.4.tar.gz

*Option 3:*
  Installing *PyCPL* or *PyEsoRex* from a working copy of their respective
  GitLab repository, or a directory containing the unpacked source distribution
  package of *PyCPL* or *PyEsoRex*, the following needs to be done. For instance
  in the case of *PyCPL*:

  .. code:: shell

    (pycpl)$ cd <tmpdir>/pycpl-1.0.4
    (pycpl)$ python3 -m pip install .

At this point the installation of *PyCPL* and *PyEsoRex* is finished and you may unset the
environment variable ``PYCPL_RECIPE_DIR`` as it is no longer needed:

.. code:: shell

  (pycpl)$ unset PYCPL_RECIPE_DIR

.. _userguide.installation.verifying-pycpl-pesorex:

Verifying the Installation
^^^^^^^^^^^^^^^^^^^^^^^^^^

Finally verify that the installation actually succeeded:

.. code:: shell

  (pycpl)$ cd <tmpdir>
  (pycpl)$ python3

From the Python interpreter prompt import the ``cpl`` and the
``pyesorex`` modules:

.. code:: python

  >>> import cpl
  >>> import pyesorex

If this succeeds without any messages printed to the terminal then you have
successfully installed *PyCPL* and *PyEsoRex* API!

To verify that the *PyEsoRex* CLI is correctly installed:

.. code:: shell

  (pycpl)$ pyesorex --version

or 

.. code:: shell

  (pycpl)$ pyesorex --help
