.. _metadaqif:

Introduction
############

Acronyms
========

OCM
   Observation Coordination Manager

DPM
   Data Product Manager

Definitions
===========

.. glossary::

    Commentary Keyword
        In this document a FITS Commentary Keyword refers to those keywords that are neither
        :term:`Value Keyword`\ s or :term:`ESO Keyword`\ s. Commentary Keywords are special because
        the same keyword name may occur multiple times in the same HDU, to e.g. continue a comment
        over multiple records.

        .. note::

            Strictly speaking the ``HIERARCH ESO`` keyword is also a commentary keyword but is
            treated specially in applications that support the convention.

        Examples are::

            COMMENT  Commentary keyword names may occur multiple times in a header, it may also
            COMMENT  be contextual, as in this case, where a comment is continued over multiple
            COMMENT  records.
            HISTORY  File modified by user 'USER' on host on 2021-09-03T01:28:26

    ESO Keyword
    ESO Hierarch Keyword
        Refers to FITS keywords following the ESO HIERARCH keyword conventions |rd-esokw|, i.e.
        keywords of the form::

            HIERARCH ESO INS FILT1 ENC = 2 / Filter wheel absolute position [Enc].
            HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.

        The first token, ``INS`` in the example above, is referred to as the *category*.

    Logical Keyword Name
        The logical name of a FITS keyword depends on the type of keyword, but there are common
        traits: Trailing white spaces are insignificant and are not part of the Logical Keyword
        Name.

        For FITS :term:`Value Keyword` and :term:`Commentary Keyword` the logical name is the
        white-space trimmed name component. The logical names for

        .. code-block::

            NAXIS1  =             2048 / # of pixels in axis1
            COMMENT This table was written by 'APPLICATION'

        are ``NAXIS1`` and ``COMMENT`` respectively.

        For :term:`ESO Keyword` the logical name is the part between ``HIERARCH ESO`` up to ``=``.
        For example the logical name is ``INS FILT1 ID`` for the HIERARCH keyword::

            HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.

    Value Keyword
        A FITS Value Keyword are those keywords that have a value indicator in bytes 9 and
        10 (c.f. |rd-fits|). Example keywords are::

            SIMPLE  =                T / Standard FITS
            BITPIX  =                8 / # of bits per pix value
            NAXIS   =                0 / # of axes in data array

Reference Documents
===================

.. _rd1:

[RD1]
    | Central Control System Development Standards;
    | `ESO-366378 v1 <https://pdm.eso.org/kronodoc/HQ/ESO-366378>`_

.. _rd2:

[RD2]
    | Data Interface Control Document;
    | `ESO-044156 v6 <https://pdm.eso.org/kronodoc/HQ/ESO-044156/6>`_


.. _rd3:

[RD3]
    | Definition of the Flexible Image Transport System (FITS);
    | `<https://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf>`_

.. _rd4:

[RD4]
    | The ESO HIERARCH Keyword Conventions;
    | `<https://fits.gsfc.nasa.gov/registry/hierarch/hierarch.pdf>`_


Metadata Acquisition Interface
##############################

Metadata acquisition interface, ``metadaqif``, is a relatively small interface used by components
that provide metadata to FITS data products created by OCM and DPM.

The following sections document the interface in a language agnostic manner. For data structures
only the data members are documented, not the accessors generated by MAL. Similarly names are not
fully qualified as it is different across the supported languages.


.. module:: metadaqif

Interfaces
==========

.. class:: MetaDaq

    Metadata acquisition interface that supports acquiring arbitrary data to either FITS files
    and/or FITS keywords.

    Interface supports concurrent acquisitions which is why the |daq| id is provided in requests and
    replies as a *handle*.

    .. note::

        It is out of scope for this interface to provide behaviour modification of the application
        performing the acquisition. This is domain specific and needs to be provided by other means
        such as configuration or another MAL interface.

    **Success**

    A successful sequence looks like follows:

    #. :py:meth:`StartDaq` initiates a new acquisition, which if successful returns
       :py:obj:`DaqReply` as acknowledgement. State is now :py:attr:`DaqState.Acquiring` which
       indicates to clients that |daq| has started and is in progress.

       It is not specified or required that data must be physically acquired from this point. An
       implementation may use another form of synchronization to determine exactly how the
       acquisition is performed (see :py:attr:`DaqState.Acquiring`).

    #. :py:meth:`StopDaq` is issued at the time the |daq| should stop. FITS files in progress of
       being produced are closed and placed in a *stable* predetermined location for later
       retrieval. Only after files have been closed and keyword produced is a reply sent. Again if
       another form of synchronization is used it is anyway expected that that reply is sent only
       when |daq| has completed.

    **Aborting**

    If something happens that results in that the in-progress acquisition should be aborted and data
    discarded the sequence looks like:

    #. :py:meth:`StartDaq` (see above).
    #. :py:meth:`AbortDaq` |daq| is stopped and any acquired data can be discarded.

    **Error Recovery**

    If there is a communication problem or something else happens that the current state of a |daq|
    is lost clients may use :py:meth:`GetDaqStatus` to query for current status and recover from
    there.


    .. method:: StartDaq(id) -> DaqReply

        Start new |daq| that is eventually stopped with :py:meth:`~MetaDaq.StopDaq` or aborted with
        :py:meth:`~MetaDaq.AbortDaq`.

        .. note::

            If *id* is not provided (left empty) it is expected that the server implementing the
            interface will generate a unique identifier automatically.

        If the same *id* is reused it should be considered a fatal error.

        :param str id: Optional (may be empty) unique identifier of |daq|.
        :return: If *id* was provided it is returned as an acknowledgement, otherwise the *id*
            generated by server is returned.
        :rtype: DaqReply
        :raises DaqException: On fatal error.

    .. method:: StopDaq(id) -> DaqStopReply

        Stops data acquisition and returns created FITS filenames and/or keywords. Produced files
        should be closed so it is safe to immediately read them.

        If an error occurred such that the acquisition has failed this is communicated by throwing
        an exception.

        :param str id: Id of data acquisition to stop.
        :return: Structure containing produced FITS files and/or FITS keywords.
        :rtype: DaqStopReply
        :raises DaqException: On fatal error.

    .. method:: AbortDaq(id) -> DaqReply

        Aborts data acquisition and discards any data that has been acquired.

        :param str id: Id of data acquisition to abort.
        :rtype: DaqReply
        :raises DaqException: On fatal error.

    .. method:: GetDaqStatus(id) -> DaqStatus

        Get status of current or past *Data Acquisitions*.

        It is unspecified exactly how far back the history should go. But the bare minimum is to be
        able to provide status for last two (e.g. any current and previous).

        :param str id: Id of data acquisition to get status for.
        :rtype: DaqStatus
        :raises DaqException: On fatal error.


Data Structures
===============

.. class:: DaqState

    Enumeration of data acquisition states. The expected state transitions are as follows:

    .. graphviz::
        :align: center

        digraph DaqStates {
            # Config
            node [shape=Mrecord,fontname=helvetica,fontsize=11];
            graph [fontname = "helvetica", bgcolor=transparent];

            # States
            NotStarted [label="{NotStarted|\l}"];
            Acquiring [label="{Acquiring|\l}"];
            Succeeded [label="{Succeeded|\l}"];
            Aborted [label="{Aborted|\l}"];
            Failed [label="{Failed|\l}"];

            # Transitions
            NotStarted -> Acquiring;
            Acquiring -> Succeeded;

            edge [weight=0];
            Acquiring -> {Aborted, Failed};
        }

    .. note::

        There may be more internal (transitional states) but this enumeration covers the ones that
        are observable with metadaqif.

    .. attribute:: NotStarted

        |daq| is created but not yet started.

    .. attribute:: Acquiring

        Logical state where data is acquired. It does not imply that at each time point data is
        physically being acquired. Implementations can choose to e.g.:

        - Synchronize to an implementation specific event or events, such as a sequence number or
          time point.
        - Sample at specific intervals.
        - Sample once at the beginning of a |daq| when :py:meth:`MetaDaq.StartDaq` is first received
          and then again at the end when :py:meth:`MetaDaq.StopDaq` is received.
        - Continuously acquire time series until requested to stop.

       Configuring the implementation specific behaviour is out of scope for :py:mod:`metadaqif`.

    .. attribute:: Succeeded

        Final state for a successful |daq|. All data has been acquired and any FITS files are
        completed and closed.

    .. attribute:: Aborted

        Final state for an aborted |daq|.

    .. attribute:: Failed

        Final state for failed |daq|.


.. class:: DaqException

    Exception used by MetaDaq.

    .. attribute:: id
        :type: str

        |daq| identifier.

    .. attribute:: message
        :type: str

        Exception message.


.. class:: DaqStatus

    Contains the |daq| reply.

    .. attribute:: id
        :type: str

        |daq| identifier.

    .. attribute:: state
        :type: DaqState

        |daq| state at the time the reply was sent.

    .. attribute:: message
        :type: str

        Message, if any.

    .. attribute:: files
        :type: List[str]

        List of FITS files created for this |daq| in the format ``[user@]host:/absolute/path``.

    .. attribute:: keywords
        :type: str

        JSON-encoded FITS keywords, or empty. See :ref:`json-keywords` for more details.

        Example:

        .. code-block:: json

            [
               {
                  "type":"valueKeyword",
                  "name":"OBJECT",
                  "value":"OBJECT,SKY"
               },
               {
                  "type":"esoKeyword",
                  "name":"OBS TPLNO",
                  "value":2
               }
            ]

    .. attribute:: timestamp
        :type: double

        Timestamp of last status update in number of seconds in TAI time standard with epoch set to
        1 January 1970 00:00:00 TAI, which is 31 December 1969 23:59:51.999918 UTC |rd1|.


.. class:: DaqReply

    Common reply type.

    .. attribute:: id
        :type: str

        |daq| identifier.


.. class:: DaqStopReply

    Reply structure for DaqStop.

    .. attribute:: id
        :type: str

        |daq| identifier.

    .. attribute:: files
        :type: List[str]

        List of FITS files created for this |daq| in the format ``[user@]host:/absolute/path``.

    .. attribute:: keywords
        :type: str

        JSON-encoded FITS keywords, or empty. See :ref:`json-keywords` for more details.

        Example:

        .. code-block:: json

            [
               {
                  "type":"valueKeyword",
                  "name":"OBJECT",
                  "value":"OBJECT,SKY"
               },
               {
                  "type":"esoKeyword",
                  "name":"OBS TPLNO",
                  "value":2
               }
            ]

.. _json-keywords:

JSON Keywords
=============

FITS keywords are provided as a JSON-array of ``FitsKeyword`` objects, where each object describe
one keyword. The following example show one FITS standard value keyword, one ESO hierarchical
keyword and one literal keyword:

..  code-block:: json

    [
       {
          "type":"valueKeyword",
          "name":"OBJECT",
          "value":"OBJECT,SKY"
       },
       {
          "type":"esoKeyword",
          "name":"OBS TPLNO",
          "value":2
       },
       {
          "type":"literalKeyword",
          "name":"ORIGIN  = 'ESO-PARANAL'    / European Southern Observatory"
       }
    ]

.. note::

    The annotations use `Python Variable Annotation <https://www.python.org/dev/peps/pep-0526/>`_
    syntax:

    - ``str`` represent a string.
    - ``number`` represent a JSON number.
    - ``boolean`` represent a JSON boolean.
    - ``object`` represents a JSON object.
    - ``Union[A, B]`` represents a union where either type ``A`` or ``B`` are valid.
    - ``Optional[A]`` indicates that the property of type ``A`` is optional and can be omitted.


``FitsKeyword`` (Union[ValueKeyword, EsoKeyword, LiteralKeyword])
    This union object represent any valid FITS keyword.

``ValueKeyword`` (object)
    This object represents a FITS value keyword[\ |rd-fits|].

    ``type`` ("valueKeyword")
        This is a union discriminator and must have the literal string value ``"valueKeyword"``.

    ``name`` (str)
        :term:`Logical Keyword Name` (up to 8 characters).

        Examples:

        - ``RA``
        - ``DEC``
        - ``OBJECT``

    ``value`` (Union[str, boolean, number])
        For both ``valueKeyword`` and ``esoKeyword`` the value field provide the typed keyword
        value. See :ref:`json-keyword-types` for mapping between JSON and FITS.

    ``comment`` (Optional[str])
        Optional comment.

``EsoKeyword`` (object)
    This object represents a HIERARCH ESO FITS keyword [\ |rd-esokw|].

    ``type`` ("esoKeyword")
        This is a union discriminator and must have the literal string value ``"esoKeyword"``.

    ``name`` (str)
        :term:`Logical Keyword Name` which does not include the ``HIERARCH ESO`` prefix.

        Examples:

        - ``DET CHIP GAIN``
        - ``TEL AIRM START``
        - ``ADA GUID RA``

    ``value`` (Union[str, boolean, number])
        For both ``valueKeyword`` and ``esoKeyword`` the value field provide the typed keyword
        value. See :ref:`json-keyword-types` for mapping between JSON and FITS.

    ``comment`` (Optional[str])
        Optional comment.

``LiteralKeyword`` (object)
    This object represents a fully formatted FITS keyword record.

    ``type`` ("literalKeyword")
        This is a union discriminator and must have the literal string value ``"literalKeyword"``.

    ``value`` (str)
        Fully formatted FITS keyword record with optional trailing spaces.


.. _json-keyword-types:

.. list-table:: JSON to FITS type conversion
   :header-rows: 1
   :widths: auto

   * - JSON
     - FITS
     - Notes
     - Example
   * - string
     - string
     - Do not use single-qotes ``'``.

       Timepoints are represented as strings.
     - ``"foobar"`` -> ``'foobar'``, ``"2020-07-28T04:57:00.8836"`` ->
       ``'2020-07-28T04:57:00.8836'``
   * - boolean
     - logical
     -
     - ``true`` -> ``T``, ``false`` -> ``F``
   * - number
     - integer or float
     - Dictionary will be used to format the value correctly (not yet implemented).

       integer range: ``-9223372036854775807 ..`` ``9223372036854775807``

       float range: ``-1.79769313486231e+308 ..`` ``1.79769313486231e+308``
     -
