Metadata Collector

Overview

The Metadata Collector (MDC) is a reusable component that can receive requests from e.g. the science instrument Observation Coordination Manager (OCM) to produce FITS files containing keywords and binary tables. These FITS files are then merged by the science instrument Data Product Manager (DPM) to produce a single file that is delivered to the archive system.

The component is based on the same recording infrastructure as the Telemetry Recorder, which means it is also composed from a user-defined set of recording units, where each unit ingests data from a specific source and writes it into a corresponding FITS file. More on this can be found in the Telemetry Recorder section.

Commands

A Metadata Collector component can be steered by sending commands to its provided MAL interface. The set of commands that is currently available is indicated in the following table, together with the expected behaviour:

Command

Behaviour

Init

Initialises the component:

  • Constructs and initialises the selected Recording Units.

  • Creates OLDB monitoring datapoints for session monitoring.

Exit

Terminates the process.

Stop

Stops an ongoing initialisation procedure started by the Init command.

Reset

Resets the state of the component to the state before Init was received.

Enable

No particular actions are performed here.

Disable

No particular actions are performed here.

Update

Reloads dynamic component configuration.

StartDaq

Starts a new recording session:

  • Creates directory structure for session data (FITS files) in DATAROOT.

  • Updates session monitoring information in OLDB to indicate new session started.

  • Notifies recording units to start recording (spawn threads, create files, etc.)

StopDaq

Stops an ongoing recording session:

  • Notifies individual recording units to stop and finalise recording (stop threads, close files, etc.)

  • Updates session monitoring information in OLDB to indicate that session has concluded.

AbortDaq

Aborts an ongoing recording session:

  • Notifies recording units to stop and finalise recording (stop threads, close files, etc.)

  • Updates session monitoring information in OLDB to indicate that session has concluded.

GetDaqStatus

Requests information abort a certain recording session.

In contrast to the Telemetry Recorder, the Metadata Collector does not support commands Run, Idle and Recover. However, being an RTC Component, it still supports basic commands Init, Enable, Disable, Reset and Update, which are required to bring the component to state On::Operational and to trigger configuration updates.

To manage metadata acquisition sessions, the Metadata Collector implements ECS interface metadaqif, and thus provides additional commands StartDaq, StopDaq, AbortDaq and GetDaqStatus. These commands can be used in state On::Operational to control and monitor individual metadata acquisition sessions. A detailed description of the metadaqif can be found here.

Note that the Metadata Collector can only handle a single recording session at a time. If a client attempts to start a new session while another session is still ongoing, the StartDaq command will be rejected.

Once a recording session is started and the component is in state On::Operational::Acquiring, it will ingest data from various sources and write them to a set of output files located in folder $DATAROOT/component_id/session_id/.

In contrast to the Telemetry Recorder, a recording session cannot conclude on its own, meaning that each recording session must be finished with a StopDaq or AbortDaq command. The generated files can be accessed once the session has concluded.

In case a session fails, the component will auto-transition to state On::Operational::Idle and mark the session as failed. This can be observed with command GetDaqStatus.

Customisation

The MDC needs to be customized both at compile-time and run-time. Customisation is handled in the same manner as for the Telemetry Recorder, please refer to the Telemetry Recorder section for more information.

A tutorial on how to implement a custom Metadata Collector is provided in the Metadata Collector Tutorial.

Configuration

Configuration is handled in the same manner as for the Telemetry Recorder, please refer to the Telemetry Recorder section for more information.

Monitoring

Component monitoring is handled in the same manner as for the Telemetry Recorder, please refer to the Telemetry Recorder section for more information.

Limitations and Known Issues

At this point, not all toolkit-provided recording units are optimised for performance.

Only FITS binary tables and human readible CSV files are supported output formats. Recording units that can produce FITS keywords will be added in a sub-sequent release.

Currently, recording units based on the FitsDataWriter do not support the following data types: unit64_t, std::complex, MatrixSpan.

Since names of recording units are also used to name recording threads, the maximum name length is currently 15 characters. Longer names will cause exceptions being thrown.