Creating a Custom Metadata Collector

This example shows how to implement a custom Metadata Collector (MDC) that can receive requests from e.g. the science instrument Observation Coordination Manager (OCM) to produce FITS files containing keywords and binary tables. These FITS files are then merged by the science instrument Data Product Manager (DPM) to produce a single file that is delivered to the archive system.

Provided Example

The provided example implements a Metadata Collector that records events from the Event Service as well as datapoints from the Runtime Configuration Repository.

Note that the example is not intended to be used in an actual implementation, but rather as a template for an actual Metadata Collector.

Note

For simplicity reasons, the example uses the file-based implementation of OLDB, Persistent and Runtime Configuration Repositories, as well as file-based service discovery. This means that the underlying format of configuration and data points is different than the one used when the above mentioned services are used with the standard backends such as CII configuration service, CII OLDB, etc.

Source Code Location

The example source code can be found in the following sub-directory of the rtctk project:

_examples/exampleMdc

Modules

The provided example is composed into the following waf modules:

app - The application including default configuration
scripts - Helper scripts for deployment and control

Dependencies

The provided example component depends on the following modules:

rtctk.componentFramework.rtcComponent - basic RTC Component functionality
rtctk.componentFramework.services.dataRecording - common recording infrastructure
rtctk.reusableComponents.metadataCollector.lib - business logic class with metadaqif implementation

In addition, the following applications are used to steer the component:

rtctkClient - to send basic commands to the application
rtctkMdcTestClient - to send metadaqif commands to the application

Running the Example Telemetry Recorder

The Metadata Collector is run as part of an SRTC system in operation, but for development and testing purposes the component can also be run in isolation.

To make the example as simple as possible, a script rtctkExampleMdc.sh is provided to bring up and command an example Metadata Collector.

rtctkExampleMdc.sh

After installing the RTC Tk, run the example using the following sequence of commands:

# Deploy and start the example applications
rtctkExampleMdc.sh provision
rtctkExampleMdc.sh deploy

# Open a second terminal and use the rtctk client to bring the component to state On::Operational
rtctkExampleMdc.sh send Init
rtctkExampleMdc.sh send Enable

# Use the MDC testClient to simulate metadaqif commands to start/stop/abort
# acquisition sequences.
rtctkExampleMdc.sh send StartDaq session_1
rtctkExampleMdc.sh send GetDaqStatus session_1
rtctkExampleMdc.sh send StopDaq session_1

# Use the rtctk client to bring the component back to state On::NotOperational
rtctkExampleMdc.sh send Disable
rtctkExampleMdc.sh send Reset

# Gracefully terminate the applcations and clean-up
rtctkExampleMdc.sh undeploy
rtctkExampleMdc.sh unprovision

# This sequence of commands can also be run using
# rtctkExampleMdc.sh run

After a recording session has concluded, the recorded data can be found in the $DATAROOT directory. The created sub-directories correspond to the component name (mdc_1) and the session_id (session_1) that was provided as an argument to the StartDaq command.

Development Guide

This section explains how to create a simple Metadata Collector from scratch. For a more detailed description, see the Telemetry Recorder section that explains the inner workings of both the Telemetry Recorder and the Metadata Collector.

Recording Units

A Metadata Collector hosts one or many recording units. The RTC Tk provides recording units for IPCQ, Event, and DataPoint recording. Each recording unit has its own state and might wait for a condition before starting. E.g. the IPCQ recording unit supports waiting for a specific sample id before actually writing the data. It is also possible to follow another recording unit (leader), so that the recording is done while the other recording unit is recording.

RecordingInfo

To allow recording of arbitrary structs (like IPCQ topics or custom event types), a RecordingInfo trait class needs to be defined. To record a custom shared memory topic or event type, this class needs to be specialised by the user to allow the recorder to extract the data from these types.

In the provided example, only Toolkit-provided event types are recorded. Therefore no custom trait classes needs to be created (because they are already defined in the component framework). To see how a custom trait class is defined please refer to the Telemetry Recorder tutorial.

Business Logic Factory

The RTC Toolkit provides a reusable BusinessLogic class in rtctk::metadataCollector::BusinessLogic that supports metadaqif commands and manages multiple Recording Units. Since construction of the recording units is deferred to ActivityInitialising, instrument RTC developers need to provide a nested factory method in main.cpp that defines which recording units a component shall have. This factory is then passed to the generic component-runner method.

#include <rtctk/componentFramework/rtcComponentMain.hpp>
#include <rtctk/componentFramework/eventRecordingUnit.hpp>
#include <rtctk/componentFramework/dataPointRecordingUnit.hpp>
#include <rtctk/metadataCollector/businessLogic.hpp>

using namespace rtctk::componentFramework;
using namespace rtctk::metadataCollector;

// main entry point for user code
void RtcComponentMain(const Args& args) {

    // will be invoked at component startup
    auto bl_factory = [](const std::string& name, ServiceContainer& services) {

        // inner factory will be invoked in ActivityInitialising
        auto ru_factory = [](const std::string& name, ServiceContainer& services) {

            RecUnitListType rec_units;

            AddRecUnit<EventRecordingUnit<>>(rec_units, name, "unit_1", services);

            AddRecUnit<EventRecordingUnit<>>(rec_units, name, "unit_2", services);

            AddRecUnit<EventRecordingUnit<>>(rec_units, name, "unit_3", services);

            AddRecUnit<DataPointRecordingUnit<std::vector<float>>>(
                rec_units, name, "unit_4", services, DataPointPath{"/"+name+"/dp_to_record"});

            return rec_units;

        };

        // returns the biz logic object
        return std::make_unique<BusinessLogic>(name, services, std::move(ru_factory));
    };

    // invoke component runner function
    RunAsRtcComponent<BusinessLogic>(args, std::move(bl_factory));
}

The code above creates a new Metadata Collector with four recording units:

unit_1 is a EventRecordingUnit that records events from a configurable event topic
unit_2 is a EventRecordingUnit that records events from a configurable event topic
unit_3 is a EventRecordingUnit that records events from a configurable event topic
unit_4 is a DataPointRecordingUnit that records a vector of floats from the Runtime Configuration Repository

Note that each unit has a unique identifier, which is used to retrieve configuration for that unit from the Runtime Configuration Repository and to publish the current state of that unit to OLDB.

To add more recording units, it is sufficient to add AddRecUnit calls in the same manner.

Hint

Do not forget to add the RecordingInfo and the settings in the Runtime Configuration Repository, whenever applicable.