Telemetry Recorder

Overview

The Telemetry Recorder is a reusable component that reads telemetry data from a specific shared memory queue and stores it into FITS files. In addition, the component can also read related data from other sources and store them into supporting FITS files or human-readable CSV files.

A Telemetry Recorder instance only supports one recording session at a time, therefore instrument RTC developers should foresee multiple recorder instances if the need for multiple, concurrent recording sessions arises.

A Telemetry Recorder hosts a user-defined set of recording units, where each unit ingests data from a specific source and writes it into a corresponding FITS file. This allows recording different data streams concurrently in a single recording session.

Commands

A Telemetry Recorder component can be steered by sending commands to its provided MAL interface. The set of commands that is currently available is indicated in the following table together with the expected behaviour:

Command	Behaviour
Init	Initialises the component: Constructs and initialises the selected Recording Units. Creates OLDB monitoring datapoints for session monitoring.
Exit	Terminates the process.
Stop	Stops an ongoing initialisation procedure started by the Init command.
Reset	Resets the state of the component to the state before Init was received.
Enable	No particular actions are performed here.
Disable	No particular actions are performed here.
Update	Reloads dynamic component configuration.
Run	Starts a new recording session: Creates directory structure for session data (FITS files) in DATAROOT. Updates session monitoring information in OLDB to indicate a new session has started. Notifies recording units to start recording (spawn threads, create files, etc.)
Idle	Stops an ongoing recording session: Notifies individual recording units to stop and finalise recording (stop threads, close files, etc.) Updates session monitoring information in OLDB to indicate that session has concluded.
Recover	recovers from a failed recording session: Notifies recording units to stop and finalise recording (stop threads, close files, etc.) Updates session monitoring information in OLDB to indicate that session has concluded.

After receiving the Init and Enable commands, a new recording session can be started with command Run and stopped with command Idle.

Once a recording session is started and the component is in state On::Operational::Running, it ingests data from various sources and writes it to a set of output files located in folder $DATAROOT/component_id/session_id/.

In case the recorder is configured to stop recording on its own, it will automatically transition to state On::Operational::Idle once a configurable stop-condition is met. In this case, it is not necessary to send the Idle command.

In case a recording session fails, the component transitions to state On::Operational::Error. Error recovery can be requested by sending the command Recover.

Customisation

Instrument RTC developers customise the Telemetry Recorder by selecting and adding recording units as required. Recording unit objects of different types are constructed with specific arguments and then passed to the component via dependency injection. This means, coarse component customisation happens already at compile-time where different recording units are composed together. For fine-grained customisation, recording units can also be configured at run-time using dedicated configuration parameters that are explained later.

Note

A tutorial on how to create a custom Telemetry Recorder is provided in the Telemetry Recorder Tutorial.

To facilitate fast and uniform component development, the RTC Toolkit provides an initial set of recording unit types that can be instantiated and added to a Telemetry Recorder:

IpcqRecordingUnit: Listens to an IPCQ shared memory topic and writes data samples into FITS binary tables. For customisation, the unit is parametrised with a RecordingInfo trait class.

DataPointRecordingUnit: Records datapoints from the Runtime Repository or OLDB into FITS binary tables. For customisation, the unit is parametrised with the Datapoint Type and the Datapoint Path.

DynamicDataPointRecordingUnit: Records datapoints from the Runtime Repository or OLDB into FITS binary tables. Datapoints are read from the Runtime Repo. For customisation, a custom RecorderFactory can be supplied, otherwise recording will be done to FITS files.

EventRecordingUnit: Records arbitrary JSON events into JSON lines (.jsonl) text files. For customisation, the unit can be parametrised an optional FilterFunction to filter out unwanted samples.

To provide sufficient degrees of freedom, multiple instances of the same recording unit type can be created and added to the recorder component.

If necessary, it is possible to implement and add totally custom recording units to cover specific needs. This can be achieved by creating a new recording unit class that inherits from the base class RecordingUnit, which defines a common interface and provides basic functionality. When implementing the custom unit, the toolkit-provided FitsDataRecorder class can be used to store ingested data as FITS binary tables. Alternatively, a CsvDataRecorder class can be used to store ingested data as CSV files. The main drawback of the FitsDataRecorder is that it requires all data to be fixed length for a recording session. This is done to prevent performance issues in the recording stemming from the way dynamically-sized types are stored in FITS binary tables. To still be able to crop fixed-sized data, a dedicated method SetColumnLength can be used. Note that the CsvDataRecorder does not have this restriction.

Configuration

The configuration of a Telemetry Recorder can be divided into recording session configuration and recording unit configuration.

Recording Session Configuration

Currently the following recording session configuration options are supported:

Parameter	Nature	Type	Description
session_id	dynamic	string	unique identifier of the current/last recording session.

The session_id needs to be updated from sequencer scripts or custom recorder GUIs before starting a new recording session.

Recording Unit Configuration

The following recording unit configuration options are common to all recording unit types:

Parameter	Nature	Type	Description
leader_list	static	vector_string	[optional] list of recording units that can trigger and stop the recording of this unit.

If the leader_list parameter is present and non-empty, the leader/follower functionality is enabled for this particular recording unit. The follower unit starts recording after the first leader has started and it will stop recording after the last leader has finished. With this feature, recording units can be chained together such that e.g. interesting datapoints are being recorded when a leading shared memory recording unit starts or stops.

IPCQ Recording Unit

The following configuration options are specific to IPCQ recording units:

Parameter	Nature	Type	Description
shm_queue_name	static	string	name of the ipcq.
cpu_affinity	static	int32	[optional] CPU affinity setting for the thread that reads from the IPCQ and writes to disk.
telemetry_subset	dynamic	vector_string	[optional] Only record the given parts of the topic.
subsample_factor	dynamic	int64	Defines how many samples shall be skipped after each read sample. To record every sample, set it to 1, to record every 10th, set it to 10.
start_at_sample_id	dynamic	int64	Have the recording unit wait for a minimum sample_id, before starting the recording and triggering of its followers. To start immediately, set it to 0.
stop_after_num_samples	dynamic	int64	This lets the recording unit stop automatically after having recorded the given amount of samples. To not stop automatically, set it to 0.

If present, the optional telemetry_subset parameter allows to specify a subset of telemetry data columns that should be recorded, this means that only a subset of data that is read from SHM is actually written to file.

A subsample_factor larger than 1 changes the reading pattern of the recording unit to subsampled mode, where 1 sample is read and then N-1 samples are skipped (note that N is the value for the subsample_factor).

If start_at_sample_id is set to zero, the unit will start recording immediatly. A value N (larger than zero) will cause the unit to defer recording to file until the received sample id matches N.

If stop_after_num_samples is set to zero, the unit keeps on recording until it is explicitly stopped. A value N (larger than zero) will cause the unit to stop automatically after N samples have been recorded.

DataPoint Recording Unit

The following configuration options are specific to DataPoint recording units:

Parameter	Nature	Type	Description
capture_mask	static	int32	bit mask to configure wheter data should be captured on_start, on_stop or on_change.

The binary capture mask can be set as follows:

capture on_start: Bit 1

capture on_stop: Bit 2

capture on_change: Bit 3

As an example, capture_mask value 3 means to capture on_start and on_stop.

Dynamic DataPoint Recording Unit

The following configuration options are specific to Dynamic DataPoint recording units:

Parameter	Nature	Type	Description
datapoint_list	static	vector_string	List of Datapoints to record. Requires prefix: “rtr:” for runtime repo or “oldb:” for OLDB DataPoints.

The Dynmic Datapoint Recording Unit creates internal DataPoint Recording Units, which can be configured by changing the settings for the recording unit with the same unit_id, but suffixed with “_nn”, where nn is the two digit numbered entry in the datapoint_list.

Event Recording Unit

The following configuration options are specific to this type of unit:

Parameter	Nature	Type	Description
topic_name	static	string	event channel topic name.

Configuration Example

The following example configuration shows a Telemetry Recorder with a leading IPCQ recording unit and a following DataPoint recording unit:

static:
  rec_units:
    ipcq_unit:
      shm_queue_name: !cfg.type:string exampleTelRecQueue
      cpu_affinity: !cfg.type:int32 0
    dp_unit:
      leader_list: !cfg.type:vector_string [ipcq_unit]
      capture_mask: !cfg.type:int32 3
dynamic:
  session_id: !cfg.type:string session_42
  rec_units:
    ipcq_unit:
      telemetry_subset: !cfg.type:vector_string [intensities, slopes]
      subsample_factor: !cfg.type:int64 10
      start_at_sample_id: !cfg.type:int64 0
      stop_after_num_samples: !cfg.type:int64 10

The IPCQ recording unit is configured to record 10 samples from exampleTelRecQueue, starting immediately when the first sample arrives (from sample id 0) with a subsample_factor of 10. The unit is pinned to core 0 and the telemetry_subset is configured such that only the two columns intensities and slopes are recorded.

The following DataPoint recording unit records a runtime repository datapoint both at the start and at the end of the recording session.

Monitoring

For each Telemetry Recorder instance the following monitoring data points are published to OLDB:

Parameter	Type	Description
state	string	current component state (retrieved from state machine engine).

Recording Session Monitoring

In addition, the following recording session information is published to OLDB:

Parameter	Type	Description
session_info/session_id	string	unique identifier of the current/last recording session.
session_info/files_produced	vector_string	list of FITS files produced.
session_info/files_size	vector_string	sizes of produced FITS files in bytes.
session_info/total_size	string	total size of FITS files produced in bytes.
session_info/space_info/capacity	string	storage capacity in bytes.
session_info/space_info/free	string	free storage in bytes.
session_info/space_info/available	string	available storage in bytes.

This can be used to get detailed information about the ongoing recording session. Note that this information can also be integrated into custom Recorder GUIs.

Recording Unit Monitoring

For more fine-grained monitoring, individual recording units also publish monitoring information to OLDB. Since the composition of recording units is user defined, the overall set of available monitoring datapoints depends on the user customisation.

The following monitoring attributes are common to all types of recording units:

Note that UNIT_ID is only a placeholder for the user-defined recording unit identifier.

Parameter	Type	Description
rec_units/UNIT_ID/state	string	current state (Stopped, Preparing, Idle, Waiting, Running, Finished, Failed).
rec_units/UNIT_ID/type	string	type of the recording unit (IPCQ, DataPoint, TypedEvent, JsonEvent).

The following table describes each recording unit state closer:

Unit State	Description
Stopped	The unit was just constructed. This usually happens when the component receives the command `Init`.
Preparing	The unit is preparing for operation (spawn threads, load configuration) This usually happens when the component receives the command `Run` or `StartDaq`.
Idle	The unit is ready to receive a start signal.
Waiting	The unit got a start signal but is waiting for an additional condition. The condition can be `leader_unit_started` or `start_sample_id_reached`.
Running	The unit is currently recording.
Finished	The unit sucessfully finished its recording. This happens when the component receives the command `Idle` or `StopDaq` or when the unit reaches its `self-termination-condition`.
Failed	The unit aborted recording due to some error.

IPCQ Recording Unit

The following monitoring data points are specific to IPCQ recording units:

Parameter	Type	Description
rec_units/UNIT_ID/queue_name	string	name of the shared memory queue.
metrics/counter/UNIT_ID/samples_written	int64	performance counter: number of samples written in this session.
metrics/counter/UNIT_ID/last_sample_id_written	int64	performance counter: id of last sample written to file.
metrics/counter/UNIT_ID/frequency_estimate	double	performance counter: Estimated frequency of the data writer [Hz].
metrics/counter/UNIT_ID/duration_estimate	int64	performance counter: Duration of writing data to file [us].
metrics/counter/UNIT_ID/buffer_occupancy	double	performance counter: SHM read buffer occupancy [%].

Data Point Recording Unit

The following monitoring data points are specific to Datapoint recording units:

Parameter	Type	Description
rec_units/UNIT_ID/dp_name	string	path name of the datapoint to record.
metrics/counter/UNIT_ID/samples_written	int64	performance counter: number of samples written in this session.

Typed Event and JSON Event Recording Unit

The following monitoring data points are specific to Typed Event and JSON Event recording units:

Parameter	Type	Description
rec_units/UNIT_ID/topic_name	string	name of the DDS event topic to record.
metrics/counter/UNIT_ID/samples_written	int64	performance counter: number of samples written in this session.

Errors

Not Operational

The main errors that can occur during component initialisation are caused by wrong or missing mandatory configuration parameters.

Operational

While the Telemetry Recorder is running the main sources of errors are:

Problems while opening, resetting or reading from the shared memory queue.
Problems while opening, writing or closing the FITS file.

Limitations and Known Issues

At this point, not all toolkit-provided recording units are optimised for performance.
Only FITS binary tables and human readable JSON lines files are supported as output formats. Recording units that can produce FITS keywords will be added in a sub-sequent release.
Currently, recording units based on the FitsDataWriter do not support the following data types: std::complex, MatrixSpan.
The capture mask for DynamicDataPointRecordingUnit is not applied to sub-units (RTCTK-1371).