DDT Server

Overview

The DDT Server is a reusable component that reads telemetry data from a specific shared memory queue and publishes a (sub)set of data to one or multiple DDT datastreams.

Note

DDT stands for Data Display Tool; a collection of libraries and applications that allows ELT to transfer and display data coming from the telescope.

A DDT Server hosts several user defined DDT Forwarders. Each forwarder is composed of a Reader that reads data from a specific data source, and one or many DDT Publishers that publish data as DDT datastreams. A user-provided extraction function converts input data from the reader to a format that can be ingested by the publishers.

The RTC Tk provided IPCQ DDT Forwarder ingests samples from a shared memory queue. After the data extraction with the user-provided extraction methods, the results are published via DDT Publishers.

The forwarder can be configured to subsample incoming data, this feature should be used to reduce the data rate so that it can be handled by the DDT libraries (reduce to e.g. 5-20 Hz).

The DDT provides a specific transfer library and tools to view the transmitted data. The user is not required to know about the DDT details, except for the entries explained here. More details on the DDT can be found in its User Manual https://pdm.eso.org/kronodoc/HQ/ESO-424030.

Commands

A DDT Server component can be steered by sending commands to its provided MAL interface. The set of commands that is currently available is indicated in the following table, together with the expected behaviour:

Command

Behaviour

Init

Initialises the component:

  • Constructs and initialises DDT Forwarders with Readers and Publishers.

Exit

Terminates the process

Stop

Stops an ongoing initialisation procedure started by the Init command.

Reset

Resets the state of the component to the state before Init was received.

Enable

No particular actions are performed here.

Disable

No particular actions are performed here.

Update

Reloads dynamic component configuration.

Run

Starts data forwarding:

  • Notifies forwarders to start receiving samples from shared memory queue.

Idle

Stops data forwarding:

  • Notifies forwarders to stop receiving samples from shared memory queue. (stop threads, close datastream, etc.)

Recover

Recovers from an asynchronous error:

  • Notifies forwarders to stop and finalise transmission (stop threads, close datastream, etc.)

After receiving the Init and Enable commands, data forwarding can be started with command Run and stopped with command Idle.

Once forwarding is started and the component is in state On::Operational::Running, it ingests samples and forwards them to configured DDT streams.

Customisation

Instrument RTC developers customise the DDT Server by selecting and adding forwarders as required.

The IPCQ DDT Forwarder can be configured to split the ingested shared memory topic into an arbitrary number of DDT stream topics. This customisation happens already at compile-time by providing a traits class that defines the static details of the forwarder unit, including the publisher types. Additional customisation happens at run-time using dedicated configuration parameters that are explained later.

Note

A tutorial on how to create a custom DDT Server is provided in the DDT Server Tutorial.

IPCQ DDT Forwarder

To make use of the forwarder, instantiate an IpcqDdtForwarder and configure (template) it with a user-defined configuration struct IpcqDdtForwarderInfo (naming of this struct is up to the user).

The IpcqDdtForwarderInfo struct configures the IpcqDdtForwader, it contains:

  • ID: identifier of the forwarder.

  • Topic: shared memory topic type.

  • StreamInfo: possibly multiple stream info structs (naming of structs is up to users)

  • DdtPublisherTupleType: definition of publisher types using StreamInfo structs.

Note, at the moment, the RTC Tk only provides a DDT publisher for 2 dimensional images (DdtPublisherImage2d) which should be used to define DdtPublisherTupleType.

The StreamInfo structs are used to configure the DDT streams emitted by the publishers and they contain:

  • ID: identifier of the publisher.

  • PixelType: type of individual pixels (int, float …).

  • WIDTH: image width in pixels.

  • HEIGHT: image heights in pixels.

  • Extract: data extraction function

Custom DDT Forwarders

If necessary, it is possible to implement and add totally custom forwarders, readers and publishers. This can be achieved by creating a new forwarder class that inherits from the base class DdtForwarder which defines a common interface and provides basic functionality. Similarly custom DdtPublishers can be added but they also need to honor the default publisher interface.

Configuration

The following code snippet shows an example DDT Server configuration:

static:
  forwarders:
    pixel_subsampled:
      shm_queue_name: !cfg.type:string examplePixelsQueue
      thread_policies:
        cpu_affinity: !cfg.type:string 0
      publishers:
        raw_pixels:
          broker_endpoint: !cfg.type:string zpb.rr://127.0.0.1:5001
dynamic:
  forwarders:
    pixel_subsampled:
      subsample_factor: !cfg.type:int64 1
      publishers:
        raw_pixels:
          enabled !cfg.type:bool True

From the configuration tree one can see that the DDT Server component is composed of forwarders, that are again composed of publishers.

Note that the forwarder IDs e.g. pixel_subsampled and publisher IDs e.g. raw_pixels must match with the IDs specified in the corresponding configuration structs.

IPCQ DDT Forwarder Configuration

The following configuration options are specific to IPCQ DDT Forwarders:

Parameter

Nature

Type

Description

shm_queue_name

static

string

name of the shared memory queue

thread_policies:cpu_affinity

static

string

[optional] CPU affinity setting for the thread that reads from the IPCQ and forwards data to the sink_list.

subsample_factor

dynamic

int64

To forward every sample, set it to 1, to forward every 10th, set it to 10.

A subsample_factor larger 1 changes the reading pattern of the source to subsampled mode, where 1 sample is read and then N-1 samples are skipped (note that N is the value for the subsample_factor).

Two Dimensional DDT Publishers

The following DDT configuration options are applicable to DdtPublisherImage2d objects:

Parameter

Nature

Type

Description

broker_endpoint

static

string

[optional] URI of the DDT Broker, if not set zpb.rr://127.0.0.1:5001 is used.

enabled

dynamic

bool

[optional] is publisher enabled, if not set True is used.

Monitoring

For each DDT Server instance the following monitoring data points are published to OLDB:

Parameter

Type

Description

state

string

current component state (retrieved from state machine engine)

DDT Forwarder Monitoring

For more fine-grained monitoring, individual DDT forwarders publish monitoring information to OLDB. Since the composition of forwarders is user-defined, the overall set of available monitoring datapoints depends on the user customisation.

Note that FWD_ID is only a placeholder for the user-defined forwarder identifier.

Parameter

Type

Description

forwarders/FWD_ID/state

string

current state (Stopped, Starting, Idle, Running, Error)

forwarders/FWD_ID/type

string

type of the forwarder (IpcqDdtForwarder)

The following table describes each forwarder state closer:

Unit State

Description

Stopped

The forwarder was just constructed. This usually happens when the component receives command Init.

Starting

The forwarder is preparing for operation (spawn threads, load configuration) This usually happens when the component receives commands Run.

Idle

The forwarder is ready to receive a start signal.

Running

The forwarder is forwarding data to DDT.

Error

The forwarder encountered some error and stopped forwarding data.

IPCQ DDT Forwarder Monitoring

The following table provides an overview of monitoring data points specific to IPCQ DDT Forwarders:

Parameter

Type

Description

metrics/default/counter/forwarders/FWD_ID/samples_forwarded

int64

performance counter: number of samples forwarded

metrics/default/counter/forwarders/FWD_ID/last_sample_id_forwarded

int64

performance counter: id of last sample id forwarded

metrics/default/counter/forwarders/FWD_ID/frequency_estimate

double

performance counter: Estimated frequency of the forwarder [Hz]

metrics/default/counter/forwarders/FWD_ID/duration_estimate

int64

performance counter: Duration of forwarding data [us]

metrics/default/counter/forwarders/FWD_ID/buffer_occupancy

int64

performance counter: SHM read buffer occupancy [%]

Errors

Not Operational

The main errors that can occur during component initialisation are caused by:

  • wrong or missing mandatory configuration parameters.

  • problems while establishing connection to the DDT Broker.

Operational

While the DDT Server is running the main sources of errors are:

  • problems while opening, resetting or reading from the shared memory queue.

  • problems while publishing data to the DDT Broker.

Limitations and Known Issues

At this point, the toolkit-provided forwarders and publishers are not optimised for performance.

At this state, only one DDT forwarder type is supported, namely IpcqDdtForwarder. In future versions, other forwarders will be added: e.g. an OldbDdtForwarder or a RuntimeRepoDdtForwarder.

At the moment there is only one DDT publisher type, namely DdtPublisherImage2D. In future versions more publishers will be available e.g. for single and N-dimensional DDT streams.

As a limitation of the underlying DDT Transfer Library, the supported pixel datatypes are:

  • uint8_t

  • uint16_t

  • uint32_t

  • int16_t

  • int32_t

  • float

  • double

Another limitation of the underlying DDT Transfer Library are limited publication frequencies, so do not attempt to publish to DDT at loop rate. Use subsampling to publish at reduced rates not higher than 5-20 Hz.

Since names of the forwarders are also used to name processing threads, the maximum name length is currently limited to 15 characters. Longer names will cause compilation errors or exceptions being thrown.