Telemetry Republisher
Overview
The Telemetry Republisher RTC Component reads telemetry data in MUDPI format from the HRTC, and forwards (republishes) it using DDS reliable Multicast to one or more SRTC nodes. In addition, it is possible to configure the Telemetry Republisher to simulate the DDS topics, meaning that it generates MUDPI samples instead of republishing them.
Prerequisites
Note
It is important to maintain each sample’s time information throughout the entire telemetry path,
from its source (e.g. HRTC or WFS) to consumption in the SRTC
(e.g. Data Task and/or Telemetry Recorder).
Therefore, all MUDPI samples shall be timestamped at the source of the data,
i.e. when the data is produced.
The information should be stored in the MUDPI header timestamp field.
If a MUDPI sample consists of multiple MUDPI frames, all of them must have the same timestamp.
Telemetry data is published via FastDDS provided by ELT Development Environment. FastDDS QoS
profiles have to be provided in telemDataPathDdsQos.xml installed in
$INTROOT/resource/config/rtctk/dds/ or any other location under rtctk/dds/ in $CFGPATH.
A different file name can be specified in the component’s configuration (see Configuration
section below for details).
Note
The FASTDDS_DEFAULT_PROFILES_FILE environment variable shall not be used (set).
This is particularly problematic if the file that the variable points to contains the same
QoS profiles.
The Telemetry Republisher works optimally for MUDPI traffic with Ethernet MTU of size 9000. Nevertheless, it works also with different sizes. Thus it is recommended to first exercise Telemetry Republisher with default (set by OS) MTU, and afterwards change the MTU size for specific MUDPI traffic.
Customisation
Telemetry Republisher allows data to be customised before it is written to DDS. The user-customisable function that does this is called “wrangler”. The wrangler function gets the complete payload of a MUDPI sample. It can modify the data as needed and write the data to an output buffer, which is then published via DDS. By implementing a wrangler function, data can be reordered or discarded from the MUDPI payload before it is sent over DDS. The toolkit provides a default Republisher, which will simply copy the payload to DDS. This is usually sufficient and only in rare cases implementing a custom wrangler should be needed.
The customisation is done by creating a custom program that provides a factory that returns
the wrangler function for a given topic_id.
This allows specifying different wranglers per topic ID.
A wrangler is a function that can modify the data during republishing,
allowing for example to change byte order or remove unnecessary information.
To build your own Telemetry Republisher with a custom wrangler, a custom waf module needs to be created. E.g.:
from wtools.module import declare_cprogram
declare_cprogram(
target="myTelRepub",
use="rtctk.reusableComponents.telRepub.lib"
)
The code for this new program then needs to provide a factory for the TelRepubBusinessLogic which will then create the TelRepubBusinessLogic with the wrangler factory. E.g.:
#include <rtctk/telRepub/wrangler.hpp>
#include <gsl/span>
#include <iterator>
#include <rtctk/componentFramework/rtcComponentMain.hpp>
#include <rtctk/telRepub/telRepubBusinessLogic.hpp>
#include <system_error>
using namespace rtctk::componentFramework;
using namespace rtctk::telRepub;
std::error_code ExampleWranglerCopy(const gsl::span<const gsl::span<const uint8_t>> input, std::vector<uint8_t>& output) {
// reserve total length
auto total_length = 0;
for (auto subspan : input) {
total_length += subspan.size();
}
output.reserve(total_length);
// copy
for (auto subspan : input) {
output.insert(output.end(), subspan.begin(), subspan.end());
}
return std::error_code{};
}
void RtcComponentMain(const Args& args) {
auto bl_factory = [](const std::string& name, ServiceContainer& services) {
// inner factory will be invoked in ActivityInitialising
RepubWranglerFactory wrangler_factory = [](int32_t id) -> WranglerFunction {
switch(id) {
case 0:
return &ExampleWranglerCopy;
default:
return nullptr;
}
};
// returns the biz logic object
return std::make_unique<TelRepubBusinessLogic>(name, services, std::move(wrangler_factory));
};
RunAsRtcComponent<TelRepubBusinessLogic>(args, std::move(bl_factory));
}
Note
The spans provided to the wrangler are spans pointing to the individual payloads of the MUDPI packets. The wrangler is responsible to copy the data to the destination buffer.
The WranglerFactory can always return a nullptr for a factory function; in this case the default
copying wrangler will be used. The wrangler function takes a span of spans of bytes. The spans
of bytes represent the payloads of the MUDPI packets. The output vector is pre-allocated to the
size of all the payloads. The content of this vector will be the content of the AgnosticTopic
DDS message. The error code can be used to report an error during wrangling.
Running
The Telemetry Republisher can be started (after deployment) by invoking the command:
$ rtctkTelRepub tel_repub_1
This starts the default TelRepub which uses a wrangler that simply copies all payloads of each MUDPI sample into a single DDS message. When having a custom wrangler the custom Telemetry Republisher needs to be started the same way.
The component can be stopped either by sending Exit command or by pressing Ctrl-C.
For more information about optional command line arguments refer to section RTC Component.
Commands
Telemetry Republisher instances can be steered by sending commands to their provided CII MAL interfaces.
The set of commands is indicated in the following table, together with the expected behaviour:
Command |
Behaviour |
|---|---|
Init |
Reads configuration from the Runtime Repository, creates entities for listening to MUDPI traffic on a UDP socket, and creates entities for publishing the DDS agnostic topic. At this stage reading and publishing is not yet started, but a subscription to the created DDS topic(s) can be established. |
Exit |
Terminates the process and dumps statistics. |
Stop |
Stops the initialisation procedure started by the Init command. |
Reset |
Resets the state of the component to the state before the Init command was received. This means that all DDS objects are destroyed, shared memory is released, and the processing and monitoring threads are also destroyed. |
Enable |
No particular actions are performed here. |
Disable |
No particular actions are performed here. |
Update |
Reloads dynamic component configuration. Currently a no-op, since all configuration is static. |
Run |
Requests transition to On::Operational::Running, which:
|
Idle |
Requests transition to On::Operational::Idle, which:
|
ClearAlerts |
Clears all alerts. |
Configuration
As with other components, the persistent configuration for the Telemetry Republisher is defined
in a YAML file. The configuration file name has to correspond to the name of the component instance.
At the moment, the configuration only contains static configuration attributes. This means the
configuration cannot be changed while in state Operational. To apply a changed configuration,
the component needs to be re-initialized (i.e. send Init again).
The republisher configuration structure looks like:
static:
# ... common telemetry republisher configuration
# ...
mudpi_receivers:
<receiver-name>:
# ... receiver's configuration
dds_topics:
<dds-topic-name>:
#... DDS topic's configuration
# ... more DDS topics
# ... more receivers (and DDS topics inside)
simulated_dds_topics:
<simulate-dds-topic-name>:
# ... simulated DDS topic's configuration
# ... more simulated DDS topics
Where <receiver-name>, <dds-topic-name> and <simulate-dds-topic-name>
are corresponding concrete names.
As we can see the telemetry republisher’s configuration can be divided into three groups:
common configuration
receivers with DDS topics configuration
(optional) simulated DDS topics configuration
Common Configuration
Example configuration (YAML): The DDS QoS Profile used for setting QoS DDS entities like DDS participant and DDS publisher, and allowed network interfaces for DDS can be specified in the common configuration part.
Configuration Path |
Type |
Description |
|---|---|---|
|
|
(optional) DDS domain identifier. If not specified, domain id 0 is used. |
|
|
(optional) DDS QoS Profile to be used for setting QoS of DDS entities like DDS participant
and DDS publisher. The specified QoS Profile needs to be contained in the
|
|
|
(optional) Name of DDS QoS XML file. The file should be found in |
|
|
(optional) List of network interfaces to be used by DDS.
The interfaces are for the local machine where the component is running.
If given, this list will replace any settings under the |
|
|
(optional) Time in milliseconds, which determines how often the monitoring thread inside ActivityRunning is executed to check and report errors/alerts. If not specified, the default value is 1000 ms. |
|
(optional) NUMA policies for the Monitoring thread (ActivityRunning). |
An example of a common configuration block:
static:
dds:
qos_profile: !cfg.type:string RtcTk_Default_Profile
qos_file: !cfg.type:string telemDataPathDdsQos.xml
interface_white_list: !cfg.type:vector_string
- 127.0.0.1
- 192.168.5.44
- lo
# ...
Receivers with DDS topics Configuration
The Telemetry Republisher can listen to one or more receivers that are specified under the mudpi_receivers section using YAML keywords. Each receiver is specified with an arbitrary name - YAML keyword.
Note
The receiver name needs to be specified in snake_case.
For each receiver the following can be specified:
Configuration Path |
Type |
Description |
|---|---|---|
|
|
Network adapter IP address. If not specified together with the Important: It must be an IP address and not for example an adapter or hostname. |
|
|
Port to listen to. |
|
|
(optional) Specifies the multicast group address to listen on.
If specified, the |
|
|
(optional) Receiver socket buffer size in bytes. If the specified buffer size is bigger than the operating system maximum size (‘/proc/sys/net/core/rmem_max’), the size is truncated to the system’s maximum size. |
|
(optional) Defines NUMA policies for the UDP receiver thread. |
|
|
|
(optional) Timeout in milliseconds to receive MUDPI frames. This is applied to all the MUDPI topics received on a particular receiver. If not specified or the value is 0, no timeout is applied, i.e. this timeout is not checked. Default value is 20000ms. This timeout is checked by the monitoring thread, whose polling interval is configured by
|
For each receiver there should be specified DDS topics, i.e. topics that are mapped to MUDPI topics received by the particular receiver. DDS topics shall be specified under dds_topics section of the particular MUDPI receiver. The name can be arbitrary. This name is used when the DDS topic is created by the Telemetry Republisher.
Note
The DDS topic name needs to be specified in snake_case and be unique per telemetry republisher instance (and, if the same DDS domain is used, also unique in the whole system)!
For each DDS topic the following can be specified:
Configuration Path |
Type |
Description |
|---|---|---|
|
|
Map to MUDPI topic id. |
|
|
(optional) The maximum number of MUDPI frames allowed in the internal queue between the MUDPI receiver and DDS publisher. Default size: 200. Important: The cumulative size of queues of all DDS topics in the same receiver must not exceed 65535! |
|
(optional) NUMA policies for the DDS publisher thread. |
|
|
|
(optional) The expected increment of the sample_id between successive received samples (useful for sub-sampled telemetry topics). If not specified, the default value is 1. |
|
|
(optional) Timeout in milliseconds to receive a MUDPI sample for all the topics. If not specified or the value is 0, no timeout is applied, i.e. this timeout is not checked. Default value is 20000ms. This timeout is checked by the monitoring thread, whose polling interval is configured by
|
Example configuration for two receivers named my_first_receiver and telem_rcv, where the first has two DDS topics (test_topic_00 and test_topic_01), and the second contains the test_topic_02 DDS topic:
static:
# ...
mudpi_receivers:
my_first_receiver:
network_adapter_ip: !cfg.type:string 127.0.0.1
port: !cfg.type:uint16 6000
buffer_size_bytes: !cfg.type:uint32 20000
thread_policies:
cpu_affinity: !cfg.type:string 1
topic_00:
dds_topics:
test_topic_00:
mudpi_topic: !cfg.type:int32 100
queue_size: !cfg.type:uint32 2000
test_topic_01:
mudpi_topic: !cfg.type:int32 101
telem_rcv:
network_adapter_ip: !cfg.type:string 127.0.0.1
port: !cfg.type:uint16 6500
dds_topics:
test_topic_02:
mudpi_topic: !cfg.type:int32 200
# ...
The configuration can be inspected using the Configuration Tool e.g. to check if a certain configuration datapoint exists.
Simulated DDS Topic Configuration
Simulated DDS topics, i.e. DDS topic where payload is not received from a MUDPI receiver but generated, can be specified under the simulated_dds_topics section.
Note
Just like for real payloads, the DDS topic name needs to be specified in snake_case and be unique per telemetry republisher instance (and, if the same DDS domain is used, also unique in the whole system)!
For each simulated DDS topic the following can be specified:
Configuration Path |
Type |
Description |
|---|---|---|
|
|
Frequency to generate the topic data. Important: The frequency should be reasonable to avoid getting the system too busy. |
|
|
(optional) The size of the payload in bytes for simulated topic. Default payload size is: 32768 bytes. |
|
(optional) NUMA policies for the DDS publisher thread. |
Example configuration for three simulated DDS topics (sim_topic_ex, simulated_topic and sim_slopes_topic):
static:
# ...
simulated_dds_topics:
sim_topic_ex:
sim_freq: !cfg.type:uint16 10
sim_payload_size_bytes: !cfg.type:uint32 128000
simulated_topic:
sim_freq: !cfg.type:uint16 50
sim_slopes_topic:
sim_freq: !cfg.type:uint16 10
Errors
Not Operational
During initialization i.e in state On::NotOperational::Initialising, several errors can occur:
In case of a problem when creating the MUDPI receiver, an error message is logged and the component falls back to the lower state On::NotOperational::NotReady.
2025-07-22 11:29:39.775Z [ERROR , rtctk] Problem to create MUDPI receiver part for my_first_receiver: bind: Cannot assign requested address
2025-07-22 11:29:39.851Z [ERROR , rtctk] Nested exceptions:
1. ActivityInitialising: failed
2. Problem to create MUDPI receiver part for: my_first_receiver
3. bind: Cannot assign requested address
In the above case the problem is binding UDP socket to a particular IP address.
When the total queue sizes of all DDS topics within the same receiver exceed 65,535, an error similar to the following is logged
[13:24:12:238][ERROR , rtctk] Problem to create MUDPI receiver: rcv_00: boost.lockfree: freelist size is limited to a maximum of 65535 objects
Operational
While publishing, i.e. in state On::Operational::Running, the following log messages can occur:
If an unexpected SampleId is encountered for a particular MUDPI topic, the Telemetry Republisher drops the sample and resynchronises but it reports this as a non-fatal error:
2025-07-22 11:29:39.851Z [ERROR , rtctk] DDS Topic: "test_topic" MUDPI Topic [3] SampleId: 1356 FrameId: 1 Expected SampleId 1351. Unexpected SampleId, frame rejected.
- Reasons for receiving an unexpected sampleId can be:
start of the Telemetry Republisher (it synchronises at the beginning)
restart of the MUDPI data source without reinitializing the Telemetry Republisher
misconfiguration of the parameter expected_sample_id_increment
frames or samples being lost or received out of order.
Similarly, the Telemetry Republisher will also drop a sample and resynchronise if an unexpected FrameId is encountered. This is again logged as a non-fatal error:
2025-07-22 11:29:40.103Z [ERROR , rtctk] DDS Topic: "test_topic" MUDPI Topic [3] SampleId: 694 FrameId: 17 Expected FrameId 14. Unexpected FrameId, frame rejected.
- Reasons for receiving an unexpected frameId can be:
frames being lost or received out-of-order.
After such errors, the Telemetry Republisher tries to resynchronise and if successful logs a message like this:
2025-07-22 11:29:40.144Z [INFO , rtctk] DDS Topic: "test_topic" MUDPI Topic [3] SampleId: 843 FrameId: 1. Synchronised again
In case there is not enough space (=free slots) in the UDP receiver frame buffer, we get an error message like:
2025-07-22 11:29:40.144Z [ERROR , rtctk] BufferManager: no free slot available in 'Buffer Manager for UDP receiver index 1'.
2025-07-22 11:29:40.144Z [ERROR , rtctk] Topic [3] No free slot available in buffer, trying to recover.
It could be a consequence of slow publishing of a DDS topic, which might indicate a problem with network, and/or DDS QoS configuration, and/or slow DDS subscriber. It could also be a consequence of a too small queue size for the topics.
In case there is no mapping between the MUDPI topic Id and the DDS topic, a message like the following is logged:
2025-07-22 11:29:40.668Z [WARN , rtctk] Topic [1234] has no mapping to DDS topic
This message means that the topic with Id 1234 has no corresponding mapping. It might be that there is no defined DDS topic in the Configuration that maps to the 1234 topic, i.e. it has no mudpi_topic datapoint.
Timeout to send (write) to the dds topic is reported with following the component log:
2025-07-22 11:29:40.733Z [ERROR , rtctk] [test_topic_02] SampleId: 374896008. DDS write timeout!
This indicates slow subscribers, or some other DDS problem.
When there is a problem where the UDP received packet size does not match the size of the MUDPI header (= 32 bytes) + size of payload + size of the MUDPI checksum (= 2 bytes), an error message like the following is logged:
2025-07-22 11:29:41.275Z [ERROR , rtctk] DDS Topic: "test_topic" MUDPI Topic [0] SampleId: 121 FrameId: 2. Packet Size mismatch: actual size 8000 expected size 8002
If the MUDPI checksum is wrong an error message as follows is logged:
2025-07-22 11:29:41.339Z [ERROR , rtctk] DDS Topic: "test_topic" MUDPI Topic [0] SampleId: 59 FrameId: 2. Wrong MUDPI Checksum: 24 should be: 15707
Monitoring
For each Telemetry Republisher instance the following monitoring data points are published to OLDB:
Parameter |
Type |
Description |
|---|---|---|
state |
string |
current component state (retrieved from state machine engine) |
Additionally, subscription to DDS topics can be observed in the logs. E.g. a log message stating the “connection” to the particular DDS topic (test_topic_00), i.e. corresponding DDS subscriber, comes up:
2025-07-22 11:29:41.340Z [INFO , rtctk] on_publication_match (test_topic_00)
When a DDS subscriber “disconnects” from the particular (test_topic_00), the Telemetry Republisher with name tel_repub_1 logs a message similar to this:
2025-07-22 11:29:41.450Z [INFO , rtctk] on_publication_match (test_topic_00)
Component Metrics
The Telemetry Republisher uses the Component Metrics Service to publish useful metrics to the OLDB.
Under the OLDB path /<component-name>/metrics/counter/dds_topics/<topic-name>/,
the following performance counter metrics related to DDS topics
(including MUDPI processor and Wrangler) can be found:
OLDB Path |
Type |
Description |
|---|---|---|
|
|
MUDPI frames received. |
|
|
MUDPI samples received. |
|
|
last MUDPI sample ID received. |
|
|
frequency estimate of MUDPI samples [Hz]. Note: The frequency estimate is calculated at the time it received the last MUDPI sample. This means that when no MUDPI samples are being received, the last estimate is shown. Use timeout alerts to detect such situations. |
|
|
MUDPI frame errors: received frameId different than expected one (with same sampleId), wrong frame’s CRC, frameId out of range or wrong size. |
|
|
MUDPI sample errors: received sampleId different than expected one. |
|
|
DDS samples published. |
|
|
last DDS sample ID published. |
|
|
publication duration for DDS samples [us]. |
|
|
DDS publication errors. |
|
|
Duration of the wrangling step [us]. |
Under the OLDB path /<component-name>/metrics/counter/udp_receivers/<receiver-name>/ the
following performance counter metrics related to receivers can be found:
OLDB Path |
Type |
Description |
|---|---|---|
|
|
Number of frames received by the UDP receiver. |
|
|
Frames dropped by the UDP receiver because the queue was full. |
|
|
Buffer usage (min, mean, max, global_max) by the UDP receiver [%]. |
|
|
Time between receives (min, max, mean) for the UDP receiver [us]. |
|
|
Time needed for receiving (min, max, mean) MUDPI frames by the UDP receiver [us]. |
Note
For details about Data Points, please refer to Component Metrics OLDB Data Points.
As the OLDB data path needs to be lowercase, topic names (<topic-name>)
and receiver names (<receiver-name>) are lower case.
Alerts
The following alerts are used in Telemetry Republisher.
Note
Most of the alerts like timeouts are detected by the Monitoring thread which is regularly executed. Its execution time as well as other thread policies are configurable. See: Common Configuration.
Note
Latched means that the alert will be set active if the respective problem occurs in the Running
state. The alert remains active until it is explicitly cleared together with all other
alerts using the ClearAlerts command,
or either a Run -> Idle or Idle -> Run transition is made.
Under OLDB path /<component-name>/alert/source/dds_topics/<topic-name> the following alerts
related to DDS topics (including MUDPI processor and Wrangler) can be found:
Alert Id |
Latched |
Description |
|---|---|---|
|
Yes |
Indicates a problem to receive a MUDPI sample within a given time for the particular topic. |
Under OLDB path /<component-name>/alert/source/udp_receivers/<receiver-name> the following
alerts related to UDP receivers can be found:
Alert Id |
Latched |
Description |
|---|---|---|
|
Yes |
Indicates a problem to receive a MUDPI frame within a given time for any topic in the particular receiver. |
Note
As the OLDB data path needs to be lowercase, topic names (<topic-name>)
and receiver names (<receiver-name>) are lower case.
In addition to being written to the OLDB, all alerts are also logged as WARN-level messages. For example:
2025-07-22 11:29:45.977Z [WARN , rtctk.alert] Alert status: **active**
- [dds_topics/loop_data_01/mudpi_processor/receive_sample_timeout] since: 2025-05-30 07:11:32.738883070 text: Timeout (20000[ms]) to receive MUDPI topic_id: 1 (mapped to DDS topic: loop_data_01)
- [dds_topics/loop_data_02/mudpi_processor/receive_sample_timeout] since: 2025-05-30 07:11:32.771502146 text: Timeout (20000[ms]) to receive MUDPI topic_id: 2 (mapped to DDS topic: loop_data_02)
- [udp_receivers/rcv_01/receive_timeout] since: 2025-05-30 07:04:49.711908218 text: Timeout (20000[ms]) to receive UDP.
Limitations and Known Issues
It is assumed and thus not checked that the timestamp of all MUDPI frames for a particular sampleId is the same.