Putting Everything Together for a Minimalistic SRTC System

This tutorial explains how to create, configure and run a simple SRTC system consisting of multiple, collaborating RTC Components.

To make life easier for developers, a working example is provided with the RTC Toolkit source code in the directory _examples/exampleEndToEnd.

The following components are used in this end-to-end example:

• Nomad and Consul services.

• CII OLDB as the backend for the Online Database (OLDB).

• CII Config-ng service (in local file mode) for the Persistent Configuration Repository (PSR).

• Deployment Daemon.

• RTC supervisor.

• RTC (example) components.

Note

The example was created to demonstrate how the RTC Toolkit works. The example is set up and configured so that it can be run on a single (virtual) development machine. The example is not optimised for throughput. Special configuration and additional code optimisation are necessary to use it in a real-scale RTC.

The end to end examples consists of the following component types:

Executable	Component Type	Description
rtctkDeploymentDaemon	Deployment Daemon	Takes care of deploying/starting of the other components.
rtctkExampleE2eRtcSup	RTC Supervisor	Guiding state of other components in response to commands and publishing global state in OLDB.
rtctkExampleHrtcSup	RTC Supervisor	A minimal HRTC Supervisor example application simulating interaction with an HRTC.
rtctkExampleMudpiPub	MUDPI Publisher	MUDPI samples generator to simulate MUDPI telemetry coming from HRTC.
rtctkTelRepub	Telemetry Republisher	Acquiring MUDPI frames and publishing Agnostic DDS Samples.
rtctkExampleE2eTelSub	Telemetry Subscriber	Receiving Agnostic DDS samples and storing Example Topic samples in shared memory.
rtctkExampleDataTaskTelemetry	Data Task	Retrieving Example Topic samples from shared memory, processing them and storing results in RTR.
rtctkExampleTelRec	Telemetry Recorder	Retrieving Example Topic samples from shared memory and storing them in FITS files.

The components realise a simple Telemetry Data Path and are being supervised by an RTC Supervisor. Sequencer functionality is being emulated using a shell script.

The system makes use of most of the components and libraries provided by the RTC Toolkit and introduces additional supporting tools like data sources, etc.

The example is comprised of the following Waf modules:

• config - Contains example persistent configuration and a Nomad config drop-in file.

• scripts - Helper script for provisioning, deployment and state control.

• rtcSup, hrtcSup, mudpiPub, telSub - Custom component source code for this example

Prerequisites

• Nomad and Consul must be configured and running.

• The CII OLDB and the CII configuration services (CII Config-ng) must be running.

Running the Example

After installing the RTC Toolkit (see Installation), the executables of the working example should be present in $INTROOT/bin and they should already be globally available via $PATH.

To run the example, first ensure that CII services, Nomad agents and Consul agents are started (see Installation).

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the SRTC Gateway and Computation node is depicted below. This configuration runs a server/client agent.

A simple example configuration for the Nomad agent for running the example on a single node could look as follows:

data_dir = "/opt/nomad/data"

bind_addr = "127.0.0.1"

advertise {
http = "127.0.0.1"
rpc  = "127.0.0.1"
serf = "127.0.0.1"
}

server {
bootstrap_expect = 1
enabled = true
}

client {
enabled = true
servers = ["127.0.0.1:4647"]

options = {
    "driver.raw_exec.enable" = "1"
}

template = {
    "disable_file_sandbox" = true
}

#It is important to have defined meta/node
meta {
    "node" = "hrtc-gw,srtc1"
}
}

The file containing details on how to start Nomad and Consul can be found in the example’s folder in: config/nomad_consul/simple-cfg.hcl.

Note

It is assumed that the example is run as user eltdev. In case it needs to be run as another user, the user name in the example shell script and the deployment config need to be adjusted accordingly.

The example can be executed using the following command:

$ rtctkExampleEndToEnd.sh run

The command will bring up all components and let them run for about 30 seconds, then it will terminate gracefully.

For more flexible use, developers can step manually through the example using the following sequence of commands:

# Deploy and start the example applications
rtctkExampleEndToEnd.sh provision

where Persistent Configuration Repository and OLDB are prepared, and registered to Service Discovery (Consul). In addition, the Deployment Daemon is started.

rtctkExampleEndToEnd.sh deploy

where the Deployment Daemon deploys the system, i.e. starts components and the Runtime Configuration Repository using deployment information from the Persistent Configuration Repository.

# Use the client to step through the life-cycle of the respective component instance
rtctkExampleEndToEnd.sh send rtc_sup Init
rtctkExampleEndToEnd.sh send rtc_sup Enable

where RTC Supervisor guides the deployed components to Init and then to the Enable state.

rtctkExampleEndToEnd.sh send hrtc_sup Run
rtctkExampleEndToEnd.sh start_data

where we start sending data - MUDPI samples.

Note

Here, the HRTC Supervisor is only send to Running state for symbolic reason. In a real RTC, the HRTC Supervisor would tell the HRTC to start sending telemetry.

Afterwards, we need to manually send components to the Running state using the following commands:

rtctkExampleEndToEnd.sh send tel_repub_1 Run
rtctkExampleEndToEnd.sh send tel_sub_1 Run
rtctkExampleEndToEnd.sh send data_task_1 Run
rtctkExampleEndToEnd.sh send tel_rec_1 Run

We can tell the HRTC Supervisor to symbolically close and open the loop:

rtctkExampleEndToEnd.sh send hrtc_sup Close
rtctkExampleEndToEnd.sh send hrtc_sup Open

We can stop the example using the following sequence of commands, which brings components to the Idle state:

# Follow how components work
rtctkExampleEndToEnd.sh send tel_rec_1 Idle
rtctkExampleEndToEnd.sh send data_task_1 Idle
rtctkExampleEndToEnd.sh send tel_sub_1 Idle
rtctkExampleEndToEnd.sh send tel_repub_1 Idle

and stop sending the data:

rtctkExampleEndToEnd.sh send hrtc_sup Idle
rtctkExampleEndToEnd.sh stop_data

Note

Here, the HRTC Supervisor is only send to Idle state for symbolic reason. In a real RTC, the HRTC Supervisor would tell the HRTC to stop sending telemetry.

and ask RTC Supervisor to Disable and afterwards Reset all the components:

rtctkExampleEndToEnd.sh send rtc_sup Disable
rtctkExampleEndToEnd.sh send rtc_sup Reset

finally, we undeploy and unprovision the example:

rtctkExampleEndToEnd.sh undeploy

after this, only the Deployment Daemon and the services remain registered

rtctkExampleEndToEnd.sh unprovision

unprovision also terminates the deployment daemon and unregisters the services.

Development Guide

This section explains how to configure multiple RTC Components to work in concert.

For information on how to develop individual or custom RTC Components refer to section Creating a Simple RTC Component.

All YAML files mentioned in the following sections refer to files that can be found under _examples/exampleEndToEnd/config/resource/config/rtctk/exampleEndToEnd. These are then written to the configuration service (CII Config-ng) when rtctkExampleEndToEnd.sh provision is run.

Configuring the example SRTC

The initial configuration files for populating the Persistent Configuration Repository are archived with the code in the following location: _examples/exampleEndToEnd/config/resource/config/rtctk/exampleEndToEnd/initial.

A detailed description of the config file syntax and available options can be found in the CII Configuration manual.

Configuring the MUDPI Publisher

The rtctkExampleMudpiPub component is used as a MUDPI telemetry data source. It is configured as source of three topics.

The MUDPI Topic id, sample size, frames per sample, destination host, port and multicast group are specified in the mudpi_pub_1.yaml configuration file as follows:

static:
    publishers:
        slopes:
            enabled: !cfg.type:boolean true
            network_adapter_ip: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 0
            buffer_size: !cfg.type:uint32 8972
        intensities:
            enabled: !cfg.type:boolean true
            network_adapter_ip: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 1
            buffer_size: !cfg.type:uint32 8972
        commands:
            enabled: !cfg.type:boolean true
            network_adapter_ip: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 2
            buffer_size: !cfg.type:uint32 8972
dynamic:
    period_us: !cfg.type:int64 100000
    publishers:
        slopes:
            sample_size: !cfg.type:uint32 36928
        intensities:
            sample_size: !cfg.type:uint32 18464
        commands:
            sample_size: !cfg.type:uint32 25264

Configuring the Telemetry Republisher

The rtctkTelRepub configuration is in the tel_repub_1.yaml file.

Here we have one MUDPI receiver named receiver_wfs receiving three MUDPI topics, which are republished to corresponding DDS topics.

static:
    mudpi_receivers:
        receiver_wfs:
            network_adapter_ip: !cfg.type:string 127.0.0.1
            port: !cfg.type:uint16 50000
            multicast_group: !cfg.type:string 225.4.4.4
            thread_policies:
                cpu_affinity: !cfg.type:string 1
            dds_topics:
                slopes_topic:
                    enabled: !cfg.type:boolean true
                    mudpi_topic: !cfg.type:int32 0
                intensities_topic:
                    enabled: !cfg.type:boolean true
                    mudpi_topic: !cfg.type:int32 1
                commands_topic:
                    enabled: !cfg.type:boolean true
                    mudpi_topic: !cfg.type:int32 2

More information on how to configure the Telemetry Republisher can be found in Telemetry Republisher and Telemetry Republisher Example.

Configuring the Telemetry Subscriber

This tutorial uses the rtctkExampleE2eTelSub executable, a custom Telemetry Subscriber for this end-to-end example.

Specification of the DDS topics to be acquired:

static:
    # ...
    dds_topics:
        slopes_topic:
            enabled: !cfg.type:boolean true
            multicast_address: !cfg.type:string "225.3.2.2"
        intensities_topic:
            enabled: !cfg.type:boolean true
            multicast_address: !cfg.type:string "225.3.2.2"
        commands_topic:
            enabled: !cfg.type:boolean true
            multicast_address: !cfg.type:string "225.3.2.2"

Specification of the shared memory topic to be stored:

static:
    # ...
    shm_topic_name: !cfg.type:string scao_loop

The shared memory topic is populated from the Agnostic DDS topic using blender code coming from _examples/exampleEndToEnd/telSub/src/main.cpp:RtcComponentMain:blender.

More information on how to configure the Telemetry Subscriber can be found in Telemetry Subscriber and Customise a Telemetry Subscriber.

Configuring the Data Task

This tutorial re-uses the rtctkExampleDataTaskTelemetry executable from the Data Task tutorial, but uses a different configuration file data_task_1.yaml.

static:
    queue_name: !cfg.type:string scao_loop
    samples_to_read: !cfg.type:uint32 5
    samples_to_skip: !cfg.type:uint32 10
    sample_timeout: !cfg.type:uint32 5000
    slopes: !cfg.type:uint32 9232
    modes: !cfg.type:uint32 50
dynamic:
    slopes2modes: !cfg.type:vector_single [0.0]
    avg_slopes: !cfg.type:vector_single [0.0]
    avg_modes: !cfg.type:vector_single [0.0]

Note that it is up to the instrument RTC developers to put information such as the shared memory queue name into each data task config file or into a common file that can be shared between components.

Also note that the input vector slopes2modes only has a dummy value here; the actual value is computed in the bash script. We did this to avoid committing large FITS files to the rtctk repo.

Additional information about the Data Task can be found in Data Task and Creating a Custom Data Task.

Configuring the Telemetry Recorder

This tutorial re-uses the rtctkExampleTelRec executable from the Telemetry Recorder tutorial, but uses a different configuration file tel_rec_1.yaml.

static:
  rec_units:
    ipcq:
      shm_queue_name: !cfg.type:string scao_loop
    event:
      topic_name: !cfg.type:string configuration_topic
dynamic:
  session_id: !cfg.type:string 'session_1'
  rec_units:
    ipcq:
      subsample_factor: !cfg.type:int64 '10'
      start_at_sample_id: !cfg.type:int64 '0'
      stop_after_num_samples: !cfg.type:int64 '10'
    repo:
      datapoint_list:
        idx_00:
          pathname: !cfg.type:string  /data_task_1/dynamic/avg_slopes
        idx_01:
          pathname: !cfg.type:string  /data_task_1/dynamic/avg_modes

Note that it is up to the instrument RTC developers to put information such as the shared memory queue name into each config file or into a common file that can be shared between components.

Additional information about the Telemetry Recorder can be found in Telemetry Recorder and Creating a Custom Telemetry Recorder.

Configuration of the RTC Supervisor

The RTC Supervisor is configured with the list of the components to be supervised. The configuration of this list is coming from the Runtime Configuration Repository datapoint /deploy/component_list, which is normally created and set by the Deployment Daemon during the deployment process where information is retrieved from the Persistent Configuration Repository deployment.yaml file:

component_list: !cfg.type:vector_string
    - rtc_sup
    - tel_repub_1
    - tel_sub_1
    - data_task_1
    - tel_rec_1
    - mudpi_pub_1

The “component_list” defines the list of the components to be supervised and, implicitly, the order in which state transitions will be performed if the corresponding <transition>_alone flag is set in “rtc_sup.yaml”. The component names are used to retrieve the command request/response URIs and the URIs of the state publication from the Service Discovery.

More information regarding the RTC Supervisor can be found in the RTC Supervisor section.

Configuration of the HRTC Supervisor

As there is no real HRTC present, the HRTC Supervisor only holds minimal and symbolical configuration. For the future, we plan to extend this HRTC Supervisor example to make it more realistic.

static:
    hrtc:
        ip: !cfg.type:string 127.0.0.1
        port: !cfg.type:uint16 8000

Distributed Scenario

Although the provided example is written to be run on a single machine, it should be relatively easy to adopt it for running on many machines. Let’s assume two machines:

Node	Nomad meta node	IP address	Description
Simple SRTC Gateway and Computation	srtc1	10.0.0.11	Machine where supervisor, telemetry subscriber and data task run.
Simple HRTC Gateway	hrtc-gw	10.0.0.12	Machine where telemetry republishing and MUDPI publisher run.

Both nodes need to be configured to:

• Remotely access CII services, in particular OLDB.

• Run Nomad agents.

• Run Consul agents.

CII Services

In regards to CII, if services are to be hosted in srtc1, then execute as root:

# sudo cii-postinstall groupserver 10.0.0.11
# cii-services start all

and in the hrtc-gw machine, we need to configure it to reference the srtc1 node instead of deploying another set of services. Execute as root:

# sudo cii-postinstall groupclient 10.0.0.11

More details on CII post installation and roles can be found at CII Manual.

Consul and Nomad

Nomad and Consul process are called agents. Agents can run as client, server or both at the same time. For a minimal distributed setup, we need one consul server, one nomad server, one nomad client per host and one consul client per host.

The srtc1 host runs agents as Consul server/client and Nomad server/client. All other hosts (in this case hrtc-gw) should run Consul as client, and Nomad as client.

Warning

Since nodes in a deployment environment may involve many network interfaces, the way in which different interfaces are specified in the following configuration files may be altered in the future.

Note

Nomad and Consul shall use the management network for their communication.

Consul

Below we provide the configuration (/etc/consul.d/consul.hcl) for the Consul service in the SRTC Gateway and Computation node. This configuration runs a server/client agent. Please notice that the “advertise_addr” entry cannot be the host name, i.e. it needs to be the actual IP address.

datacenter = "dc1"
data_dir = "/opt/consul"
server = true
ui = true
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
#   should be an IP address on the management network
advertise_addr = "10.0.0.11"
bootstrap = true
bootstrap_expect = 1

The Consul service configuration (/etc/consul.d/consul.hcl) for the HRTC Gateway is indicated below. This configures the agent to run only as client. Any other additional client node should have a Consul configuration that looks the same. Please notice that the “bind_addr” entry cannot be the host name, i.e. it needs to be the actual IP address.

datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "10.0.0.12"
retry_join = ["10.0.0.11"]

Nomad

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the SRTC Gateway and Computation node is depicted below. This configuration runs a server/client agent.

datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
# all communication shall use management network
advertise {
  # address for web browser access to Nomad Server
  http = "10.0.0.11"
  rpc  = "10.0.0.11"
  serf = "10.0.0.11"
}
server {
  enabled = true
  bootstrap_expect = 1
}
client {
  enabled = true
  #   all "servers" should be on the management network
  servers = ["10.0.0.11"]
  #  should be used interface that can communicate with  "servers" i.e. 10.0.0.11
  network_interface = "enp0s8"
  meta {
      "node" = "srtc1"
  }
  options = {
    "driver.raw_exec.enable" = "1"
  }
  template = {
    "disable_file_sandbox" = true
  }
}

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the HRTC Gateway node is shown below. This configures the agent to run only as client.

datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
advertise {
  # address for web browser access to Nomad Client
  http = "10.0.0.12"
  rpc  = "10.0.0.12"
  serf = "10.0.0.12"
}
client {
  enabled = true
  servers = ["10.0.0.11"]
  # should be used interface that can communicate with "servers" i.e. 10.0.0.11
  network_interface = "enp0s8"
  meta {
      "node" = "hrtc-gw"
  }
  options = {
    "driver.raw_exec.enable" = "1"
  }
  template = {
    "disable_file_sandbox" = true
  }
}
consul {
  address = "127.0.0.1:8500"
}

In both Nomad service configuration files, the following entries may need to be changed:

• client->meta->node allows to identify the particular node and may be used to set restrictions in jobs started by the Deployment Daemon.

• client->network_interface indicates from which network interface the NOMAD_IP address is determined. This IP address is used by the rtctkDeploymentGen job tool to create the services URIs entries. If not present, Nomad will try to calculate an appropriate interface, but the results may vary from machine to machine.

Service Discovery Endpoint

The Service Discovery Endpoint (SDE) is used by every RTC Component, Reusable Component, or client to locate bootstrap data. Therefore, every tool in the RTCTK needs it.

The SDE (while in the hosts that have a consul agent deployed) can be

• consul://10.0.0.11:8500 or

• consul://127.0.0.1:8500 (also used as default if --sde is not provided)

If the user is executing a client tool that is outside of the cluster, please use:

• consul://10.0.0.11:8500

Running the Example

As the deployment configuration (deployment.yaml) already considers those two nodes (hrtc-gw and srtc1) there is nothing to be changed there.

Run as user eltdev in the srtc1 node the following command:

$ rtctkExampleEndToEnd.sh run