Putting Everything Together for a Minimalistic SRTC System

This tutorial explains how to create, configure and run a simple SRTC system consisting of multiple, collaborating RTC Components.

To make life easier for developers, a working example is provided with the RTC Toolkit source code in the directory _examples/exampleEndToEnd.

Note

The example in _examples/exampleEndToEnd is only provided with the source code. It is not installed when installing using RPM packages.

The following components are used in this end-to-end example:

Nomad and Consul services.
CII OLDB as the backend for both the Runtime Configuration Repository (RTR) and for the Online Database (OLDB).
CII Config-ng service (in local file mode) for the Persistent Configuration Repository (PSR).
Deployment Daemon.
RTC supervisor.
RTC (example) components.

Note

The example was created to demonstrate how the RTC Toolkit works. The example is set up and configured so that it can be run on a single (virtual) development machine. The example is not optimised for throughput. Special configuration and additional code optimisation are necessary to use it in a real-scale RTC.

The end to end examples consists of the following component types:

Executable	Component Type	Description
rtctkMudpiPub	MUDPI Publisher	MUDPI samples generator.
rtctkTelRepub	Telemetry Republisher	Acquiring MUDPI frames and publishing Agnostic DDS Samples.
rtctkExampleTelSub	Telemetry Subscriber	Receiving Agnostic DDS samples and storing Example Topic samples in shared memory.
rtctkExampleDataTaskTelemetry	Data Task	Retrieving Example Topic samples from shared memory, processing them and storing results in RTR.
rtctkRtcSupervisor	RTC Supervisor	Guiding state of other components in response to commands and publishing global state in RTR.
rtctkDeploymentDaemon	Deployment Daemon	Deployment Daemon takes care of deploying/starting of the other components.

The components realise a simple Telemetry Data Path and are being supervised by an RTC Supervisor. Sequencer functionality is being emulated using a shell script.

The system makes use of most of the components and libraries provided by the RTC Toolkit and introduces additional supporting tools like data sources, etc.

The examples are comprise of the following Waf modules:

resource/config - Contains configuration of the example system.
scripts - Helper script for deployment and control.
nomad_consul - Nomad and Consul example configuration.

Prerequisites

Nomad and Consul must be running.

The CII OLDB and the CII configuration services (CII Config-ng) must be running.

Running the Example

After installing the RTC Toolkit (see Installation), the executables of the working example should be present in $INTROOT/bin and they should already be globally available via $PATH.

To run the example, first ensure that CII services, Nomad agents and Consul agents are started (see Installation).

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the SRTC Gateway and Computation node is depicted below. This configuration runs a server/client agent.

A simple example configuration for the Nomad agent for running the example on a single node could look as follows:

data_dir = "/opt/nomad/data"

bind_addr = "127.0.0.1"

advertise {
http = "127.0.0.1"
rpc  = "127.0.0.1"
serf = "127.0.0.1"
}

server {
bootstrap_expect = 1
enabled = true
}

client {
enabled = true
servers = ["127.0.0.1:4647"]

options = {
    "driver.raw_exec.enable" = "1"
}

template = {
    "disable_file_sandbox" = true
}

#It is important to have defined meta/node
meta {
    "node" = "hrtc-gw,srtc1"
}
}

The file containing details on how to start Nomad and Consul can be found in the example’s folder in: config/nomad_consul/simple-cfg.hcl.

Note

When we want to run the example (more specifically, the components from the example) as a different user, e.g. eltdev, rather than the Nomad agent user, e.g. root, as is the case when the agent runs as a system service, we need to provide the user name, e.g. eltdev, to the script rtctkExampleEndToEnd.sh with the -u option. This needs to be specified for commands: provision, deploy and run. We assume this scenario in the rest of the document.

The example can be executed using the following command:

$ rtctkExampleEndToEnd.sh -u eltdev run

The command will bring up all components and let them run for about 30 seconds, then it will terminate gracefully.

For more flexible use, developers can step manually through the example using the following sequence of commands:

# Deploy and start the example applications
rtctkExampleEndToEnd.sh -u eltdev provision

where Persistent Configuration Repository, Runtime Configuration Repository and OLDB are prepared, and registered to Service Discovery (Consul).

rtctkExampleEndToEnd.sh -u eltdev deploy

where Deployment Daemon is started and it deploys the system, e.g. starts components from information from the Persistent Configuration Repository.

# Use the client to step through the life-cycle of the respective component instance
rtctkExampleEndToEnd.sh send rtc_sup Init
rtctkExampleEndToEnd.sh send rtc_sup Enable

where RTC Supervisor guides the deployed components to Init and then to the Enable state.

rtctkExampleEndToEnd.sh start_data

where we start sending data - MUDPI samples. Afterwards, we need to manually send components to the Running state using the following commands:

rtctkExampleEndToEnd.sh send tel_repub_1 Run
rtctkExampleEndToEnd.sh send tel_sub_1 Run
rtctkExampleEndToEnd.sh send data_task_1 Run

We can stop the example using the following sequence of commands, which brings components to the Idle state:

# Follow how components work
rtctkExampleEndToEnd.sh send data_task_1 Idle
rtctkExampleEndToEnd.sh send tel_sub_1 Idle
rtctkExampleEndToEnd.sh send tel_repub_1 Idle

and stop sending the data:

rtctkExampleEndToEnd.sh stop_data

and ask RTC Supervisor to Disable and afterwards Reset all the components:

rtctkExampleEndToEnd.sh send rtc_sup Disable
rtctkExampleEndToEnd.sh send rtc_sup Reset

and in the end we stop the source of data, undeploy and unprovision the example:

# Gracefully terminate the applications and clean-up
rtctkExampleEndToEnd.sh undeploy
rtctkExampleEndToEnd.sh unprovision

Development Guide

This section explains how to configure multiple RTC Components to work in concert.

For information on how to develop individual or custom RTC Components refer to section Creating a Simple RTC Component.

All YAML files mentioned in the following sections refer to files that can be found under _examples/exampleEndToEnd/config/resource/config/rtctk/exampleEndToEnd. These are then written to the configuration service (CII Config-ng) when rtctkExampleEndToEnd.sh provision is run.

Configuration of the Telemetry Data Path

The data path of this example is configured by changing static and dynamic configuration keywords used by the components. The precise location of the keyword definitions depends on the implementation of the particular example; most of the keywords are set by the script rtctkExampleEndToEndPopulate.sh, you can change the values used by modifying this script or changing the parameters after the “deploy” stage has been performed.

The initial configuration files for populating the Persistent Configuration Repository are archived with the code in the following location: _examples/exampleEndToEnd/config/resource/config/rtctk/exampleEndToEnd/initial. The detailed description of the syntax and available options can be found in the CII Configuration manual.

Configuring the MUDPI Publisher

The rtctkMudpiPub component is used as a MUDPI telemetry data source. It is configured as source of three topics.

The MUDPI Topic id, sample size, frames per sample, destination host, port and multicast group are specified in the mudpi_pub_1.yaml configuration file as follows:

static:
    publishers:
        slopes:
            hostname: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 0
            buffer_size: !cfg.type:uint32 8972
        intensities:
            hostname: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 1
            buffer_size: !cfg.type:uint32 8972
        commands:
            hostname: !cfg.type:string 127.0.0.1
            multicast_group: !cfg.type:string 225.4.4.4
            port: !cfg.type:uint16 50000
            topic_id: !cfg.type:int32 2
            buffer_size: !cfg.type:uint32 8972
dynamic:
    period_us: !cfg.type:int64 100000
    publishers:
        slopes:
            sample_size: !cfg.type:uint32 36928
        intensities:
            sample_size: !cfg.type:uint32 18464
        commands:
            sample_size: !cfg.type:uint32 25264

Configuring the Telemetry Republisher

The rtctkTelRepub configuration is in the tel_repub_1.yaml file.

Here we have one MUDPI receiver named receiver_wfs receiving three MUDPI topics, which are republished to corresponding DDS topics.

static:
    mudpi_receivers:
        receiver_wfs:
            network_adapter_ip: !cfg.type:string 127.0.0.1
            port: !cfg.type:uint16 50000
            multicast_group: !cfg.type:string 225.4.4.4
            thread_policies:
                cpu_affinity: !cfg.type:string 1
            dds_topics:
                slopes_topic:
                    mudpi_topic: !cfg.type:int32 0
                intensities_topic:
                    mudpi_topic: !cfg.type:int32 1
                commands_topic:
                    mudpi_topic: !cfg.type:int32 2

More information on how to configure the Telemetry Republisher can be found in Telemetry Republisher and Telemetry Republisher Example.

Configuring the Telemetry Subscriber

This tutorial re-uses the rtctkExampleTelSub executable from the Telemetry Subscriber tutorial, but uses different configuration parameters.

Specification of the DDS topics to be acquired:

static:
    # ...
    dds_topics:
        slopes_topic:
            multicast_address: !cfg.type:string "225.3.2.2"
        intensities_topic:
            multicast_address: !cfg.type:string "225.3.2.2"
        commands_topic:
            multicast_address: !cfg.type:string "225.3.2.2"

Specification of the shared memory topic to be stored:

static:
    # ...
    shm_topic_name: !cfg.type:string scao_loop

The shared memory topic is populated from the Agnostic DDS topic using blender code coming from exampleTelsub/app/src/main.cpp:RtcComponentMain:blender().

More information on how to configure the Telemetry Subscriber can be found in Telemetry Subscriber and Customise a Telemetry Subscriber.

Configuring the Data Task

This tutorial re-uses the rtctkExampleDataTaskTelemetry executable from the Data Task tutorial, but uses a different configuration file data_task_1.yaml.

static:
    queue_name: !cfg.type:string scao_loop
    samples_to_read: !cfg.type:uint32 5
    samples_to_skip: !cfg.type:uint32 10
    sample_timeout: !cfg.type:uint32 5000
    slopes: !cfg.type:uint32 9232
    modes: !cfg.type:uint32 50
dynamic:
    slopes2modes: !cfg.type:vector_single [0.0]
    avg_slopes: !cfg.type:vector_single [0.0]
    avg_modes: !cfg.type:vector_single [0.0]

Note that it is up to the instrument RTC developers to put information such as the shared memory queue name into each data task config file or into a common file that can be shared between components.

Also note that the input vector slopes2modes only has a dummy value here; the actual value is computed in the bash script. We did this to avoid comitting large FITS files to the rtctk repo.

Additional information about the Data Task can be found in Data Task and Creating a Custom Data Task.

Configuration of the RTC Supervisor

The RTC Supervisor is configured with the list of the components to be supervised. The configuration of this list is coming from the Runtime Configuration Repository datapoint /deploy/component_list, which is normally created and set by the Deployment Daemon during the deployment process where information is retrieved from the Persistent Configuration Repository deployment.yaml file:

component_list: !cfg.type:vector_string
    - rtc_sup
    - tel_repub_1
    - tel_sub_1
    - data_task_1
    - mudpi_pub_1

The “component_list” defines the list of the components to be supervised and, implicitly, the order in which state transitions will be performed if the corresponding <transition>_alone flag is set in “rtc_sup.yaml”. The component names are used to retrieve the command request/response URIs and the URIs of the state publication from the Service Discovery.

More information regarding the RTC Supervisor can be found in RTC Supervisor and in Customising the RTC Supervisor.

Distributed Scenario

Although the provided example is written to be run on a single machine, it should be relatively easy to adopt it for running on many machines. Let’s assume two machines:

Node	Nomad meta node	IP address	Description
Simple SRTC Gateway and Computation	srtc1	10.0.0.11	Machine where supervisor, telemetry subscriber and data task run.
Simple HRTC Gateway	hrtc-gw	10.0.0.12	Machine where telemetry republishing and MUDPI publisher run.

Both nodes need to be configured to:

Remotely access CII services, in particular OLDB.
Run Nomad agents.
Run Consul agents.

CII Services

In regards to CII, if services are to be hosted in srtc1, then execute as root:

# /elt/ciisrv/postinstall/cii-postinstall role_groupserver 10.0.0.11
# cii-services start all

and in the hrtc-gw machine, we need to configure it to reference the srtc1 node instead of deploying another set of services. Execute as root:

# /elt/ciisrv/postinstall/cii-postinstall role_groupclient 10.0.0.11

More details on CII post installation and roles can be found at CII Manual.

Consul and Nomad

Nomad and Consul process are called agents. Agents can run as client, server or both at the same time. For a minimal distributed setup, we need one consul server, one nomad server, one nomad client per host and one consul client per host.

The srtc1 host runs agents as Consul server/client and Nomad server/client. All other hosts (in this case hrtc-gw) should run Consul as client, and Nomad as client.

Warning

Since nodes in a deployment environment may involve many network interfaces, the way in which different interfaces are specified in the following configuration files may be altered in the future.

Note

Nomad and Consul shall use the management network for their communication.

Consul

Below we provide the configuration (/etc/consul.d/consul.hcl) for the Consul service in the SRTC Gateway and Computation node. This configuration runs a server/client agent. Please notice that the “advertise_addr” entry cannot be the host name, i.e. it needs to be the actual IP address.

datacenter = "dc1"
data_dir = "/opt/consul"
server = true
ui = true
bind_addr = "0.0.0.0"
client_addr = "0.0.0.0"
#   should be an IP address on the management network
advertise_addr = "10.0.0.11"
bootstrap = true
bootstrap_expect = 1

The Consul service configuration (/etc/consul.d/consul.hcl) for the HRTC Gateway is indicated below. This configures the agent to run only as client. Any other additional client node should have a Consul configuration that looks the same. Please notice that the “bind_addr” entry cannot be the host name, i.e. it needs to be the actual IP address.

datacenter = "dc1"
data_dir = "/opt/consul"
bind_addr = "10.0.0.12"
retry_join = ["10.0.0.11"]

Nomad

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the SRTC Gateway and Computation node is depicted below. This configuration runs a server/client agent.

datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
# all communication shall use management network
advertise {
  # address for web browser access to Nomad Server
  http = "10.0.0.11"
  rpc  = "10.0.0.11"
  serf = "10.0.0.11"
}
server {
  enabled = true
  bootstrap_expect = 1
}
client {
  enabled = true
  #   all "servers" should be on the management network
  servers = ["10.0.0.11"]
  #  should be used interface that can communicate with  "servers" i.e. 10.0.0.11
  network_interface = "enp0s8"
  meta {
      "node" = "srtc1"
  }
  options = {
    "driver.raw_exec.enable" = "1"
  }
  template = {
    "disable_file_sandbox" = true
  }
}

The Nomad service configuration (/etc/nomad.d/nomad.hcl) for the HRTC Gateway node is shown below. This configures the agent to run only as client.

datacenter = "dc1"
data_dir = "/opt/nomad"
bind_addr = "0.0.0.0"
advertise {
  # address for web browser access to Nomad Client
  http = "10.0.0.12"
  rpc  = "10.0.0.12"
  serf = "10.0.0.12"
}
client {
  enabled = true
  servers = ["10.0.0.11"]
  # should be used interface that can communicate with "servers" i.e. 10.0.0.11
  network_interface = "enp0s8"
  meta {
      "node" = "hrtc-gw"
  }
  options = {
    "driver.raw_exec.enable" = "1"
  }
  template = {
    "disable_file_sandbox" = true
  }
}
consul {
  address = "127.0.0.1:8500"
}

In both Nomad service configuration files, the following entries may need to be changed:

client->meta->node allows to identify the particular node and may be used to set restrictions in jobs started by the Deployment Daemon.
client->network_interface indicates from which network interface the NOMAD_IP address is determined. This IP address is used by the rtctkDeploymentGen job tool to create the services URIs entries. If not present, Nomad will try to calculate an appropriate interface, but the results may vary from machine to machine.

Service Discovery Endpoint

The Service Discovery Endpoint (SDE) is used by every RTC Component, Reusable Component, or client to locate bootstrap data. Therefore, every tool in the RTCTK needs it.

The SDE (while in the hosts that have a consul agent deployed) can be

consul://10.0.0.11:8500 or
consul://127.0.0.1:8500 (also used as default if --sde is not provided)

If the user is executing a client tool that is outside of the cluster, please use:

consul://10.0.0.11:8500

Running the Example

As the deployment configuration (deployment.yaml) already considers those two nodes (hrtc-gw and srtc1) there is nothing to be changed there.

Run as user eltdev in the srtc1 node the following command:

$ rtctkExampleEndToEnd.sh -u eltdev run