Release Notes

Summary

This release includes the following artefacts:

• RTC Toolkit (rtctk) - 5.0.1

And depends on the following:

• ELT Development Environment (ELT DevEnv) - v6.1.0

• Middleware Abstraction Layer (CII MAL) - v5.0.0

• Common Integration Infrastructure Services (CII SRV) - v5.0.0

• DDS Middleware (FastDDS) - v3.3.0

• Application Framework (RAD) - v6.2.0

• Roadrunner (numapp, ipcq and perfc) - v0.9.0

• Data Display Tool (DDT) - v1.2.1

• Control UI Toolkit (CUT) - v3.1.6

• STD Interface (stdif) - v4.0.0

• METADAQ Interface (metadaqif) - v3.0.0

• Low-latency Networking Library (llnetio) - v0.5.0

• TAI Clock - v0.7.0

• Msgsend - v0.8.2

• Extensible Test Runner (etr) - v3.3.0

What is the purpose of this release?

The purpose of this release is to provide a working version of the most important toolkit components for users to:

• Get a feel for how the RTC Toolkit, its modules and APIs work.

• Get familiar with core concepts, technologies and workflows.

• Facilitate feedback from developers to correct issues and improve the toolkit.

What is the scope of this version?

This release does not yet cover the full RTC Toolkit functionality.

Fig. 2 Availability of various software artefacts.

Assets that are not yet delivered with this version are depicted without a colored frame in Fig. 2. Assets that are part of this release, but are not yet feature-complete, are depicted with an orange frame. Assets that are part of this release and also considered feature-complete are depicted with a green frame.

The most important features of this release are:

Component Framework

• Framework for RTC Components

• Persistent Configuration Repository interface

• Runtime Configuration Repository interface

• Online Database interface

• Service Discovery interface

• Logging integration

• Recording Infrastructure

• Event channel service

• Component metrics service

• Alert service

Reusable Components

• RTC Supervisor

• Telemetry Republisher

• Telemetry Subscriber

• Telemetry Recorder

• Metadata Collector

• Data Task libraries

• DDT Server

• Deployment Daemon

Examples and Tutorials

• Creating bare RTC Components

• Customising reusable components

• Creating a minimalistic SRTC system

• Creating a distributed minimalistic SRTC system

Other Features

• CII MAL integration

• CII OLDB integration

• CII configuration service integration

• Optional CUDA support in waf

• Python support for Data Tasks

• Tools for configuration, control and monitoring

• Supporting tools (for testing and troubleshooting)

Documentation

• User’s manual (using Sphinx)

• API documentation (using Doxygen)

Limitations and Constraints

This version provides the core functionality to implement SRTC components that form part of a Telemetry Data Path. However, not all features are yet available or optimised.

• There are still some limitations when using the Persistent Configuration Repository. See section Limitations and Known Issues in the Persistent Configuration Repository for details.

• The Configuration Tool has some issues related to logging and error reporting. See section Limitations and Known Issues in the Configuration Tool for details.

• The Control and Monitoring Tool has some isses and some functionality is still missing. In addition, there are some problems related to startup, error reporting and logging. See section Limitations and Known Issues in the Control and Monitoring Tool for details.

• RTC Supervisor has limitations for generating the overall system status, user extensibility, error recovery and command abortion. See section Limitations and Known Issues in the RTC Supervisor for details.

• Telemetry Recording is not yet fully optimised. Only certain output formats are supported and certain types are missing. In addition, there are currently some naming limitations and timing constraints. See section Limitations and Known Issues in the Telemetry Recorder for details.

• The Deployment Daemon can enter an intermediate state requiring manual intervention during a failed deployment. See section Limitations and Known Issues in the Deployment Daemon for details.

• Possible rare deadlocks in the OldbAdapter. See section Limitations and Known Issues in the Runtime Configuration Repository for details.

• The Waf build system currently has problems handling transitive dependencies in cases where a mix of wdep and pkg-config dependencies are provided, such as the case in the RTC Toolkit. This means that many transitive dependencies from upstream projects need to be manually added in down-stream projects that build against the toolkit. In most cases, adding the following dependency checks for pkg-config dependencies to the configure method in your project’s wscript file will resolve the dependency issues:

  pkgs = (
      "CLI11",
      "log4cplus",
      "cpp-netlib-uri",
      "CCfits",
      "cfitsio",
      "yaml-cpp",
      "backtrace",
      "numa",
      "fmt",
      "benchmark",
      "openblas",
      "libpcap",
      "ppconsul",
      "llnetio",
      "ipcq",
      "numapp",
      "perfc",
      "tbb",
  )
  for pkg in pkgs:
      cnf.check_cfg(package=pkg, uselib_store=pkg, args="--cflags --libs")
  

  However, the dependencies can change in upstream projects. Therefore, if the above workaround is still not enough, look at the top level wscript file in the RTC Toolkit source code to identify additional missing dependencies. You should be able to mostly copy the relevant code in the configure method from that file.

Disclaimer

ESO does not warrant that the provided functions of the RTC Toolkit will meet all requirements or that the operation of the components and libraries will be flawless.

ESO does not ensure that solutions included in this version of the RTC Toolkit are not subject to changes in future releases. The future upgrade to newer versions of ICS Framework, Core Integration Infrastructure (CII) and adaptation to CCS development standards may introduce significant modifications to the actual interfaces and services.

While every precaution has been taken in the development of the RTC Toolkit software and in the preparation of the documentation, ESO assumes no responsibility for errors or omissions, or for damage resulting from the use of the software or of the information contained in the documentation.

Note

The RTC Toolkit is distributed outside ESO for the development of applications related to the ELT Project and ruled by the “General Conditions of ESO Contracts”. Any other use is not permitted without prior authorization from ESO.

The rights of third party products, whose software is included for convenience in the development environment, are ruled by their copyright notice included in their software.

Dependencies

Development Environment ELT DevEnv

The ELT Linux Development Environment (DevEnv) comprises a collection of hardware, software procedures and tools for the development, testing and debugging of software components for the ELT. It has to support large-scale and long-term maintenance of software.

Further information regarding the ELT Development Environment can be found here:

• ELT DevEnv 6

• Developer Guide

Deployment

The ELT standard tools for deployment (which is the action of activating or deactivating software processes on remote server hardware) are based on Nomad and Consul. The toolkit integrates with Nomad and Consul, such that the end user does not need to interact with these directly. Instead, the user interacts with the deployment daemon to perform the deployment of the SRTC components (see Deployment Daemon), and with the service discovery mechanism for rudimentary name service support (see Service Discovery).

Core Integration Infrastructure CII

The RTC Toolkit makes use of the CII MAL (Middleware Abstraction Layer) for non-time-critical inter-component messaging. The two main mechanisms used are request/reply and publish/subscribe. The following documents provide details regarding CII MAL:

• MAL API

• MAL Mappings

• MAL ICD Generation

In addition, this version also supports:

• CII OLDB (Online Database)

• CII configuration service (Config-ng)

Note

The CII_LOGS environment variable is deprecated and no longer used. The cii-postinstall tool does not set this variable anymore.

DDS Middleware FastDDS

FastDDS is an open source implementation of OMG’s Data Distribution Service (DDS). It is a middleware framework standard for data exchange using a publish–subscribe pattern. It is used in the RTC Toolkit for reliable multicast telemetry data transfer, in particular between Telemetry Republisher and Telemetry Subscriber components.

FastDDS and related tools are provided as part of the ELT Development Environment.

Note

FASTRTPS_DEFAULT_PROFILES_FILE environment variable is not used anymore to specify the FastDDS QoS file for different parts of the system like Telemetry Republisher, Telemetry Subscriber and Event Channel. Please refer to the particular (component) section for details about how FastDDS QoS profiles are handled. It is advised not to set/use the environment variable anymore as it can cause the system to misbehave.

Rapid Application Development RAD

The RAD application framework is an ESO software product that enables the development of event-driven applications for the ELT. It is based on call-backs and state machines.

The RTC Toolkit mainly uses RAD to instantiate its state machines and handle internal events. More details about RAD can be found in its user manual.

Roadrunner

Roadrunner is a project containing independent and reusable C++ libraries for high performance applications.

RTC Toolkit uses the following features:

• ipcq - a shared memory queue library.

• NUMA++ - a library to help simplify application of NUMA policies.

• perfc - a library to implement performance counters.

These are essentially a set of APIs to interact with libnuma and certain pthread APIs to control memory policy, CPU affinity and scheduler. For details see the documentation provided in the source code for each roadrunner component.

Data Display Tool DDT

Data Display Tool is a collection of libraries and applications that shall be used in the scope of the ESO ELT for the transfer, display and manipulation of data coming from the telescope.

The DDT is grouped into the components Data Transfer, Data Visualisation, Image Handling and Python Components. The RTC Toolkit depends only on the Data Transfer library.

The DDT includes applications such as:

• ddtBroker - The Data Broker is a command line application which is used to transfer the data from the Publisher to the Subscriber either on the same or on different hosts. When running the setup on different hosts, a single Data Broker needs to be started on each of the hosts.

• ddtViewer - The DDT Standard Viewer is a reference implementation of a DDT Subscriber GUI using all of the existing DDT Widgets. The DDT Standard Viewer displays one Image Widget plus some auxiliary widgets that are connected to the Image Widget. Via the context menu of the Image Widget the user can access a number of DDT Dialogs, plus a File Open dialog, which allows for the loading of FITS files from disk.

For more information please refer to the User Manual:

• DDT User Manual

Control UI Toolkit (CUT)

The Control UI Toolkit enables ELT Developers to create graphical user interfaces using declarative MVC components, based on Qt and Taurus. It provides models, views and widgets that can quickly access the control system datapoints through their URIs.

The RTC Toolkit uses the CUT library to implement its GUI panels for the Configuration Tool and Control and Monitoring Tool.

STD Interface (stdif)

The STD Interface is the standard MAL interface implemented by every SRTC component.

METADAQ Interface (metadaqif)

The RTC Toolkit uses the METADAQ Interface in implementations of the Metadata Collector component.

Low-latency Networking Library (llnetio)

The Low-latency Networking Library is used in the RTC Toolkit for MUDPI communication.

TAI Clock

The TAI Clock library is a C++ library which provides a std::chrono compatible clock counting SI seconds since 1970-01-01T00:00:00 TAI.

Msgsend

The RTC Toolkit uses the libraries in Msgsend for sending commands to SRTC components from the Python based clients in cases where the MAL interface must be deduced dynamically.

Extensible Test Runner etr

The RTC Toolkit makes use of the Extensible Test Runner for integration testing.

The following features are available:

• Run Robot Framework tests with the robot plugin.

• Request test resources with the resources plugin.

• Modify template files with Jinja2 template engine and the jinja2 plugin.

• Deploy software with Nomad with the nomad plugin (experimental support).

For details see the etr user manual.

Porting Instructions

• The Telemetry Republisher configuration has been simplified:

  • The dds topics configurations need to be moved under corresponding receiver under: dds_topics:.

  • There is no more need to map dds topics to the corresponding receiver, so rcv has to be removed.

  • The name of receivers and DDS topics can be arbitrary (not any more rcv_XXX and topic_YYY), but they need to obey snake case format.

  • In case of DDS the keyword is used also as topic name, so no need for name anymore.

  • Simulated dds topics need to be specified under: simulated_dds_topics:.

  • If you change the DDS topic name (at least to snake case) do not forget to change the name in the Telemetry subscriber as well.

  • dds_* configuration data points have been moved under dds: node. E.g.:

  dds_qos_profile:
  dds_interface_white_list:
  ...
  

  has to be converted to:

  dds:
    qos_profile:
    interface_white_list:
  ...
  

• The Telemetry Subscriber global dds configuration was aligned with the Telemetry Republisher:

  • The dds_* configuration data points have been moved under dds: node, except the dds_topics setting.

  • The dds_topics entry was converted from a list to a dictionary and each entry has a mandatory entry multicast_address.

  • The multicast_addresses setting was been removed. E.g.:

  dds_topics:
    - topic1
    - topic2
  multicast_addresses:
    - 224.0.0.1
    - 224.0.0.2
  dds_qos_profile:
  dds_interface_white_list:
  ...
  

  has to be converted to:

  dds_topics:
    topic1:
      multicast_address: !cfg.type:string "224.0.0.1"
    topic2:
      multicast_address: !cfg.type:string "224.0.0.2"
  dds:
    qos_profile:
    interface_white_list:
  ...
  

• The API of the TelemetrySubscriber for the Blender has changed. Specifically the DataSampleView structure:

  • The previous .data and .size fields (pointer and size) are replaced with a gsl::span in .data.

  • The .timestamp has also been changed to be a taiclock::TaiClock::time_point.

  • Custom Blender implementations will need to be adjusted to the new API.

  • The DataSamplesView uses a std::map instead of a vector. The map is indexed with the topic name, ensuring that the index is stable even when removing or adding topics.

• FastDDS QoS settings:

  • If the FastDDS QoS settings provided by the RTC Toolkit have been modified, or if alternative QoS configurations are used elsewhere, the payload of the DDS-agnostic topic is changed to unbounded (i.e., no maximum size). As a result, the PREALLOCATED history memory policy QoS is no longer supported. Instead, use PREALLOCATED_WITH_REALLOC (the default) or DYNAMIC_REUSABLE.

  • If there are problems reading custom DDS QoS files, please check the FastDDS documentation for renamed attributes. Some QoS attributes were renamed from camelCase to snake_case notation, e.g. heartbeat_period `instead of `heartbeatPeriod.

• The MUDPI publisher component configuration structure has changed:

  • period configuration has been moved under dynamic and renamed to period_us to specify period in micro second instead nano seconds.

  • from statc under dynamic have been moved also: sample_size, max_samples, sample_id_increment, data_file, as well as publisher specific configuration sample_size and payload_pattern.

• Runtime Repository:

  • a new default implementation was added that brings its own server process and is therefore no longer based on CII OLDB.

  • the server process is started as part of a deployment set using the Deployment Daemon, so additional entries need to be added to the deployment section (deployment.yaml) of each deployment set:

    service_list: !cfg.type:vector_string
      - rtr
    
    rtr:
      node: !cfg.type:string srtc_gw
      type: !cfg.type:string runtime_repository
      user: !cfg.type:string eltdev
    

• rtctkDeploymentGen command line tool:

  • services command: RTR (2nd) argument has been removed as the Runtime Repository is deployed by the Deployment Daemon.

  • job command: it is not necessary to provide additional arguments for the component name (–cid) and Service Discovery Endpoint (–sde), after the –.

  • Service Discovey Endpoint can be provided as –sde option (if different than default one).

  • job command: when rtctkDeploymentGen is used for generating Nomad Job file for the deployment daemon (option –deployment-daemon is specified) it is not necessary to specify the command (executable) anymore. E.g.

    rtctkDeploymentGen --sde consul://127.0.0.1:999 --node deployment_daemon_host --user eltdev job --deployment-daemon deployment_daemon
    

• File-based Service Discovery:

  • the YAML format of the service_disc.yaml files has been changed to use the config-ng format.

    common:
        persistent_repo_endpoint: !cfg.type:string MY_PSR_ENDPOINT
        runtime_repo_endpoint: !cfg.type:string MY_RTR_ENDPOINT
        oldb_endpoint: !cfg.type:string MY_OLDB_ENDPOINT
    my_component:
        req_rep_endpoint: !cfg.type:string zpb.rr://MY_IP:MY_PORT/
        pub_sub_endpoint: !cfg.type:string zpb.ps://MY_IP:MY_PORT/
    

See also the Migration Guide section in the CUT user manual for porting instructions related to custom GUIs.