Installation

This is a step by step guide on how to install the RTC Toolkit from scratch.

Machine Preparation

Install a real or virtual machine according to the ELT Linux Installation Guide.

Note

Use the default for the ELT_ROLE setting: ELT_ROLE=ELTDEV

Note

If you want to use a graphical display and not only a text based terminal, i.e. enable the display manager at system start, then also add the setting: ELT_DM=yes

Note

This version of the RTC Toolkit requires a machine with at least 8GB of RAM to compile.

ELT Development Environment

The RTC Toolkit is written and tested for the ELT Development Environment, which provides a common set of development tools and dependencies. Before installing the RTC Toolkit, you should become familiar with the tools available on this platform as described in the Guide to Developing Software for the ELT document.

Note

The version 6.1.0 of ELT Development Environment with platform 25.21.latest shall be used with version 5.1.0 of RTC Toolkit. This is the ELT Development Environment based on Fedora 40. This can be ensured by adding the version number explicitly when beginning the installation of the ELT Development Environment as follows:

# sh <(curl https://ftp.eso.org/pub/elt/repos/fedora/eso/GetPlatform.sh) -- 25.21.latest

Install RTC Toolkit and Dependencies

The RTC Toolkit and its dependencies are provided as RPM packages. These can be installed using DNF as root using the following command:

# dnf -y install elt-rtctk*

Account Configuration

Login as your user.
Create the directories for the installation and data areas:
```
$ cd <the location for introot>
$ getTemplate -d introot INTROOT

$ cd <the location for dataroot>
$ getTemplate -d dataroot DATAROOT
```
This will create two sub-directory trees underneath the chosen top level directories, one called INTROOT for the software installation and another called DATAROOT for data files. These directories are primarily used for end user development. The RTC Toolkit is typically not installed in these locations unless it is being built from source.
The environment must contain the definitions of the relevant environment variables such as INTROOT, PREFIX, etc. These environment variables will be automatically defined by means of an Lmod file private.lua, which in turn uses the system modulefile definitions in /elt/common/modulefiles/ to setup additional environment variables such as LD_LIBRARY_PATH and PYTHONPATH.

In your home directory create the private.lua file:
```
$ mkdir $HOME/modulefiles
$ vi $HOME/modulefiles/private.lua
```
Add the following initial content to the private.lua file:
```
local introot = "<introot location>"      -- put actual introot location here
local dataroot = "<dataroot location>"    -- put actual dataroot location here

setenv("INTROOT", introot)
setenv("PREFIX", introot)
setenv("DATAROOT", dataroot)

load("rtctk")
load("introot")
```
Update the CFGPATH environment variable with the location of the local Persistent Configuration Repository files.

Assuming the location for the local files is $DATAROOT/config/persistent_repo; for this example, one can setup the CFGPATH with the prepend_path command in the private.lua file as follows:
```
prepend_path("CFGPATH", dataroot .. "/config/persistent_repo")
```
Important

Make sure to use the prepend_path command and not just the setenv command in the Lmod file, because CFGPATH is treated similarly to the PATH or LD_LIBRARY_PATH environment variables, i.e. CFGPATH may already contain preconfigured paths that should not be removed, but only added to.
(Optional) Logs are forwarded to the CII logging facilities and human readable logs are written to $INTROOT/logsink by default. However, it may be convenient to write the human readable logs to a different location. In such a case, set the RTC_LOGS environment variable to point to an alternative directory.

For example, if the location for the human readable logs was chosen to be /data/logsink, RTC_LOGS is configured by appending the following to the private.lua file:
```
setenv("RTC_LOGS", "/data/logsink")
```
Please make sure to create the folder in the appropriate location and set the directory’s write permissions so that the user account, e.g. eltdev, is able to create and modify files there. Following the example above, this can be accomplished using the following commands (may be need to run them as root):
```
$ mkdir -p /data/logsink
$ chown eltdev /data/logsink
$ chmod u+r,u+w,u+x /data/logsink
```

Save the private.lua file.

The following is a complete minimal example of the file contents assuming that the user account is eltdev, the INTROOT and DATAROOT directories are in the home directory, and that the human readable logs should go to /data/logsink:

local introot = "/home_local/eltdev/INTROOT"
local dataroot = "/home_local/eltdev/DATAROOT"
setenv("INTROOT", introot)
setenv("PREFIX", introot)
setenv("DATAROOT", dataroot)
load("rtctk")
load("introot")
prepend_path("CFGPATH", dataroot .. "/config/persistent_repo")

Update the environment by logging out and back in again.

Important

Log out and then in again so that the modulefiles directory becomes known to the environment and the newly created private.lua is loaded. This is needed when the directory modulefiles and file private.lua are created for the first time.

File private.lua is loaded by default upon login. In case more .lua files (with different names) will be added to $HOME/modulefiles, they can be made known to the environment by running the following command:

$ module load

You can check which Lmod modules are available after login with:

$ module avail

The output should look similar to the following (the exact set of available/loaded modules might change with the software versions, but private and introot should be loaded as a minimum):

------------------------------- /home/vagrant/modulefiles -------------------------------
   private (L)

----------------------------- /elt/common/modulefiles/core ------------------------------
   eltdev (L)    introot (L)

---------------------------- /elt/common/modulefiles/default ----------------------------
   ciisrv (L)    ddt     (L)    mal       (L)    rad        (L)    rtctk (L)
   cut    (L)    elt-trs (L)    metadaqif (L)    roadrunner (L)    stdif (L)

------------------------- /usr/share/lmod/lmod/modulefiles/Core -------------------------
   lmod    settarg

  Where:
   L:  Module is loaded

If the avail list is too long consider trying:

"module --default avail" or "ml -d av" to just list the default modules.
"module overview" or "ml ov" to display the number of modules for each name.

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of
the "keys".

Note

For more information about Lmod refer to section 3.2 Environmental Modules System (Lmod) in the Guide to Developing Software for the ELT.

Configuration of Required Services

The required CII services need to be configured the first time after a fresh installation of the development environment. These CII services packages are no longer installed by default and they need to be installed as root using the following command:

# dnf -y install \
 elt-ciisrv-postinstall \
 elt-ciisrv-srv-oldb

CII postinstall

Configuring the CII is done by running the cii-postinstall tool. This must be run under the root account to succeed. In the following example the tool is run with the role_ownserver role that will prepare a simple configuration where the CII services are run on the local host:

# /elt/ciisrv/postinstall/cii-postinstall role_ownserver

Such a configuration is only appropriate for development and basic testing. See the CII Services Management documentation for details about using alternative roles for more advanced configurations where the CII services are run on dedicated remote servers.

Metadata Instances

Certain additional metadata instances specific to the RTC Toolkit need to be added to the CII configuration. This can be done by running the rtctkConfigTool binary that was freshly installed as follows:

$ rtctkConfigTool init-metadata

This command will return a zero exit code upon successful completion, or error messages will be printed if an error occurs. The command line option --log-level=DEBUG can be used for verbose output.

It is only necessary to run the above command once. Running it multiple times will simply overwrite any modifications that might have been made to the metadata instances.

Startup of Required Services

The RTC Toolkit makes use of a number of ELT supplied services, in particular those provided by CII, Nomad and Consul.

Ensure these services are correctly turned on before use and turned off when no longer required.

CII Service startup

The CII Service provides a command line tool (cii-services) to manage the services. These commands can be executed with the user eltdev on the ELT DevEnv release 6.1.0.

To start and stop the services use the following commands:

$ cii-services start all
$ cii-services stop all

The shutdown of the CII services using the command shown above can be performed when you no longer wish to exercise RTC Toolkit components. In general, the CII services should be left running.

Note

The complete set of CII services can be resource intensive. In cases where memory and CPU capacity is limited, for example, in smaller development virtual machines, it may be useful to bring up only the minimal services needed by the RTC Toolkit with the following command:

$ cii-services start oldb

Checking the status of the running services does not require root privileges and can be done using the following command (example output is also indicated):

$ cii-services info
CII Services Tool (20240307)

# install .................. .................
[oldb]    config-client-ini                              |install:
[oldb]    cii-oldb-default-redis   |active:yes  |boot:n  |install:/usr/bin/redis-server
[oldb]    cii-oldb-calc-daemon     |active:yes  |boot:n  |install:/elt/ciisrv/bin/srv-oldb-calculation
[oldb]    cii-oldb-calc-scheduler  |active:yes  |boot:n  |install:/elt/ciisrv/bin/srv-oldb-scheduler
[log]     directory                                      |install:/var/log/elt
[log]     rsyslog                  |active:yes  |boot:y  |install:/usr/sbin/rsyslogd
[log]     systemd-journald         |active:yes  |boot:y  |install:/usr/lib/systemd/systemd-journald
[log]     logrotate                |active:yes  |boot:y  |install:/usr/sbin/logrotate
[trace]   jaeger-all               |active:yes  |boot:n  |install:/usr/local/bin/jaeger-all-in-one
[telem]   srv-telemetry            |active:no   |boot:n  |install:
[alarm]   cii_ias                  |active:no   |boot:n  |install:
[alarm]   kafka                    |active:no   |boot:n  |install:
[alarm]   kafka-zookeeper          |active:no   |boot:n  |install:
[alarm]   cii_alarm_mon            |active:no   |boot:n  |install:
[deprec]  filebeat                 |active:no   |boot:n  |install:
[deprec]  logstash                 |active:no   |boot:n  |install:
[deprec]  kibana                   |active:no   |boot:n  |install:
[deprec]  elasticsearch            |active:no   |boot:n  |install:
[deprec]  minio                    |active:no   |boot:n  |install:

# discovering ..........
# access .......
[oldb]   config-redis                  |access:yes  |host:ciiconfservicehost  |ip:127.0.0.1  |port:6379
[oldb]   redis-server                  |access:yes  |host:127.0.0.1           |ip:127.0.0.1  |port:6379
[oldb]   pubsub-server                 |access:yes  |host:127.0.0.1           |ip:127.0.0.1  |port:6379
[oldb]   calc-node                     |access:yes  |host:localhost           |ip:127.0.0.1  |port:9117
[oldb]   redis-ext testExtRedisServer  |access:yes  |host:127.0.0.1           |ip:127.0.0.1  |port:6379
[trace]  jaeger                        |access:no   |host:localhost           |ip:127.0.0.1  |port:14269
[telem]  telem                                      |host:localhost           |ip:127.0.0.1  |port:9115
[alarm]  alarm-ias                                  |host:localhost           |ip:127.0.0.1
[alarm]  alarm-mon                     |access:n/a  |host:localhost           |ip:127.0.0.1  |port:5602

# stats ....
config-redis   |total_connections_received:20 |rejected_connections:0
redis-server   |total_connections_received:21 |rejected_connections:0
jaeger
telem

# function ..........2024-06-30T19:27:04.660+0000, TRACE, cii_services/MainThread, Cii Services Tool - log function test - xyz_xgfls

Log      |functional:yes
OLDB DP  |functional:yes
OLDB CE  |functional:yes
IntCfg   |functional:yes

Consult the CII documentation for details of the commands and their output. See: CII Docs.

Consul Agent Startup

Consul is a discovery and name resolution service that has a good integration with Nomad. Consul usage is not fully defined within the ELT project. Thus, RTC Toolkit provides just a preview of how Consul could be used in the examples.

Consul is only needed when the Service Discovery Endpoint uses the “consul” scheme. When needed, it is recommended to start it before Nomad. This avoids a waiting period for Nomad to register in Consul as a service.

The ELT Development environment currently comes with a default Consul agent configuration file (/etc/consul.d/consul.hcl), which is only appropriate if the host is deployed into an existing Consul cluster as a client.

For a minimal single host development environment, the contents of /etc/consul.d/consul.hcl can be replaced with the following configuration (may need to be edited as root):

datacenter = "dc1"
data_dir = "/opt/consul"
server = true
ui = true
retry_join = ["127.0.0.1"]
addresses = {
    "http" = "127.0.0.1"
    "grpc" = "127.0.0.1"
    "https" = "127.0.0.1"
    "dns" = "127.0.0.1"
}
advertise_addr = "127.0.0.1"
bootstrap = true
bootstrap_expect = 1

See ICS_Framework for some pointers on the use of Consul in the ICS.

To check if the Consul agent service is running, use the following command:

$ systemctl status consul

The Consul agent service can be started using the following command (as eltdev user):

$ systemctl start consul

To verify if the agent is running use the following command as any user:

$ consul catalog services

Note

If the agent is not running the following error will be displayed:

Error listing services: Get "http://127.0.0.1:8500/v1/catalog/services":
dial tcp 127.0.0.1:8500: connect: connection refused

The Consul agent can be stopped by executing (as eltdev user):

$ systemctl stop consul

The Consul agent can also be run as a normal process under any user using the provided configuration file (option -c). Before running as a normal user, please make sure that Consul is not already started as a service. To run as a normal process, the following command needs to be executed:

$ consul agent -config-file /etc/consul.d/consul.hcl

Nomad Agent Startup

Nomad is used as the mechanism to start and monitor processes. The usage of Nomad within the ELT is not yet fully defined. Thus, RTC Toolkit provides just a preview of how Nomad could be used in the examples.

Before starting any process using Nomad, a Nomad “agent” must be running. The ELT Development environment comes with a simple single host Nomad agent configuration file (/etc/nomad.d/nomad.hcl), which runs the service under the root account. That means that all jobs executed by Nomad will be executed under root by default. However, the RTC Toolkit is able to specify an alternative user account, under which a component should run.

For a minimal single host development environment, suitable to run the examples from the tutorial sections, the contents of /etc/nomad.d/nomad.hcl can be replaced with the following configuration (may need to be edited as root):

data_dir = "/opt/nomad/data"
bind_addr = "127.0.0.1"
advertise {
    http = "127.0.0.1"
    rpc  = "127.0.0.1"
    serf = "127.0.0.1"
}
server {
    bootstrap_expect = 1
    enabled = true
}
client {
    enabled = true
    servers = ["127.0.0.1:4647"]
    options = {
        "driver.raw_exec.enable" = "1"
    }
    template = {
        "disable_file_sandbox" = true
    }
    meta {
        "node" = "hrtc-gw,srtc1"
    }
}

See ICS Framework for some pointers on the use of Nomad in the ICS.

To check if the Nomad agent service is running, use the following command:

$ systemctl status nomad

The Nomad agent service can be started using the following command (as eltdev user):

$ systemctl start nomad

To verify if the agent is running use the following command as any user:

$ nomad job status

Note

If the agent is not running the following error will be displayed:

Error querying jobs: Get "http://127.0.0.1:4646/v1/jobs": dial tcp 127.0.0.1:4646:
connect: connection refused

The Nomad agent can be stopped by executing (as eltdev user):

$ systemctl stop nomad

The Nomad agent can also be run as a normal process under any user using the provided configuration file (option -c). Before running as a normal user, please make sure that Nomad is not already started as service. To run as a normal process, the following command needs to be executed:

$ nomad agent -config /etc/nomad.d/nomad.hcl

Processes started under Nomad start with the environment of the user running the Nomad Agent, so you may wish to run the Nomad agent as your normal development user during development to ensure any processes started have exactly the same environment as when you execute them from the command line.

Install from Source Code - Optional

The RTC Toolkit can also be installed from a source code tarball. This step is optional, since the software is already available as RPMs, as mentioned in the previous sections. We do not expect this to be the typical procedure for installing the toolkit and should be reserved for cases where one cannot install from RPMs.

Important

In contrast to the RPM installation, The rtctk.lua file is not provided and not needed when installing the software from pure source code. Therefore the file should not be loaded in the private.lua file, i.e. remove or comment out the following line in private.lua:

load("rtctk")

Add the following lines to load the needed dependencies instead:

load("stdif")
load("metadaqif")
load("rad")
load("ddt")
load("roadrunner")
load("cut")
load("mal")
load("ciisrv")

Make sure that the account used to build the software is configured as indicated in the Account Configuration section before continuing, i.e. all previous sections are applicable, except for the RPM installation step.

Note

If the RTC Toolkit RPMs have not been installed at all, then the following command can only be run after completing the installation instructions from source:

$ rtctkConfigTool init-metadata

Installation of Dependencies

To be able to compile and use the RTC Toolkit, the dependencies listed in Release Notes section need to either be installed as RPM packages or be built and installed from source code first.

Make sure the following 3rd party packages are also installed:

boost - Portable C++ utility libraries.
gtest/gmock - Google testing/mocking framework.
Guidelines support library - provides a C++ span implementation.
CLI11 - Command line parser for C++.
Fmt library - String formatting library for C++.
libbacktrace - A library to produce symbolic backtraces.
Google Benchmark - A library to benchmark code snippets, similar to unit tests.
yaml-cpp - A C++ YAML parsing library.
nlohmann/json.hpp - A C++ JSON parsing library.
cfitsio - I/O library for FITS files.
CCfits - A C++ wrapper for cfitsio.
Intel Thread Building Blocks - Used for inter thread communication.
libpcap - Library for handling network packet capture.
ppconsul - API library for Consul.
openblas - BLAS implementation for linear algebra.
pybind11 - Library for producing Python bindings to C++ code.
click - Command line parser for Python.
numpy - Python package for efficient multi-dimensional array computations.
astropy - Python package containing utilities for astronomy and astrophysics.
pylint - Python linting tool.
pytest - A Python test runner.
nose - A Python test runner.
Doxygen - Documentation generator for API documentation.
Sphinx - Documentation generator for user manual.
Plantuml - UML diagram generator.

The complete set of needed software can be installed as RPM packages with the following command:

# dnf -y install \
    waf \
    boost-devel \
    python3-devel \
    pylint \
    elt-wtools \
    gtest \
    gmock \
    python3-pytest \
    python3-pytest-custom_exit_code \
    python3-pytest-qt \
    elt-stdif-devel \
    elt-metadaqif-devel \
    elt-taiclock-devel \
    elt-llnetio-devel \
    elt-mal-devel \
    elt-rad-devel \
    elt-ddt-devel \
    elt-cut \
    elt-cut-widgets \
    elt-roadrunner-common \
    elt-roadrunner-numapp-devel \
    elt-roadrunner-ipcq-devel \
    elt-roadrunner-perfc-devel \
    elt-ciisrv-client-api-devel \
    elt-ciisrv-oldb-client-devel \
    elt-ciisrv-config-ng-devel \
    elt-ciisrv-robot-library \
    elt-ciisrv-tooling \
    elt-ciisrv-qt-widgets \
    elt-msgsend \
    libatomic \
    guidelines-support-library-devel \
    fast-dds-devel \
    cli11-devel \
    fmt-devel \
    libbacktrace-devel \
    google-benchmark-devel \
    yaml-cpp-devel \
    json-devel \
    CCfits-devel \
    cfitsio-devel \
    libpcap-devel \
    libppconsul-devel \
    tbb-devel \
    pybind11-devel \
    python3-pyside6 \
    python3-click \
    python3-numpy \
    python3-astropy \
    python3-taurus \
    python3-wheel \
    python3-setuptools \
    python3-pip \
    python3-ipython \
    openblas \
    doxygen \
    plantuml \
    python3-pillow \
    python3-sphinx \
    python3-sphinx-eso-theme \
    python3-sphinxcontrib-plantuml \
    elt-trs-common-devel \
    elt-ciisrv-cii-common-types-devel \
    log4cplus-devel \
    elt-etr

After these packages have successfully installed, log out and back into the host for the environment variables to be updated.

Installation of the RTC Toolkit

Download the tarball for RTC Toolkit version 5.1.0 from ESO Gitlab. Unpack it and then execute the steps below to build and install the software.

$ cd rtctk-*/
$ waf configure build install

Note

There are some optional dependencies in the toolkit. These may be indicated as “not found” during the configure step. For example, cuBLAS is optional and only needed if building on a machine that has GPUs. If there are any mandatory dependencies that cannot be found then the configure step will fail as expected.

Documentation (manuals and API reference) can be generated by invoking:

$ waf build --with-docs

The result can be viewed by opening the respective index.html files under build/doc/.

Verify Correct Installation

To verify the build of RTC Toolkit, it is possible to run the unit tests by invoking:

$ cd rtctk-*/
$ waf test --alltests

For a more comprehensive verification that the RTC Toolkit was installed correctly, run the end-to-end integration test as follows:

$ cd test/_examples/exampleEndToEnd
$ etr -v

The integration test is expected to terminate successfully after running for about one minute.