Deployment Daemon

Overview

The Deployment Daemon is a special kind of component. It differs from a regular RTC Component in the commands that it provides, as well as in the Life-Cycle (State Machine). Its main function is to take care of deployment and undeployment of the RTC Toolkit-+based software on a SRTC system. To be more precise, to start and stop a set of components’ processes. There shall be just one instance of the Deployment Daemon per system, it shall be run before any other component, and it shall be running for the whole time.

Deployment of a component means that a component is “just” started, and not necessarily configured from Run-Time repository or similar. If the component uses a canonical state machine, which is typically the case for the RTC Components, it is found in state On::NotOperational::NotReady once it has been deployed. Another mechanism is needed to go through the Life-Cycle, typically by using the RTC Supervisor to steer the states of the components.

The Deployment Daemon internally uses Nomad to deploy/undeploy (start/stop) a component. For each component a Nomad job file is generated using the rtctkDeploymentGen command and information from the Configuration. This is then sent to the Nomad agent service to start the component. The Daemon undeploys (stops) the component by requesting the Nomad agent service to stop a job for the particular component. Nomad itself takes care of registering/unregistering the components in the Consul service, which is later used by the Service Discovery in the RTC Toolkit to query for registered components. Since Nomad and Consul are used as the deployment mechanism, it is possible to use a web browser (e.g. to view http://<nomad_host_address>:4646) and other tools to get more information and details about the deployment, status of jobs, etc.

Note

Be aware that Nomad deploys components using the same user account that has been used to start the Nomad agent service itself. This means the component inherits the same account configuration, e.g. environment variables. Usually the agent is started as a system service, which means using the root user. As we do not want to run the components as root, we need to explicitly specify the user to be used.

Note

Please refer to Consul and Nomad usage, and the general Nomad and Consul documentation for more details.

Prerequisites

The SRTC software: components based on RTC Toolkit need to be installed on the hosts where the deployment for each component should occur.

Before starting the Deployment Daemon, Nomad and Consul should be properly configured and running, as well as the CII services (specifically OLDB, Config Service, Redis and Logging Service) and the following RTC Toolkit services:

Persistent Configuration Repository with the configuration for the deployment.
Runtime Configuration Repository.

Running

The Deployment Daemon can be started using Nomad. In this way it gets registered in Consul similar to other components. As a first step we need to generate a Nomad job file for the deployment daemon using the rtctkDeploymentGen tool with the --deployment-daemon flag to indicate to create a job file specific for a deployment daemon. For example, to create the deployment_daemon.nomad job file that starts a deployment daemon (rtctkDeploymentDaemon) with the name deployment_daemon on the deployment_daemon_host node (a key-value in the meta stanza in Nomad job specs) as user (user) eltdev, use the following command:

rtctkDeploymentGen --node deployment_daemon_host --user eltdev job --deployment-daemon deployment_daemon

Warning

Be aware that the job name and the component instance name have to be the same (e.g.: deployment_daemon in the above example).

Do not forget to provide the --deployment-daemon flag to rtctkDeploymentGen command.

As the next step, we need to send the generated job to Nomad with:

nomad job run deployment_daemon.nomad

The daemon can be stopped by sending the following command to Nomad:

nomad job stop --purge deployment_daemon

As an alternative, the daemon can be started manually by executing the following command (only recommended for testing):

rtctkDeploymentDaemon deployment_daemon

As mentioned in Prerequisites, the needed RTC Toolkit services must already be running. These services must also be registered in Consul to be accessed by the Service Discovery. This can also be done using Nomad. An example would be to use the rtc_discovery_service.nomad Nomad job file to register the following services:

Persistent Repository service endpoint (URI): cii.config://local/persistent
OLDB endpoint (URI): cii.oldb:/my_instance/oldb

The job file can be generated using the command:

rtctkDeploymentGen services cii.config://local/persistent \
                            cii.oldb:/my_instance/oldb

and sent to Nomad as described above.

Note

The Run-Time Repository is started by the Deployment Daemon, so the deployment configuration needs to be specified there.

Configuration

The deployment daemon is configured from the Persistent Configuration Repository (PSR) (and not from the Runtime Configuration Repository (RTR), as is usually the case with other components). For each deployment set <deployment_name>, the configuration datapoints can be found under the <deployment_name>/defaults/deployment/ path hierarchy, which corresponds to the YAML file <deployment_name>/defaults/deployment.yaml. At least, a list of components to be deployed must be provided (component_list), and for each component name in this list, a YAML mapping node with the same name needs to be specified. Each YAML mapping node should contain the following fields:

Configuration Datapoint	Type (CII Configuration)	Description
`node`	`!cfg.type:string`	Node where the component should be deployed, i.e where the component executable is started. Note: this is not a host name, but the key-value pair in the Nomad job file specification meta block.
`executable`	`!cfg.type:string`	Component executable to be started.
`user`	`!cfg.type:string`	(Optional) User to be used by Nomad agent to execute the component executable. Note: If specified, the Nomad agent(s) need(s) to be run as root. If not specified the user that starts the Nomad agent is used.
`numactl_opts`	`!cfg.type:string`	(Optional) contains options for `numactl` that are used to execute the component. E.g `--all -C 0` to bind a component’s process to CPU #0. See the numactl(8) manpage for details.
`type`	`!cfg.type:string`	(Optional) Type of service to be executed. For each service there needs to be a corresponding jinja2 file for each Nomad job file. For details see: Extendability. The types supported by RTC toolkit are: rtctk_component (default type), ddt_broker, runtime_repository.

Note

Each deployment set shall contain a deployment configuration for the run-time repository with the name rtr. E.g.

rtr:
  type: !cfg.type:string runtime_repository
  node: !cfg.type:string srtc_node_01

The general and Nomad-related (global) configuration is specified in the following datapoints:

Configuration Datapoint	Type (CII Configuration)	Description
`description`	`!cfg.type:string`	Deployment set description.
`component_list`	`!cfg.type:vector_string`	List of components (no type or rtctk_component) that must be deployed for the deployment set.
`service_list`	`!cfg.type:vector_string`	List of services (type different than rtctk_component) that must be deployed for the deployment set.
`nomad/datacenter`	`!cfg.type:string`	(Optional) name of Nomad data center to use.
`nomad/app_kill_timeout`	`!cfg.type:string`	(Optional) Nomad job/application kill timeout. E.g. 20s.

An example of a deployment set configuration for the Run-time repository (rtr) service and 5 components:

service_list: !cfg.type:vector_string [rtr]
component_list: !cfg.type:vector_string [comp_1, comp_2, comp_3, comp_4, comp_5]
description: !cfg.type:string Example Deployment Set

nomad:
  datacenter: !cfg.type:string example_DC

rtr:
  node: !cfg.type:string srtc_node_01
  type: !cfg.type:string runtime_repository


comp_1:
  node: !cfg.type:string hrtc_gateway
  executable: !cfg.type:string rtctkTelRepub
comp_2:
  node: !cfg.type:string srtc_node_01
  executable: !cfg.type:string rtctkExampleComponent
comp_3:
  node: !cfg.type:string srtc_node_01
  executable: !cfg.type:string rtctkExampleComponent
  numactl_opts: !cfg.type:string -C 0
comp_4:
  node: !cfg.type:string srtc_node_01
  executable: !cfg.type:string rtctkExampleComponent
comp_5:
  node: !cfg.type:string srtc_node_02
  executable: !cfg.type:string rtctkExampleComponent2

Commands

The Deployment Daemon provides a set of MAL interface commands indicated in the following table, together with the expected behaviour:

Command	Behaviour
Deploy	Deploys components in parallel for the specified deployment set. As a first step it cleans the contents of the Runtime Configuration Repository (RTR). At the end it writes the name of the active deployment set in the `/deployment/active_deployment` datapoint to the RTR. The command returns when all the components have been started, which can take some time, or when an error occurs.
Undeploy	Undeploys components in parallel for the active deployment set. If the undeployment of one component fails, it proceeds with the remaining components.
GetDeploymentSets	Returns a list of all deployment sets found in the configuration.
GetActiveDeployment	Returns the name of the currently active deployment.
GetDescription	Returns the description of the active deployment set.
GetComponents	Returns a list of all available components in the current active deployment. Not all of these components are necessarily deployed (active).
GetActiveComponents	Returns a list of running (active) components of the current active deployment. This list is retrieved from the Service Discovery.
StartComponent	Starts a component with the given name that is part of the active deployment set and not already deployed (started).
StopComponent	Stops a component with the given name that has already been deployed (started) as part of the active deployment set.
GetServices	Returns a list of all available services in the current active deployment. Not all of these services are necessarily deployed (active).
GetActiveServices	Returns a list of running (active) services of the current active deployment. This list is retrieved from the Service Discovery.
SetLogLevel	Sets the log level for logging messages.
GetState	Returns the state of the deployment daemon component.
Reset	Resets the state of the deployment daemon component to the initial state.
Exit	Stops the daemon.

The Deployment Daemon can be steered via the Control and Monitoring Tool GUI or using the command line client provided for this purpose: rtctkDeploymentDaemonClient. The detailed list of commands and options can be found by running with the --help option:

$  rtctkDeploymentDaemonClient --help
    Usage: rtctkDeploymentDaemonClient [OPTIONS] COMMAND [ARGS]...

      Command line client for interacting with the Deployment Daemon.

    Options:
      --version                       Show the version and exit.
      --service-discovery-endpoint, --sde URI
                                      URI endpoint to the service discovery.
                                      Default: consul://localhost:8500
      --deployment-daemon-name, --ddn TEXT
                                      Name of deployment daemon instance. Default:
                                      deployment_daemon.
      --log-level [FATAL|ERROR|WARNING|INFO|DEBUG|TRACE|NOTSET]
                                      Sets the global logging level for the tool.
                                      This overrides any /log_level setting found
                                      in the configuration. Default: FATAL
      --log-to-file                   Makes the tool log to file. This overrides
                                      any /log_to_file setting found in the
                                      configuration. Default: False
      -t, --timeout INTEGER           timeout [sec] Default: 0:00:20
      --help                          Show this message and exit.

    Commands:
      Deploy               Invokes Deploy on the Deployment Daemon to deploy...
      Exit                 Invokes Exit on the Deployment Daemon
      GetActiveComponents  Invokes GetActiveComponents on the Deployment...
      GetActiveDeployment  Invokes GetActiveDeployment on the Deployment...
      GetActiveServices    Invokes GetActiveServices on the Deployment Daemon...
      GetComponents        Invokes GetComponents on the Deployment Daemon...
      GetDeploymentSets    Invokes GetDeploymentSets on the Deployment Daemon...
      GetDescription       Invokes GetDescription on the Deployment Daemon...
      GetServices          Invokes GetServices on the Deployment Daemon that...
      GetState             Invokes GetState on the Deployment Daemon
      Reset                Invokes Reset on the Deployment Daemon
      SetLogLevel          Sets Log Level of the Deployment Daemon to LOG_INFO
      StartComponent       Invokes StartComponent on the Deployment Daemon to...
      StopComponent        Invokes StopComponent on the Deployment Daemon to...
      Undeploy             Invokes Undeploy on the Deployment Daemon to...

Errors

The Deploy command tries to deploy all the components for a given deployment set, and it fails if there is a problem to deploy any of the components. If it fails, the error Problem to deploy: XYZ is returned with details in the back trace, e.g. Problem generating/sending Nomad job file.

For example, the following error message indicates that the deployment daemon had a problem to retrieve deployment information (list of deployment sets) from the Persistent Configuration Repository:

2023-07-06T15:05:55.405+0000, ERROR, executor/MainThread, Exception "::rtctkif::DeploymentException" caught.
EXCEPTION: {"desc": "Nested exceptions:\n1. ActivityDeploying: failed\n2. Problem to deploy: depl1\n3. Deployment set name: depl1 not found in the list of deployment sets: [ ]", "code": 501}

One cause could be that the CFGPATH environment variable was incorrectly set, or maybe not picked up by the Nomad agent.

The Undeploy command tries to undeploy all the components for the active deployment set. If there is no active deployment set then the error No active deployment is returned. The following error is produced when a problem occurs with Nomad: Problem stopping Nomad job using command: ...

Details for the possible errors can be found in the log messages.

Extendability

The deployment damon is possible to extend to start any (type of) executable (=services). This can be done by specifying a type configuration datapoint for a given deployment configuration node for that executable. In addition, it is necessary to provide a corresponding jinja template for generating a Nomad job file. All other configurations that are specified under the particular deployment configuration node, are passed to the jinja template as key-value pairs. Global Nomad configuration is passed in a similar way. The template needs to be installed in CFGPATH under resource/rtctk, and the template file should follow the convention: <type>.nomad.jinja2. The service (executable) needs to be listed in service_list to be started by the Deployment Daemon.

E.g. for DDT broker type

...
service_list: !cfg.type:vector_string [ddt_broker_test_1, ...]
...
ddt_broker_test_1:
    node: !cfg.type:string srtc1
    type: !cfg.type:string ddt_broker
...

The corresponding jinja template is: rtctk/ddt_broker.nomad.jinja2. In this case, node with value srtc1, type with value ddt_broker and name with value ddt_broker_test_1 are passed to the rtctk/ddt_broker.nomad.jinja2.

As the new type needs to support registration and discovery by Consul through the Nomad job file, the jinja template file needs to contain the following meta information in service stanza: name that should contain the executable instance name, type that contains type (in our case ddt_broker), and endpoint_type and endpoint_uri. E.g.:

...
service {
    ...
    meta {
        name = "{{ depl.name }}"
        type = "{{ depl.type }}" # = ddt_broker
        endpoint_type = "req_rep_endpoint"
        endpoint_uri = "zpb.rr://${NOMAD_IP_req_rep}:${NOMAD_PORT_req_rep}/"
    }
...

Troubleshooting

When the deployment daemon runs a Nomad job, the console output and other information can be found using a web browser and going to the relevant Nomad page, e.g. http://<nomad_host_address>:4646.

The generated Nomad job files used to deploy components can be found in the folder used by the Nomad Agent (Nomad file system) to run the deployment daemon. E.g. for deployment_daemon_e2e (if data_dir is not set to another location):

/tmp/NomadClient1086104422/91ddf3b5-a72b-651d-a73e-775ab51f3152/deployment_daemon_e2e_task/

Limitations and Known Issues

A failed deployment can leave the system in an intermediate state and requires manual cleanup (RTCTK-1311).
rtctkDeploymentDaemonClient does not respect the --timeout | -t setting in all cases and may timeout already after about 30 seconds, even if the timeout setting was set to a longer period (RTCTK-1316).