RTC Supervisor

Overview

The RTC Supervisor’s main role is to provide an entry point to the RTC for basic state guiding, monitoring, error recovery and population of the Runtime Configuration Repository. For this reason, each RTC system shall have exactly one RTC Supervisor instance.

Note

The RTC Supervisor is NOT a deployment tool; for this purpose the Deployment Daemon shall be used.

In the current version, the RTC Supervisor performs the following:

Loading the contents of the Runtime Configuration Repository from the Persistent Configuration Repository using the active Deployment Set.
Estimating the overall RTC system status by monitoring the state of all SRTC components.
Monitoring the liveliness of all supervised SRTC components and detecting if any have crashed.
Guiding the state of all supervised SRTC components by forwarding state change requests.
Updating dynamic configuration parameters in all SRTC components.
Setting Operational Modes and applying them to all supervised SRTC components.
Writing back configuration data items from the Runtime Configuration Repository to the Persistent Configuration Repository.

Note

With the introduction of Nomad and Consul it is possible that some functions listed above may be removed from future versions of the RTC Supervisor, in particular those related to monitoring.

Launching

To start the process, the name of the RTC Supervisor instance must be provided, as is shown in the following example command:

$ rtctkRtcSupervisor <supervisor>

Note

Note that in the above and in the following examples <supervisor> is only a placeholder for the actual instance name of the RTC Supervisor component.

For more information about optional command line arguments refer to section RTC Component.

Commands

The RTC Supervisor component can be steered by sending commands to its provided MAL interface. The set of commands that is currently available is indicated in the following table, together with the expected behaviour:

Command	Behaviour
Init	Initialises the RTC system: Validates Persistent Configuration Repository if `disable_validate` configuration is not set to `true`. Populates the Runtime Repository with active deployment set if `disable_populate` configuration is not set to `true`. Retrieves configuration from the Runtime Configuration Repository. Starts component monitoring and status estimation activities. Sends synchronous Init to all RTC Components.
Exit	Terminates the process.
Stop	Stops an ongoing initialisation procedure started by the Init command.
Reset	Resets the state of the RTC System to the state before Init was received.
Enable	Enables the RTC system: Sends synchronous Enable to all RTC Components.
Disable	Disables the RTC system: Sends synchronous Disable to all RTC Components.
Update	Updates the RTC system configuration: Reloads dynamic component configuration. Sends synchronous Update to all RTC Components.
WriteBack	Writes content of the Runtime Repository back to the Persistent Repository.
SetModes	Applies requested operational modes to the RTC system: Validates Persistent Configuration Repository if `disable_validate` configuration is not set to `true`. Populates the Runtime Configuration Repository with selected modes if `disable_populate` configuration is not set to `true`. Sends synchronous Update to all RTC Components.
SetDeferredModes	Specifies which modes should be applied already during Initialisation.
GetDeferredModes	Getter to query deferred modes.
GetAvailableModes	Getter to query available modes and mode options.
GetActiveModes	Getter to query currently active modes and mode options.

Customisation

The RTC Supervisor implementation is divided into a library and a server application.

The library implements the functionality required for communicating with a set of supervised rtcObjects and sending commands to them using an rtcObjectController.

The server application contains the main BusinessLogic and makes use of the library part. It also constitutes the default implementation of the usable component.

Instrument RTC developers may customise the RTC Supervisor by providing e.g. custom status estimation or error recovery functionality. It is foreseen that for this, the server application needs to be modified or extended while the client library can be reused as-is.

Once all extension points are defined and implemented, component customisation will be described in detail in this section.

Configuration

The RTC Supervisor has the following static configuration parameters:

Listing 7 rtc_sup.yaml

static:
    init_alone: !cfg.type:boolean False
    enable_alone: !cfg.type:boolean False
    disable_alone: !cfg.type:boolean False
    update_alone: !cfg.type:boolean False
    disable_validate: !cfg.type:boolean False
    disable_populate: !cfg.type:boolean False

The first four parameters define whether the associated commands (Init, Enable, Disable, Update) are sent to the supervised components in parallel or series. The last parameter allows to disable automatic population of the Runtime Configuration Repository for development or engineering purposes.

Additionally, the RtcSupervisor needs to get a list of components that it should supervise. This list is read from the /deployment/component_list datapoint in the Runtime Configuration Repository. During the population of the Runtime Configuration Repository, /deployment/component_list is copied from the Persistent Configuration Repository to the Runtime Configuration Repository to be subsequently used by RtcSupervisor. Therefore, this is the same datapoint that is used by the Deployment Daemon.

The following is an example of a YAML file containing the component_list attribute, which is used by the Deployment Daemon to deploy components.

Listing 8 deployment.yaml

component_list: !cfg.type:vector_string
    - <supervisor>
    - object1
    - object2

# hereafter is configuration for the deployment used by deployment daemon only
<supervisor>:
    node: !cfg.type:string host1
    executable: !cfg.type:string rtctkRtcSupervisor
object1:
    node: !cfg.type:string host1
    executable: !cfg.type:string rtctkObject1Component
object2:
    node: !cfg.type:string host2
    executable: !cfg.type:string rtctkObject2Component

Monitoring

For the RTC Supervisor the following monitoring datapoints are published to OLDB.

Parameter	Type	Description
state	string	state of RTC supervisor component itself.
status	string	overall estimated system status, see Status Estimation.
global_error	boolean	flag that indicates a global error, e.g one of component is in error state.
global_error_who	string	who (which component) caused the error, applicable if `global_error` is `true`.

Behaviour

This section provides a more detailed description of the RTC Supervisor behaviour.

Populating the Runtime Configuration Repository

When receiving the command Init, the content of the Runtime Configuration Repository is automatically populated by RtcSupervisor from the Persistent Configuration Repository. This population is performed before any state guiding happens and before any component accesses the Runtime Configuration Repository. This is important to have the configuration available before any component tries to access it.

Note

A long-running initialisation operation can be aborted with the Stop command. In this case the RTC Supervisor will transition back to the state On::NotOperational::NotReady but the overall system could remain in a partially initialized and therefore inconsistent state. This can be resolved by issuing the Init command again and letting it run to completion.

The population of the Runtime Configuration Repository is controlled by the following configuration datapoints:

Parameter	Type	Description
/deployment/active_deployment	RtcString	Name of the Deployment Set to be used to populate the Runtime Configuration Repository.
/<supervisor>/static/disable_validate	RtcBool	(Optional) flag that indicates if the Persistent Configuration Repository should perform more systematic checks to see that the configuration is valid, i.e. the structure and contents is correct. Default: false.
/<supervisor>/static/disable_populate	RtcBool	(Optional) flag that indicates if the Runtime Configuration Repository population should be done, where `<supervisor>` is the instance name of the RTC Supervisor. Default: false.

By default the Persistent Configuration Repository is validated and then the Runtime Configuration Repository is populated by loading the active Deployment Set indicated by the /deployment/active_deployment datapoint in the Runtime Configuration Repository. The value is used to retrieve the proper Deployment Set from the Persistent Configuration Repository. The details of the configuration layout for Deployment Sets is described in section Configuration Layout.

The /deployment/active_deployment datapoint is created and set by the Deployment Daemon during the deployment process. However, if /deployment/active_deployment is not provided or can not be read from the Runtime Configuration Repository, the RtcSupervisor tries to use the value of /default_deployment from the Persistent Configuration Repository as a fallback solution, which can also be used for simple testing or similar.

It is also possible to disable the automatic (re)loading of the datapoints into the Runtime Configuration Repository by setting the /<supervisor>/static/disable_populate boolean datapoint to true in the Runtime Configuration Repository. Where <supervisor> should be the instance name of the RTC Supervisor. This can, for example, be used if we want to retain the changes done in the Runtime Configuration during RtcSupervisor reinitialization. The flag can be set by running the following example command:

$ rtctkConfigTool set runtime /<supervisor>/static/disable_populate True

The validation performed on the Persistent Configuration Repository may take a substantial amount of time if there are a large number of complex Deployment Sets configured. For this reason the optional flag /<supervisor>/static/disable_validate can be provided to the RTC Supervisor to disable the validation step during initialising or when executing the SetModes command. This flag behaves in a similar manner to the disable_populate flag. The flag can be set by running the following example command:

$ rtctkConfigTool set runtime /<supervisor>/static/disable_validate True

Note

The disable_populate and disable_validate flags can be set in the RTC Supervisor’s configuration in the Persistent Configuration Repository, just like any other static configuration datapoint. However, this means these datapoints are copied to the Runtime Configuration Repository whenever it is being populated and will override any values that were already set manually in the Runtime Configuration Repository.

Handling Operational Modes

This section describes how to set and inspect Operational Modes using the RTC Supervisor.

For an introduction about the philosophy and purpose of Operational Modes please refer to the RTC Toolkit Design Document. For an introduction on how to define Operational Modes in the Persistent Configuration Repository please refer to section Configuration Layout.

SetModes

Operational Modes can be applied using the SetModes command once the system is initialised and in a stable state. The minimum required state for setting modes is On::NotOperational::Ready, but also higher states in On::Operational are accepted if they are stable.

$ rtctkClient <supervisor> SetModes '{"mode_a" : "option_1", "mode_b" : "option_x"}'

The example above shows how to send the command using rtctkClient with a JSON payload containing a dictionary of two operational modes with associated options.

Note

Since the JSON payload contains quotes, the whole payload needs to be enclosed in single quotes to make the Linux command line tool happy.

After the datapoints have been copied over to the Runtime Configuration Repository, a synchronous Update broadcast is sent to all supervised components to command them to reload their dynamic configuration, so that the selected modes really get applied.

SetDeferredModes

If a specific set of modes should already be applied during initialisation the command SetDeferredModes can be issued in the state On::NotOperational::NotReady to specify deferred modes. This is useful for engineering purpose.

$ rtctkClient <supervisor> SetDeferredModes '{"mode_a" : "option_1"}'

The example above shows how to send the command with a JSON payload containing a dictionary with a single operational mode and its associated option.

GetDeferredModes

This command can be used to query which deferred modes have already been set, to be applied during the next initialisation.

$ rtctkClient <supervisor> GetDeferredModes
{"mode_a" : "option_1"}

The command returns a JSON payload containing a dictionary of deferred modes with associated options.

GetActiveModes

Currently active Operational Modes can be queried using the GetActiveModes command:

$ rtctkClient <supervisor> GetActiveModes
{"mode_a" : "option_1", "mode_b" : "option_x"}

The result is a JSON payload containing a map of currently active modes and associated options.

After the component initialisation, this command should report the default modes and options plus any modes that have been specified or overwritten using the SetDeferredModes command.

Note

Note that the same information is also available in the Runtime Configuration Repository datapoint /deployment/active_modes, which acts as the single source of truth regarding active modes.

GetAvailableModes

This command can be used to query which modes and mode options are available in the current deployment set:

$ rtctkClient <supervisor> GetAvailableModes
{"mode_a" : ["option_1", "option_2"], "mode_b" : ["option_x", "option_y", "option_z"]}

The result is a JSON payload containing a map of available modes and all possible mode options that can be passed to the SetModes or SetDeferredModes commands.

Writeback to Persistent Configuration Repository

This section describes how the WriteBack command can be used to store Runtime Configuration datapoints back into the Persistent Configuration Repository.

The WriteBack command triggers a configuration parameter copy from the Runtime Configuration Repository back to the Persistent Configuration Repository. For resolving the correct destination, the algorithm takes into account the active deployment set as well as the active modes.

In general, WriteBack should be issued once the system is initialised and in a stable state. This means, the parameters that are written back must not change while the write back is being performed. States where this could be done safely are e.g. On::NotOperational::Ready or On::Operational::Idle. Of course, if operators know for sure that affected parameters will not change they may also issue the command in higher states.

The WriteBack command takes a JSON argument with a list of datapoints to write back and it can be issued in two distinct ways:

Full WriteBack:

$ rtctkClient <supervisor> WriteBack '[]'

In this case the JSON array is empty and therefore all datapoints from the Runtime Configuration Repository will be written back to the Persistent Configuration Repository. This also means, the operation is heavy and very likely long-lasting.

Selective WriteBack:

$ rtctkClient <supervisor> WriteBack '["/object1/dynamic/foo", "/object2/dynamic/bar"]'

In this case the JSON array contains a list of parameters, so only the specified datapoints will be written back to the Persistent Configuration Repository. This means, the operation is lighter and will finish sooner since there is less data to copy. The datapoint paths used in the JSON list are for the datapoints in the Runtime Configuration Repository.

Note

A long-running write back operation can be aborted with the Stop command. However, this will leave the Persistent Configuration Repository in an inconsistent state where only a subset of requested datapoints have been written back. Therefore RTC operators should be very cautious when aborting a write back operation.

State Guiding

For basic state guiding, the RTC Supervisor understands the following commands

Init
Reset
Enable
Disable

On command reception the same commands are forwarded to the list of supervised components.

Note

Run and Idle are not in the above list. The baseline currently is that the sequencer code will be responsible for sending the Run commands to the RTC components in the correct order. This may be re-evaluated.

There are configuration flags available indicating if these commands should be forwarded to the list of supervised components in parallel or in series , see section Configuration.

Status Estimation

During initialisation the RTC Supervisor subscribes to state-changed notifications of all supervised components. Whenever a component changes its state, a customisable StatusEstimator object is notified via a callback and the overall believed system status is recomputed from individual component states using the customisable method ApplyEstimationHeuristic. Once calculated, the overall status is published to the Online Database (OLDB) attribute <supervisor>/status and also via the CII MAL Pub/Sub topic std/status.

In the current implementation, the default heuristic of the StatusEstimator produces the following set of status strings:

Status	Description
`Off;Off`	The RTC is off, some or all applications are not running
`NotOperational;NotReady`	The RTC is on but some applications are not yet initialised
`NotOperational;Initialising`	The RTC is on and currently initialising
`NotOperational;Ready`	The RTC is on and ready (initialised and configured)
`Operational;Idle`	The RTC is on and operational (enabled)
`Operational;LoopOpened`	The RTC is in open loop mode (not controlling the mirrors)
`Operational;LoopClosed`	The RTC is in closed loop mode (controlling the mirrors)
`Operational;Error`	The RTC is in error, some application encountered an async. error

Note

The RTC Supervisor can only estimate the system status once it is initialised. This means the component will publish hardcoded status codes at application startup, reset, exit and also at the beginning of an initialisation activity.

Component Monitoring

Once initialised, the RtcSupervisor performs continuous component monitoring using a monitoring thread provided by the RtcMonitor class.

The monitoring threads continuously issues a GetVersion command with a short timeout to each component. If the command fails, the component will be marked as erroneous and commands will not be sent to it subsequently.

In addition, when the RTC Supervisor notices that something has gone wrong, it calls the SetError method which updates the OLDB attributes global_error and global_error_who indicating the error and its cause.

Error Recovery

Once implemented, this section will provide a comprehensive description of the error recovery functionality.

Limitations and Known Issues

User extension points are not implemented yet, so for now it is not yet possible to use the RTC Supervisor as a base class to which instrument specific functionality (e.g. custom status estimation algorithm or custom interfaces) can be added.
Functionality for Error Recovery has not been implemented yet, so there is no simple interface for error recovery available if one or more components are in error.
An incorrect global status is reported as Idle if only a subset of components is in the Running state (RTCTK-1267).