Control and Monitoring Tool

The rtctkCtrlMonTool is a command line and a graphical user interface (GUI) tool that presents the current RTC deployment and allows basic inspection and interaction with individual RTC Components.

Fig. 13 Control and Monitoring Tool GUI

Starting the Application

In order to be able to start the rtctkCtrlMonTool, the user needs first to make sure:

1. There is an RTC currently deployed.

2. The Service Discovery is present as Consul or as a file (just for testing purposes).

3. (optional) The Deployment Daemon is present.

4. An X server is available. If the user logs in using ssh, the option -X needs to be used. If a graphical session is used, there is no further configuration needed.

General information on how to use the tool can be retrieved by invoking:

$ rtctkCtrlMonTool --help

The tool can be started in GUI mode using:

$ rtctkCtrlMonTool gui

General information on how to use the GUI can be retrieved by invoking:

$ rtctkCtrlMonTool gui --help

A normal invocation of the GUI would look like (such as for the ExampleEndToEnd):

$ rtctkCtrlMonTool --sde consul://localhost:8500 --ddn deployment_daemon_e2e gui

Summary of the Control and Monitoring Tool

Fig. 14 Sections of the application

1. Menu: Contains entries to access functionality of the application that is not common to use.

2. Deployment Overview - Deployment Daemon: Presents information about the deployment sets as retrieved from the Deployment Daemon.

3. RTC Global Status - Overall RTC system status estimated by RTC Supervisor by monitoring the state of all SRTC components.

4. Reduced Alert Status - Represents reduced or combined status of one or more Alert Statuses. If any Alert is active in a set of alerts, then the Reduced Alert Status is also active. Clicking on the Reduced Alert Status LED shows indvidual component alerts based on which the Reduced Alert is determined.

5. Deployment Overview - RTC Components: Presents a list of all RTC Components declared in the Service Discovery.

6. Standard Interface and Update Commands: Allows interaction with RTC Components through their Standard Interface and Update Interface commands.

7. Logs: Shows the logs of a specific component, or the logs from the Control and Monitoring Tool.

8. Custom Commands: Depending of the interfaces an RTC Component implements, a specific widget may be shown in this dock widget.

9. Status: system CPU usage, heartbeat, OLDB and Deployment Daemon connection status.

Menu

The Menu contains the following top level entries:

• File

  • Exit: Terminates the application

• Edit

• View

  • List all RTC Logs: removes filters from the Logs table.

  • Logs Panel: A toggle option that hides or shows the Logs docking window.

  • Custom Commands Panel: A toggle option that hides or shows the Custom Commands docking window.

  • Toggle Error Dialogs: Enables or disables the dialogs that pops up when an error or exception is detected during the execution of remote commands.

• Tools

• Help

Tip

The user may also access the Menu Bar using shortcuts. The File Menu Entry can be accessed typing Alt+f. The Exit entry can then be quickly activated using x. The proper letter to use in each case is underlined in the Menu Bar and the Menu Entries.

Note

The menu is sparse at the moment, but will be populated with more options in the following versions.

Deployment Overview

Fig. 15 Deployment Overview

Fig. 16 View of component alerts

The Deployment Overview contains:

• Deployment set information and control at the top, composed of the following:

  • A drop down menu with a list of possible deployment sets retrieved from the Deployment Daemon. If one is already active it is indicated with a check mark.

  • Two buttons to control the Deployment Daemon: one to Deploy in the drop down menu selected deployment set, and one to Undeploy the currently active deployment set.

• RTC Global Status and Alert Status that, when clicked, expands to a view of the alert sources, as shown in Fig. 16

• A list of RTC Components present in the Service Discovery. For each component a row is added. The rows shown in Fig. 15 are composed of:

  Component Name

  State

  Alert Status LED

  Start Process Button

  Stop Process Button

  Show Logs Button

The State is automatically updated via OLDB subscription, or via Request-Reply polling of the getState() method in the Standard Interface. The method selection is done using the URI of the OLDB endpoint. If the URI schema is “cii.oldb”, the state is updated using subscription.

If the scheme is “file”, then it defaults to polling using the MAL Request Reply Standard Interface.

The background color (normally green) indicates that the quality of the datapoints is good.

The Alert Status LED is glowing red when an error occurs. Normally, when no error is detected, it has a dimmed green color.

The Play (Start Process) Button will request the Deployment Daemon to start the component.

The Stop (Process) Button will request the Deployment Daemon to stop the component.

Start and Stopping components only work when the Deployment Daemon is present.

The last button on the Component is the Magnifying Lens (Show Logs) Button. This will present the logs for the component in the Logs Dock Widget present below.

When a row is selected, it will start automatically a connection to their Request-Reply interfaces. When connections are successful, the Standard Interface and Update Commands Group Box will be enabled for user interaction.

Standard Interface and Update Commands

Fig. 17 Standard Interface and Update Commands (Connected)

Fig. 17 is a depiction of the section of the UI where the user may interact with the selected RTC Component’s Standard and Update Interfaces.

When the Control and Monitoring Tool first starts, this section of the application will not be active. The user needs first to select a row in the Deployment Overview. This will automatically connect to the RTC Component, and activate this section of the Control and Monitoring Tool.

Upon a successful connection, this section of the UI will be enabled for user interaction.

Clicking on a button will request the command with the same name. Since Requests to the Standard interface do not return any useful feedback information, the user needs to be aware of the expected changes. i.e.: Pressing the Init Button on a Supervisor component will start the initialization sequences of all components in the RTC. State changes can be monitored in the Deployment Overview.

Custom Commands

This panels is dynamic and is coordinated with the currently selected RTC Component in the Deployment Overview. Selecting a different RTC Component in the Deployment Overview results in a different widget being shown in this panel.

Fig. 18 Custom Commands: This example shows the widget for a generic RTC Component.

Internally it uses a plugin and factory mechanism to show the most apt widget based on the MAL Interfaces an RTC Component implements. The RTCTK and Control and Monitoring Tools comes with a basic set of widgets for the common RTC Components.

A Developer may follow the tutorial Creating a Custom RTC Widget to add widgets that interact with RTC Components of their own.

Logs

The Logs Table automatically monitors the logs from the requested RTC Component. Use the Magnifying Lens (Show Logs) Button in the desired component to display its logs in this table.

Columns Shown

By default, the columns shown are:

• Level

• Date

• Message

• App Name

The user may select a different set of columns by accessing the Logs Table Header context menu. To activate this menu, the user has to right click on the headers of the Logs Table. This will show which columns are shown and which are not, as shown in Fig. 19. From the same menu, the user may add or remove columns.

Fig. 19 Logs: Column selection using context menu.

Pause and Lock Operations

Logs are parsed from the files and new entries appear every second. The table will also automatically scroll to the latest entry when they appear. This behaviour is useful for a quick glance, but can be annoying when looking for a specific problem.

By right clicking on the entries section of the table, you can access a context menu that will change this behaviour (See Fig. 20).

• Select All: selects all logs entries currently on the table.

• Deselect All: selects all logs entries currently on the table.

• Copy selection: copies to clipboard the entries as a CSV.

• Pause: No more entries will be presented to the user. They are still being cached into memory, but until the user presses again the Pause button, they are not gonna be added to the table. Its default state is off.

• Scroll-lock: When on, it moves to the last added entry. When off, the user may scroll freely while entries are added to the table. Its default state is on.

• Clear the view: Removes all entries from the table.

• Hide Toolbar: The Logs table toolbar is hidden by default, leaving more space for the logs. But if the user requires filtering operations, they can turn off this option to have access to the Toolbar with more tools.

Fig. 20 Logs: Options are hidden in the table context menu.

Toolbar

When the toolbar is shown using the context menu entry tools, the following options become available:

• Pause: No more entries will be presented to the user. They are still being cached into memory, but until the user presses again the Pause button, they are not gonna be added to the table. Its default state is off.

• Scroll-lock: When is on, it moves to the last added entry. When off, the user may scroll freely while entries are added to the table. Its default state is on.

• Clear the view: Removes all entries from the table.

• Text Elide: Column space is forced to the current size of the messages.

• Level: FATAL > ERROR > WARN > INFO > DEBUG > TRACE. The Level Combo Box in the Toolbar sets the minimum level shown.

Fig. 21 Logs: Filtering by log level.

For example, in Fig. 22 the Control and Monitoring Tool had its column Hostname added and filter of “Message LIKE(Action.*)” set.

• From: Logs earlier than this will not be shown.

• To: Logs later than this will not be shown.

• Filter: a string the specifies how to filter the different columns. Filtering is explained in https://www.eso.org/~eltmgr/CII/latest/manuals/html/docs/log.html#view-filtering

Fig. 22 Logs: Example showing different columns, and message filtering.

Errors

In case an error or exception is reported while executing the command, a dialog showing the error appears, such as in Figure Fig. 23.

Fig. 23 Error Dialog shown when an exception is received while invoking a remote Command.

The dialog includes:

• A summary of the error.

• A “Show details…” button which expands the dialog and shows the stack trace of the exception.

• A combobox with “Copy to Clipboard” and “email” options.

• A checkbox “Don’t show this error message again”, which disable the error dialog. This can be renabled from the Menu Bar ‣ View ‣ Toggle Error Dialogs.

Errors that are important to the user are presented within a light-red background section, as in Fig. 24.

Fig. 24 Errors presented to the user.

The application will try to continue working with the limitations, but the message will only disappear when the user acknowledges it by pressing the X Button.

To find more information, the logs can be checked. Filtering by log level ERROR can be useful.

Limitations and Known Issues

• At the moment, only the GUI mode is functional, the command line mode (CLI-mode) with subcommands like send and state is not yet available.

• Log messaging in Consul and in GUI needs to be improved.

• The log view does not properly display multi-line log messages (RTCTK-954).

• The command reply history log does not update for all commands (RTCTK-1326).

• If the DISPLAY environment variable is not set, rtctkCtrlMonTool gui will crash on startup (RTCTK-1330).