ELT Gateway

Configuration

The ELT gateway relies on the CII configuration API (configng). Currently, only local configuration is supported.

Local

The configuration is defined in a yaml file that is stored locally. The directory of this file shall be available in the environment variable CFGPATH that contains a colon separated list of folders where configuration files may be stored. As we are using nomad to manage the gateway processes, CFGPATH is set in the nomad configuration file and will point to the nomad allocation directory, see Gateway startup and stop and APPENDIX, its value shall not be changed. The configuration file is generated from a template file that shall be modified as specified below. This template file is located in eltgw_root/resource/config/evltgw/eltGW.tpl.yaml. eltgw_root is the root directory where all configuration files and executables are located, in principle /elt/eltgw and shall be defined in the nomad configuration file as described in Installation.

Moreover, as we are using consul for service registration, the two parameters mal.container_uri and mal.state_pub_uri that indicate the location of the gateway interface to the instrument shall preferably be set by using the NOMAD_PORT_services_rep and NOMAD_PORT_status_pub environment variables.

The yaml template configuration file shall define the following settings (see Example):

oldb.rep_uri: url of the oldb update server.

oldb.workers_nb: number of threads of the oldb update server.

oldb.vlt_gw_uri: gateway uri for handling elt oldb update requests.

oldb.scanning.data_points: list of elt oldb data points to be scanned, root url followed by ** will get the complete tree.

vlt.rep_uri: uri of servers waiting for vlt commands.

vlt.workers_nb: number of threads used in aforementioned server.

mal.vlt_gw_uri: url of the vlt gateway component.

mal.vlt_tcs_env: name of the telescope environment.

mal.vlt_tcs_tif: name of the tif control process.

mal.container_uri: uri of the mal interface, preferably set with nomad environment variable ‘zpb.rr://*:{{ env “NOMAD_PORT_services_rep” }}/’

mal.state_pub_uri: uri of the state publisher, preferably set by querying consul with ‘zpb.ps://*:{{ env “NOMAD_PORT_status_pub” }}/std/status’

mal.binary_location: location on the elt gateway of the binary tables.

alarm.uris: list of alarm uris

alarm.cas_location: location of the cas server

alarm.cas_uris: list of alarm uris requiring cas notification

datalab.location: location of Datalab Elastic Search server.

datalab.system: system (for instance instrument) the gateway belongs to.

datalab.envname: name of the gateway software environment.

tcs.state.oldb.uri: tcs state oldb uri.

Remote

Configuration on a remote server is currently not supported by configng.

Example

Below an example of configuration stored in a yaml file

oldb.rep_uri: !cfg.type:string 'tcp://*:8000'
oldb.workers_nb: !cfg.type:int32 1
oldb.vlt_gw_uri: !cfg.type:string 'tcp://vltgwtest:4445'
oldb.scanning.data_points: ["cii.oldb:///camera/**"]
vlt.rep_uri: !cfg.type:string 'tcp://*:8001'
vlt.workers_nb: !cfg.type:int32 10
mal.vlt_gw_uri: !cfg.type:string 'tcp://wvgw:4444'
mal.vlt_tcs_env: !cfg.type:string 'wxxtcs'
mal.vlt_tcs_tif: !cfg.type:string 'tifControl'
mal.container_uri: !cfg.type:string 'zpb.rr://*:{{ env "NOMAD_PORT_services_rep" }}/'
mal.state_pub_uri: !cfg.type:string 'zpb.ps://*:{{ env "NOMAD_PORT_status_pub" }}/std/status'
mal.binary_location: !cfg.type:string '/tmp'
alarm.uris: !cfg.type:vector_string  ['cii.oldb://alarmtest/tcs/humidity', 'cii.oldb://alarmtest/tcs/seeing']
alarm.cas_location: !cfg.type:string  'http://localhost:8080'
alarm.cas_uris: !cfg.type:vector_string  ['cii.oldb://alarmtest/tcs/temp1', 'cii.oldb://alarmtest/tcs/temp2']
datalab.location: !cfg.type:string 'http://ciihost:9200'
datalab.system: !cfg.type:string 'TEST_INST'
datalab.envname: !cfg.type:string 'TEST_GATEWAY'
tcs.state.oldb.uri: "cii.oldb:///TCS/tcsstate/tcsstate"

Logging

If no custom configuration is provided, a default configuration will be used. This configuration will have:

A root logger configured to an ERROR level with two appenders attached
An stdout appender using a CiiSimpleLayout layout.

A custom configuration can otherwise be defined in JSON format in a file that will be passed as a parameter to the application. The location of this file can be specified otherwise the application will look for it in the default folder $ELTGW_ROOT/resource/config. The configuration file shall contain a definition of loggers and appenders. In the example below, a root logger is defined with logging level set to info and a logger named evltgw.eltGW.eltGateway.ELTGateway is defined with logging level set to debug. These loggers have two apppenders, one for sending logging to the standard output and one that transmit to syslog.

{
    "version": 1,
    "handlers": {
        "console": {
            "class": "logging.StreamHandler",
            "formatter": "cii-brief",
            "stream": "ext://sys.stdout"
        },
        "cii-syslog": {
            "class": "logging.handlers.SysLogHandler",
            "formatter": "cii"
        }
    },
    "root": {
        "handlers": [ "console", "cii-syslog" ],
        "level": "INFO"
    }
    "loggers": {
     "evltgw.eltGW.eltGateway.ELTGateway": {
         "handlers": [ "console", "cii-syslog" ],
         "level": "DEBUG"
     }
   }
}

OLDB

On the eltGW host, the telescope runtime data is stored in OLDB in a tree named tcs. It is a counterpart to the tree defined in the vltGW environment. The creation of this tree is done with an independent command named eltGWOldbInit. It shall be launched if no tcs tree is available in OLDB before starting the elt gateway but it is not necessary to launch it systematically. It takes two parameters:

tcs tree configuration file (mandatory). You can use oldbConfig.yaml that is installed in $ELTGW_ROOT/resource/config. This file is a copy of outScanVGWConfig.yaml that is used by the VLT gateway component that contains a definition of the vlt oldb tcs.

a log4j configuration file (optional)

Example: eltGWOldbInit oldbConfig.yaml $ELTGW_ROOT/resource/config/custom_logging.json

VLT Gateway

On the VLT gateway side, the following four processes shall be running:

inMsgVGW

Configuration

The following settings shall be initialized, see example at inMsgVGWConfig.yaml

ZMQ:
- URL_WORKER: url for in-process communication between workers and the DEALER socket
- URL_CLIENT: url used by the VLT gateway to receive commands coming from the ELT gateway
- Workers: Number of threads to handle commands
TCS:
- Fits_Keywords_Dir : location of fits binary tables

inMsgVGWConfig.yaml

ZMQ:
  URL_WORKER: "inproc://workers".
  URL_CLIENT: "tcp://192.168.56.201:4444".
  Workers: 5
TCS:
  Fits_Keywords_Dir: '/data/VGW/INS_ROOT/SYSTEM/DETDATA'

inScanVGW

Configuration

The following settings shall be initialized, see example at inScanVGWConfig.yaml:

ZMQ:
- URL_WORKER: url for in-process communication between workers and the DEALER socket
- URL_CLIENT: url used by the VLT gateway to receive oldb updates coming from the ELT gateway
- Workers: Number of threads to handle oldb updates
OLDB:
- ENVIRONMENT: oldb environment where scanned points are located.
- ELTGW_INIT_SERVER: url of the elt gw for an initialization with oldb values

inScanVGWConfig.yaml

ZMQ:
  URL_WORKER: "inproc://workers".
  URL_CLIENT: "tcp://*:4445"
  Workers: 5
  OLDB:
ENVIRONMENT: wvgw
ELTGW_INIT_SERVER: "tcp://somehost:8001"

outMsgVGW

Configuration

The following settings shall be initialized, see example at outMsgVGWConfig.yaml:

ZMQ:
- ELTGW_CMD_SERVER: gateway uri where to forward VLTINS commands.
- SEND_TIMEOUT: timeout in second for sending data
- ELTGW_CLIENT_POOL_SIZE: pool size of clients to send commands to the ELTGW.

outMsgVGWConfig.yaml

ZMQ:
   ELTGW_CMD_SERVER: "tcp://134.171.3.56:8001"
   SEND_TIMEOUT: 200
   ELTGW_CLIENT_POOL_SIZE: 10

outScanVGW

Configuration

The following settings shall be initialized, see example at outScanVGWConfig.yaml:

ZMQ:

REQUEST_TIMEOUT: Timeout in s for oldb update requests sent to the ELT gateway

REQUEST_RETRIES: number of retries when sending an oldb update requests

SERVER_ENDPOINT: location of the gateway to send oldb update requests

BaseValue: root location in oldb of tcs data points

ScanPoints: list of data points to be monitored

outScanVGWConfig.yaml

ZMQ:
  REQUEST_TIMEOUT: Timeout in ms for oldb update requests sent to the ELT gateway, e.g. 2500
  REQUEST_RETRIES: number of retries when sending an oldb update requests, e.g. 3
  SERVER_ENDPOINT: location of the gateway to send oldb update requests, e.g.
                   "tcp://192.168.56.202:8000"

BaseValue: root location in oldb of tcs data points, e.g. :Appl_data:TCS

ScanPoints: list of data points to be monitored, e.g.:
  - tcsState.tcsState
  - tcsState.tcsSubstate
  - tcsState.track
  - tcsState.focus
  - tcsState.domeStatus
  - times.lst
  - times.utc
  - coord.ha
  - coord.ra
  - coord.dec
  - coord.ra2000
  - coord.dec2000
  - coord.ra2000deg
  - coord.dec2000deg
  - coord.alt
  - coord.az
  - coord.altRef
  - coord.azRef
  - coord.prltic
  - track.airmass
  - track.dvelra
  - track.dveldec
  - track.rotAngle
  - track.adaAngle
  - track.adaPosAngle
  - track.remtime
  - track.remlimit
  - track.trackingType
  - track.trackingLost
  - track.remTimeLow
  - guide.status
  - guide.ra
  - guide.dec
  - guide.probeX
  - guide.probeY
  - guide.probeRa
  - guide.probeDec
  - guide.probeInPos
  - guide.gpVignetting
  - guide.ccdOnSky
  - guide.pupilOnSky
  - guide.guidingLost
  - m2.focuValue
  - info.telescop
  - info.version
  - info.installed
  - info.oper
  - info.foculenNA
  - info.foculenCA
  - info.foculenCO
  - info.focuscaleNA
  - info.focuscaleCA
  - info.focuscaleCO
  - site.longi
  - site.lati
  - site.level
  - ao.m1Corrected
  - ao.m2Corrected
  - ao.commandState
  - ao.targetSettings
  - ao.currentIndexAvg
  - ao.focusTerm
  - chop.status
  - chop.startTime
  - chop.stopTime
  - chop.freq
  - chop.throw
  - chop.posOptImage
  - chop.peakValleyRatio
  - chop.telPointingAxis
  - chop.posAng
  - chop.offsetChop
  - chop.guide
  - chop.parmChange
  - asm.seeing
  - asm.windspeed
  - asm.winddir
  - asm.pressure
  - asm.humidity
  - asm.temperature
  - asm.lapseRate
  - moon.RA
  - moon.dec
  - m1.temperature
  - adc.actDist
  - hb.tempSer1
  - hb.tempSer2
  - hb.tempSer3
  - hb.tempSer4
  - hb.tempCnt1
  - hb.tempCnt2
  - hb.tempFlx1
  - hb.tempFlx2
  - hb.tempFlx3
  - hb.tempFlx4

Telemetry

Telemetry ingestion consists of ingesting logs issued by the INSSW and telemetry data into the Datalab elastic search. Logs and telemetry data will flow to elastic search through a chain of services that are part of the infrastructure so that the gateway will not be directly involved. We will just need to configure these services so that they know where data comes from and where it shall be sent to.

Logs ingestion

Two services are involved, Filebeat and logstash. A filebeat instance will run on the IWS as a systemd service (see Filebeat service file). It will get as a parameter a configuration file (see Filebeat configuration file) where the two following fields shall be set:

Filebeat.inputs: paths: location of the log files, e.g. /var/log/elt/*.log
output.logstash:
- hosts: list of hosts where to sent the data, e.g. [“ciilogstashhost:5044”]
- http.enabled: true
- logging.level (optional): info, debug, trace or warn.

Filebeat will serialize all the new logs and send them to the logstash instance specified in the configuration file. The logstash instance will run on a central server providing cii services. The deployment and configuration of logstash is out of the scope of this user manual.

Filebeat service file

Below an example of a filebeat service file:

[Unit]
Description=Filebeat sends log files to Logstash or directly to Elasticsearch.
Documentation=https://www.elastic.co/products/beats/filebeat
Wants=network-online.target
After=network-online.target
[Service]
ExecStart=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat
Restart=always
[Install]
WantedBy=multi-user.target

Filebeat configuration file

Below an example of a filebeat configuration, its location is passed to filebeat with the -c option:

# Internal queue configuration for buffering events to be published.
queue:
  # Queue type by name (default 'mem')
  # The memory queue will present all available events (up to the outputs
  # bulk_max_size) to the output, the moment the output is ready to server
  # another batch of events.
  mem:
    # Max number of events the queue can buffer.
    events: 4096

    # Hints the minimum number of events stored in the queue,
    # before providing a batch of events to the outputs.
    # The default value is set to 2048.
    # A value of 0 ensures events are immediately available
    # to be sent to the outputs.
    flush.min_events: 500

    # Maximum duration after which events are available to the outputs,
    # if the number of events stored in the queue is < min_flush_events.
    flush.timeout: 1s

#=========== Filebeat modules ===============================
filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

#=========================== Filebeat inputs =====================
filebeat.inputs:
  - type: log
    # Change to true to enable this input configuration.
    enabled: true
    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      - /var/log/elt/*.log
    ### Multiline options

    # Multiline can be used for log messages spanning multiple lines. This is common
    # for Java Stack Traces or C-Line Continuation

    # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
    multiline.pattern: ^Cref

    # Defines if the pattern set under pattern should be negated or not. Default is false.
    multiline.negate: true

    # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
    # that was (not) matched before or after or as long as a pattern is not matched based on negate.
    # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
    multiline.match: after

    multiline.timeout: 1s

#================================ Outputs ==================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Console output ---------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ciilogstashhost:5044"]



#================================ Logging ============================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug

#======================== HTTP Endpoint   ============================
# Each beat can expose internal metrics through a HTTP endpoint. For security
# reasons the endpoint is disabled by default. This feature is currently experimental.
# Stats can be access through http://localhost:5066/stats . For pretty JSON output
# append ?pretty to the URL.

# Defines if the HTTP endpoint is enabled.
http.enabled: true

Telemetry data ingestion

Deployment

There will be for each instrument an instance of the telemetry archiver running on the elt gateway host. A Telemetry archiver will be run as a systemd service (see Telemetry service file) on the elt gateway host. It will ingest all the updated data into the archive index of elastic search. For the time being, large data ingestion and ad-hoc ingestion of data submitted by client application are not supported. Moreover, the telemetry is not part of the ELT DevEnv so that the project cii-srv/srv-telemetry shall be compiled and installed.

Configuration

The configuration is provided through the configuration service and is an instance of the CiiTelemetryConfigClass. Its location is specified through a uri assigned to the TELEMETRY_CONFIG environment variable (e.g. cii.config://*/telemetry/forsup/config/archiverconfig). The most important configuration settings are:

telemetryServiceURI: location of the telemetry archiver, e.g.: zpb.rr://0.0.0.0:9115
largeStorageServiceURI: location of the repository for large data (>2GB), e.g.: http://ciihdfshost:9870.
engineeringArchiveURI: location of the engineering archive (elastic search), e.g.: http://ciiarchivehost:9200
engineeringArchiveBackupURI: location of the backup engineering archive, e.g.: http://ciiarchivehost:9200
serviceRange: list of service ranges monitored by the telemetry archive, e.g: [“cii.config://*/telemetry/forsup].

A telemetry archiver will subscribe to all the oldb data points specified in the configuration service by the Data Capture Configuration instances (CiiDataCaptureConfig class) mapped to the service range of the telemetry archiver. These Data capture configurations are defined under serviceRange/dataCapture and their identifier can be generated with uuidgen, e.g. cii.config://*/telemetry/forsup/datacapture/baa89fb-3163-4993-b6aa-d1082ea13b69.

A data capture configuration contains the following fields:

minimuminterval: minimum interval between two attribute updates, format: [mm:ss], e.g. “01:00” for a minimum of one minute between two intervals.
maximuminterval: time interval after which an update will be done in the telemetry in case the monitored attribute has not changed.
deltaType: ABSOLUTE for absolute value change, RELATIVE for relative value change
deltaValue: absolute minimum value variation for update if deltaType is ABSOLUTE, relative minimum change if deltaType is RELATIVE.
dataPointAddress: location of the oldb point to be monitored (e.g. cii.oldb:///tcs/tcsstate/tcsstate)
archivingEnable: true if data is to be archived.
archiveMode: VALUE for oldb data point value changes, METADATA for oldb data point metadata changes, QUALITY for oldb data point quality change. QUALITY mode is currently not supported as it puts a heavy load on oldb.

Telemetry service file

Below an example of a telemetry service file:

[Unit]
Description=Run srv-telemetry service for telemetry archiving
After=network.target

[Service]
Environment=INTROOT=/home/eltdev/INTROOT/
Environment=MAL_ROOT=/elt/mal
Environment=CIISRV_ROOT=/elt/ciisrv
ExecStart=/home/eltdev/INTROOT/bin/telemetry-service
Type=simple
Restart=always
RestartSec=5
# Timeout before service is forcefully closed, should be larger than service shutdown timeout, currently set to 60 seconds.
TimeoutSec=90

[Install]
WantedBy=multi-user.target