Getting Started

This section will guide the ICS software developer in creating a working project using a provided cookiecutter template. For getting the template from GitLab, go to the ESO Gitlab site and download the tar file of the template: Template release

The user will be able to start the software components with the specific configuration prepared as a showcase for instrument developers. More details for each of the components will be given in the respective user manuals.

Updating an existing Project Configuration

If a version of your instrument project already exists, it is recommended to start from scratch following the instructions below and updating your specific configuration in the new template afterwards. This approach might be better because of the possible changes in the template configuration and in the IFW components. If you have problems doing this, please contact ESO to get help doing this migration. Find here some guidelines to do the porting from previous version: Software Porting Guidelines.

Creating a Project Configuration

The IFW includes a project template that can be used to generate the initial package of an instrument. The generated project can be considered as a mini template instrument that could be used as starting point for the development of the control software. It is still basic but the idea is to develop it further in future versions according to the progress of the framework components.

The generated directory contains a fully working waf project with the instrument directory structure, some configuration files and some custom subsystem samples, e.g. an FCS including a special device. In this example we will use “micado” as an example instrument. After executing the cookiecutter command with the provided template, the system will request the user input to enter the information for the generation of the configuration and customized code. This template also generates the code for a special FCF device that in this case we will name as “mirror”. The ‘component_name’ is referring to an instance of FCS.

> cookiecutter ifw-template/project
[1/10] project_name (myproject): micado
[2/10] project_description (my project description): MICADO project
[3/10] project_prefix (xxx): mcd
[4/10] nomad_user (eltdev):
[5/10] component_name (mycomponent): fcs
[6/10] device_name (mydevice): mirror
[7/10] olas_directory (): <path to OLAS directory>
[8/10] hlcc_available (No):
[9/10] hlcc_uri (zpb.rr://127.0.0.1:000):
[10/10] ngc2o_available (No): No

Note

The OLAS directory is where the FITS files will be stored after they are completed to be sent to the archive system.

Note

You can skip the usage of HLCC by setting the hlcc_available to No.

The generated directory structure including the first two levels is shown below. In this case, the directory mcd-ics is a waf project that can be built. Same goes for mcd-ipt The resource directory is meant for storing the instrument resources like configuration files.

micado                  # Instrument repository
├── mcd-ics             # Valid waf project
│   ├── build
│   ├── fcs             # Custom FCF instance
│   ├── mcdstoo         # Startup/Shutdown sequencer scripts
│   ├── seq             # Sample template implementation.
│   └── wscript
├── mcd-ipt             # Valid waf project to generate an Instrument Package
│   ├── build
│   ├── ipt             # Set of .ipt.yaml sources to generate an Instrument Package via ifwIpToolCompiler
│   ├── wscript
└── mcd-resource        # Instrument resource directory
    ├── config          # Configuration files
    ├── nomad           # Nomad job files
    └── seq             # Sample OB

After the new directory is created, one could build and install the generated software.

cd micado/mcd-ics
waf configure
waf build install

cd micado/mcd-ipt
waf configure
waf build install

Update CFGPATH environment variable

If not already done, the CFGPATH environment variable shall be updated in the LMOD configuration to include the template resource directory. Add the following line to the modulefiles/private.lua file.

prepend_path("CFGPATH","<path_to_template>/micado/mcd-resource")

Note

The above setting is needed for the proper functioning of the IFW components. Make sure the setting is correct before starting Nomad.

Nomad Namespace and Service Discovery

Starting with IFW 2026, the template instrument can use Nomad’s built-in service registry instead of Consul. Each instrument operates within its own Nomad namespace (e.g., MICADO), isolated from other instruments. The NOMAD_NAMESPACE environment variable must be set in private.lua to the instrument name, and NOMAD_DISCOVERY_SERVICE must be set to nomad.

This approach is optional. Instruments can continue using Consul for service discovery by setting CONSUL_HTTP_ADDR in their private.lua and omitting NOMAD_NAMESPACE and NOMAD_SERVICE_REGISTRATION.

To verify job status via CLI when using namespaces:

nomad job status -namespace MICADO

For more details on this feature, see Experimental Features.

Note

If you defined the environment variable NOMAD_NAMESPACE=MICADO, you do not need to pass it to the nomad commands

Starting/Stopping the ICS Software

The IFW uses Nomad (see here) to manage the life cycle of the ICS SW components following the recommendation from the ELT Control project. Service discovery can be handled either by Consul (the traditional approach) or by Nomad’s built-in service registry using namespaces (available starting with IFW 2026). The template provided with this release uses Nomad’s internal service registry by default. Instruments continuing to use Consul should configure CONSUL_HTTP_ADDR in their private.lua and omit NOMAD_NAMESPACE and NOMAD_SERVICE_REGISTRATION.

The project template includes the Nomad job configuration to start-up/shutdown the ICS components that are generated by the cookiecutter template. We are also providing a Startup/Shutdown Sequencer script that uses the Nomad jobs to start/stop the complete ICS SW resembling the osfStartup tool in the VLT.

Startup/Shutdown Contents

The project template comes with a predefined startup/shutdown script to start/stop a representative sample of ICS software processes. The list of processes is here:

• DDT broker

• CCF instance with Simulator and DDT publisher

• FCF Simulators (shutter, lamp and motor)

• Subsystem Simulators (subsim1, subsim2 and subsim3)

• Custom FCF server instance with custom device (mirror).

• Custom FCF simulator (mirror)

• HLCC processes (optional)

• NGCII Optical (optional)

• OCM instance

• DPM instance

• SLM instance

• System Supervisor

These components are obviously using simulators and not real hardware. The script shall be executed by the Sequencer.

The script contains three main parts:

1. Stop all processes

2. Start all processes

3. Move all processes to Operational state.

Startup/Shutdown script in the Sequencer.

Executing Startup/Shutdown Script

Starting Sequencer GUI

In a terminal, type the following command to start the sequencer GUI.

> seqtool gui

Note

Since IFW version 2024, you need to specify the GUI configuration to enable the support of Instrument Package (IP). We will see this case in a specific paragraph about the Instrument Pacakge.

Running the Startup Script

Once the Sequencer GUI is running. Load the startup script (micado/mcd-ics/mcdstoo/src/mcdstoo/startup.py) by selecting the Load Script option as shown in the following figure. It is assumed that the software has been already built and installed.

Load script option from Sequencer File menu.

To execute the script, just press the play icon at the top of the Sequencer GUI as it is shown in the next figure. At the end of the execution, all instrument jobs shall be running and the system should be in Operational state.

Run script via the play icon.

A quick way to verify is to check the status of the Supervisor.

> supClient `geturi syssup-req` GetStatus
Operational;Idle

After a successful execution of the startup script, the nomad web UI can be used to verify the status of the nomad jobs. A total of 15 jobs shall be running.

List of TINS Nomad Jobs.

For instruments using Consul: The Consul UI can be used to verify the services registered. In this case the number of services is greater because in some cases there are two services defined per each Job.

List of TINS Consul Services.

Note

As convention, we use <service>-req as the registered service name for the request/reply port, e.g. fcs-req.

As convention, we use <service>-pub as the registered service name for the publish/subscribe port, e.g. fcs-pub.

This naming convention applies regardless of whether Consul or Nomad’s service registry is used.

Troubleshooting

If all or some processes do not start, make sure of the following:

1. Check that Nomad (and Consul, if using the Consul discovery approach) has been started and is running correctly. Try using the Systemd commands to get status of the service, see below.

> systemctl status nomad
* nomad.service - Nomad
Loaded: loaded (/usr/lib/systemd/system/nomad.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2021-04-20 07:14:11 UTC; 3 weeks 0 days ago
 Docs: https://nomadproject.io/docs/
Main PID: 413992 (nomad)
    Tasks: 541
Memory: 969.4M
CGroup: /system.slice/nomad.service
       ├─ 413992 /opt/nomad/bin/nomad agent -config /opt/nomad/etc/nomad.d
       ├─2864103 /opt/nomad/bin/nomad logmon
       ...

2. Make sure eltdev user has properly defined its environment. All environment variables shall be defined under eltdev since it is at the end the user that runs the processes through Nomad.

3. Check the status information of Nomad (and Consul, if applicable) with journalctl.

> journalctl -u nomad

4. Stop Nomad (and Consul, if applicable) and run them manually outside Systemd to get all logs and see the possible cause of the issues.

Validating the Software with a sample OB

We have prepared a very basic OB with an acquisition and observation template. The acquisition template prepares FCS, the camera system and the telescope simulator for the upcoming observation template that takes an image with the camera control system.

Note

The interaction between the Sequencer and the components is through the python client libraries provided by each component.

Before starting, the current script loaded in the Sequencer GUI must be cleared by pressing the reset button (trash icon).

Clear the current script and reset the server.

Then, the OB shall be loaded by pressing the open button as shown in the next figure. The path of the sample OB is: micado/mcd-resource/obd/tec/MICADO_OB_sample.json.

Load an OB.

To run the template, just press the play icon at the top of the Sequencer GUI.

Sample OB.

At the end of the execution, the image acquired by CCF shall be displayed in the DDT Viewer that it started by the template.

DDT Viewer with image received from CCF.

When HLCC is used, you can verify using the HLCC GUI that the telescope simulator is pointing to the right coordinates. You could change the OB parameters from the sequencer GUI and execute the OB again validating the new values are correctly received by the telescope simulator.

Note

For simplicity, we are currently only sending alpha and delta parameters.

The resulting FITS file generated by DPM is located under $DATAROOT/dpm/result.

Congratulations on reaching the end of the general Getting Started section. Further instructions you may find in the specific documentation of the components.

Updating Sample Configuration

To update the default configuration of the template, developers can modify the configuration files that are located under the resource directory, e.g. under resource/nomad.