12. Service Management

Revision:

1.0

Status:

New

Repository:

https://gitlab.eso.org/cii/info/cii-docs

Project:

ELT CII

Document ID:

Owner:

Marcus Schilling

Last modification:

October 4, 2021

Document History

Revision

Date

Changed/rev iewed

Section(s)

Modificatio n

1.0

04.10.2021

mschilli

All

Created.

Confidentiality

This document is classified as a confidential document. As such, it or parts thereof must not be made accessible to anyone not listed in the Audience section, neither in electronic nor in any other form.

Scope

This document is a user manual for the ELT Core Integration Infrastructure Software project.

Audience

This document is aimed at those ESO employees and ESO partners involved with the ELT Core Integration Infrastructure Software project, as well as other Users and Maintainers of the ELT Core Integration Infrastructure Software.

12.1. Overview

This document is a user manual for CII Services management tooling.

12.2. Introduction

This document describes the tools available for the management of CII Services

  • cii-postinstall - for setting up CII services on a host

  • cii-services - for starting / stopping / monitoring of the CII Services


12.3. cii-postinstall

You run the CII-PostInstall on a freshly installed DevEnv machine, or after an upgrade of the DevEnv. You choose a role for the host, and the utility will apply the corresponding configuration steps. For most configuration steps, root privileges are needed.

Usage

$ /elt/ciisrv/postinstall/cii-postinstall <role>

Run the command without arguments to see its help.

12.3.1. Roles

The role “ownserver” configures the services for local development on a single machine.

$ /elt/ciisrv/postinstall/cii-postinstall  role_ownserver

The role “groupserver” does a few more things than the one above. It runs the services with more resources and makes the services/databases accessible from other hosts.

$ /elt/ciisrv/postinstall/cii-postinstall  role_groupserver

The role “groupclient” configures the host to use the services provided by a groupserver. For example, assuming host “elthlccd20” has been post-installed with the role “groupserver”, then, to let applications on your host use the CII Services on that groupserver, you do:

$ /elt/ciisrv/postinstall/cii-postinstall  role_groupclient  elthlccd20

Switching Roles?

It is possible to run the utility again later, but your mileage will vary: some role-switches will work, others not.

12.3.2. Subcommands

A “role” is a collection of configuration steps, and those configuration steps can also be applied one-by-one, by executing them as a “subcommand”. To see the available subcommands, just pass “help”.

$ cii-postinstall help
CII PostInstall (20210730)
Available subcommands: role_ownserver  role_groupserver  role_groupclient  ECII397 logsink  filebeat  logstash  kibana  logtrail  serverlocation  elastic  schemas  unrestrict  oldb  schemas_local  help

12.3.3. Other Functions

In addition, there are some configuration steps that are outside of any role. You can run them as normal user (no root privileges) to configure your personal environment.

Example: Set up a Local-DB so you can use the “Config from Local Files” feature of the CII Config Service.

$ cii-postinstall schemas_local /home/eltdev/INTROOT/localdb
[...]

Tip  If you create the Local-DB at "$INTROOT/localdb" or "$HOME/localdb", the Config Service will find it there.
If you choose another location, also define this variable: export CONFIG_LOCAL_DB=/tmp/delphi/localdb

12.4. cii-services

The cii-services utility lets you start/stop/monitor the CII services. For some operations, it requires root-privileges -> unless you are running it with root-privileges, it will show a password prompt when needed.

Usage

$ cii-services
Available subcommands: info status logs start stop

12.4.1. Monitoring/Status Check

info This is a deployment-centric view that tells you whether the services are running and where they are.

$ cii-services info
CII Services Tool (20210706)
Collecting information........
Collecting information........................

Installations on this host:
ElasticSearch   |active:yes     |install:/usr/share/elasticsearch/bin/elasticsearch
Redis           |active:yes     |install:/opt/anaconda3/bin/redis-server
MinIO           |active:no      |install:/usr/local/bin/minio
ConfigService   |active:yes     |install:/elt/ciisrv/bin/config-service         |ini:
TelemService    |active:no      |install:
Filebeat        |active:yes     |install:/usr/share/filebeat/bin/filebeat
Logstash        |active:yes     |install:/usr/share/logstash/bin/logstash
Kibana          |active:yes     |install:/usr/share/kibana/bin/kibana
Jaeger          |active:no      |install:
AlarmServices   |active:no      |install:
AlarmPlugin     |active:no
AlarmConsumer   |active:no
AlarmConverter  |active:no
AlarmSupervisor |active:no
ConfigClient    |ini:

Addresses of Services:
Config / Local-DB       |access:yes     |file:/home/eltdev/localdb
Config / ElasticSearch  |access:yes     |host:ciielastichost(IP:127.0.0.1)
Config / MinIO          |access:no      |host:ciihdfshost(IP:)
Config / Service        |access:yes     |host:ciiconfservicehost(IP:127.0.0.1)  |host2:ciiconfservicehost2(IP:127.0.0.1)
OLDB / Redis            |access:yes     |host:localhost(IP:::1)
OLDB / MinIO            |access:no      |host:ciihdfshost(IP:)
Telem / Service         |access:no      |host:ciiarchivehost(IP:127.0.0.1)

Statistics of Services:
Config / Service        |NrOfRequests: 2        |UpTime: 4 minutes
OLDB / Redis            |total_connections_received:3   |rejected_connections:0
Telemetry / Service

status This is a feature-centric view, that basically tells you which features you have available. For example, “Blob Values” means that the Distributed File System MinIO (in previous versions: Hadoop) is available for storage of large values and binaries.

$ cii-services status
CII Services Tool (20210706)
Collecting information........

Status Summary:
OK Config from Central Server
OK Config from Local Files
-- Config with Blob Values
OK OLDB Normal Mode
-- OLDB with Blob Values
-- Telemetry

logs Privileged Operation. Run the command as root.

$ su -c "cii-services logs"
Password:
CII Services Tool (20210706)

=== ConfigService Journal ==============
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,979 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=StatusLogger
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,980 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=ContextSelector
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,981 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=Loggers,name=
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,985 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=Appenders,name=console
[...]

=== Logstash Journal ===================
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
Sep 07 15:32:11 elthlccd20 systemd[1]: Started logstash.
Sep 07 15:32:24 elthlccd20 logstash[33887]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Sep 07 15:32:25 elthlccd20 logstash[33887]: [2021-09-07T15:32:25,370][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.3"}
Sep 07 15:32:26 elthlccd20 logstash[33887]: [2021-09-07T15:32:26,183][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[...]

=== TelemService Journal ===============
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
-- No entries --

12.4.2. Start / Stop

Privileged Operations. You will be asked for the root password.

$ cii-services start
CII Services Tool (20210706)
Possible names: logKibana redis elasticsearch logLogstash configService logFilebeat, or a prefix thereof, or 'all'


$ cii-services start all
CII Services Tool (20210706)
About to start logKibana redis elasticsearch logLogstash configService logFilebeat ...
Password:


$ cii-services stop all
CII Services Tool (20210706)
About to stop logKibana redis elasticsearch logLogstash configService logFilebeat ...
Password:

12.5. Example

(base) elthlccd20 eltdev:~ 8 > cii-services info

CII Services Tool (20210706)
Collecting information........
Collecting information........................

Installations on this host:
ElasticSearch   |active:no      |install:/usr/share/elasticsearch/bin/elasticsearch
Redis           |active:no      |install:/opt/anaconda3/bin/redis-server
MinIO           |active:no      |install:/usr/local/bin/minio
ConfigService   |active:no      |install:/elt/ciisrv/bin/config-service      |ini:
TelemService    |active:no      |install:
Filebeat        |active:no      |install:/usr/share/filebeat/bin/filebeat
Logstash        |active:no      |install:/usr/share/logstash/bin/logstash
Kibana          |active:no      |install:/usr/share/kibana/bin/kibana
Jaeger          |active:no      |install:
AlarmServices   |active:no      |install:
AlarmPlugin     |active:no
AlarmConsumer   |active:no
AlarmConverter  |active:no
AlarmSupervisor |active:no
ConfigClient    |ini:

Addresses of Services:
Config / Local-DB       |access:no      |file:/home/eltdev/localdb
Config / ElasticSearch  |access:no      |host:ciielastichost(IP:)
Config / MinIO          |access:no      |host:ciihdfshost(IP:)
Config / Service        |access:no      |host:ciiconfservicehost(IP:)        |host2:ciiconfservicehost2(IP:)
OLDB / Redis            |access:no      |host:localhost(IP:::1)
OLDB / MinIO            |access:no      |host:ciihdfshost(IP:)
Telem / Service         |access:no      |host:ciiarchivehost(IP:)

Statistics of Services:
Config / Service
OLDB / Redis
Telemetry / Service

How to read this output:

This section reports some info about the tool itself

CII Services Tool (20210706)
Collecting information........
Collecting information........................

=> Tool version and progress dots. => If an error occurs while collecting information, we count the progress dots to spot the problem.

This section reports where applications on your host will find the CII services, and whether the cii-services tool was able to access them (= talk to them). The services may be running on your host (see next paragraph) or on another host.

Addresses of Services:
Config / Local-DB       |access:no      |file:/localdb
Config / ElasticSearch  |access:no      |host:ciielastichost(IP:)
Config / MinIO          |access:no      |host:ciihdfshost(IP:)
Config / Service        |access:no      |host:ciiconfservicehost(IP:)
OLDB / Redis            |access:no      |host:localhost(IP:::1)
OLDB / MinIO            |access:no      |host:ciihdfshost(IP:)
Telem / Service         |access:no      |host:ciiarchivehost(IP:)

=> The empty parentheses “(IP:)” tell us that hostname resolution is not configured. => This means applications running on your host are not able to locate the CII services. => You need to configure your host, see cii-postinstall

This section reports which CII services are installed on your host, and whether they are running.

Installations on this host:
ElasticSearch   |active:no      |install:/usr/share/elasticsearch/bin/elasticsearch
Redis           |active:no      |install:/opt/anaconda3/bin/redis-server
MinIO           |active:no      |install:/usr/local/bin/minio
ConfigService   |active:no      |install:/elt/ciisrv/bin/config-service         |ini:
TelemService    |active:no      |install:
Filebeat        |active:no      |install:/usr/share/filebeat/bin/filebeat
Logstash        |active:no      |install:/usr/share/logstash/bin/logstash
Kibana          |active:no      |install:/usr/share/kibana/bin/kibana
Jaeger          |active:no      |install:
AlarmServices   |active:no      |install:
AlarmPlugin     |active:no
AlarmConsumer   |active:no
AlarmConverter  |active:no
AlarmSupervisor |active:no
ConfigClient    |ini:

=> Several services (not: Telemetry and Alarms) are installed on your host. None of them is running on your host. => If you plan to use CII services running on another host, this is fine. => Otherwise, you need to start the services, see above.

This section reports some statistics from the services.

Statistics of Services:
Config / Service
OLDB / Redis
Telemetry / Service

=> An empty output like this again indicates that the the services are not accessible, but you knew that already.

Preview: After configuring and starting the services, the output will change to something like this:

CII Services Tool (20210706)
Collecting information........
Collecting information........................

Installations on this host:
ElasticSearch   |active:yes     |install:/usr/share/elasticsearch/bin/elasticsearch
Redis           |active:yes     |install:/opt/anaconda3/bin/redis-server
MinIO           |active:no      |install:/usr/local/bin/minio
ConfigService   |active:yes     |install:/elt/ciisrv/bin/config-service         |ini:
TelemService    |active:no      |install:
Filebeat        |active:yes     |install:/usr/share/filebeat/bin/filebeat
Logstash        |active:yes     |install:/usr/share/logstash/bin/logstash
Kibana          |active:yes     |install:/usr/share/kibana/bin/kibana
Jaeger          |active:no      |install:
AlarmServices   |active:no      |install:
AlarmPlugin     |active:no
AlarmConsumer   |active:no
AlarmConverter  |active:no
AlarmSupervisor |active:no
ConfigClient    |ini:

Addresses of Services:
Config / Local-DB       |access:yes     |file:/home/eltdev/localdb
Config / ElasticSearch  |access:yes     |host:ciielastichost(IP:127.0.0.1)
Config / MinIO          |access:no      |host:ciihdfshost(IP:)
Config / Service        |access:yes     |host:ciiconfservicehost(IP:127.0.0.1)  |host2:ciiconfservicehost2(IP:127.0.0.1)
OLDB / Redis            |access:yes     |host:localhost(IP:::1)
OLDB / MinIO            |access:no      |host:ciihdfshost(IP:)
Telem / Service         |access:no      |host:ciiarchivehost(IP:127.0.0.1)

Statistics of Services:
Config / Service        |NrOfRequests: 2        |UpTime: 4 minutes
OLDB / Redis            |total_connections_received:3   |rejected_connections:0
Telemetry / Service