12. Service Management¶
Revision: |
1.0 |
|---|---|
Status: |
New |
Repository: |
|
Project: |
ELT CII |
Document ID: |
|
Owner: |
Marcus Schilling |
Last modification: |
October 4, 2021 |
Revision |
Date |
Changed/rev iewed |
Section(s) |
Modificatio n |
|---|---|---|---|---|
1.0 |
04.10.2021 |
mschilli |
All |
Created. |
Confidentiality
This document is classified as a confidential document. As such, it or parts thereof must not be made accessible to anyone not listed in the Audience section, neither in electronic nor in any other form.
Scope
This document is a user manual for the ELT Core Integration Infrastructure Software project.
Audience
This document is aimed at those ESO employees and ESO partners involved with the ELT Core Integration Infrastructure Software project, as well as other Users and Maintainers of the ELT Core Integration Infrastructure Software.
12.1. Overview¶
This document is a user manual for CII Services management tooling.
12.2. Introduction¶
This document describes the tools available for the management of CII Services
cii-postinstall - for setting up CII services on a host
cii-services - for starting / stopping / monitoring of the CII Services
12.3. cii-postinstall¶
You run the CII-PostInstall on a freshly installed DevEnv machine, or after an upgrade of the DevEnv. You choose a role for the host, and the utility will apply the corresponding configuration steps. For most configuration steps, root privileges are needed.
Usage
$ /elt/ciisrv/postinstall/cii-postinstall <role>
Run the command without arguments to see its help.
12.3.1. Roles¶
The role “ownserver” configures the services for local development on a single machine.
$ /elt/ciisrv/postinstall/cii-postinstall role_ownserver
The role “groupserver” does a few more things than the one above. It runs the services with more resources and makes the services/databases accessible from other hosts.
$ /elt/ciisrv/postinstall/cii-postinstall role_groupserver
The role “groupclient” configures the host to use the services provided by a groupserver. For example, assuming host “elthlccd20” has been post-installed with the role “groupserver”, then, to let applications on your host use the CII Services on that groupserver, you do:
$ /elt/ciisrv/postinstall/cii-postinstall role_groupclient elthlccd20
Switching Roles?
It is possible to run the utility again later, but your mileage will vary: some role-switches will work, others not.
12.3.2. Subcommands¶
A “role” is a collection of configuration steps, and those configuration steps can also be applied one-by-one, by executing them as a “subcommand”. To see the available subcommands, just pass “help”.
$ cii-postinstall help
CII PostInstall (20210730)
Available subcommands: role_ownserver role_groupserver role_groupclient ECII397 logsink filebeat logstash kibana logtrail serverlocation elastic schemas unrestrict oldb schemas_local help
12.3.3. Other Functions¶
In addition, there are some configuration steps that are outside of any role. You can run them as normal user (no root privileges) to configure your personal environment.
Example: Set up a Local-DB so you can use the “Config from Local Files” feature of the CII Config Service.
$ cii-postinstall schemas_local /home/eltdev/INTROOT/localdb
[...]
Tip If you create the Local-DB at "$INTROOT/localdb" or "$HOME/localdb", the Config Service will find it there.
If you choose another location, also define this variable: export CONFIG_LOCAL_DB=/tmp/delphi/localdb
12.4. cii-services¶
The cii-services utility lets you start/stop/monitor the CII services. For some operations, it requires root-privileges -> unless you are running it with root-privileges, it will show a password prompt when needed.
Usage
$ cii-services
Available subcommands: info status logs start stop
12.4.1. Monitoring/Status Check¶
info This is a deployment-centric view that tells you whether the services are running and where they are.
$ cii-services info
CII Services Tool (20210706)
Collecting information........
Collecting information........................
Installations on this host:
ElasticSearch |active:yes |install:/usr/share/elasticsearch/bin/elasticsearch
Redis |active:yes |install:/opt/anaconda3/bin/redis-server
MinIO |active:no |install:/usr/local/bin/minio
ConfigService |active:yes |install:/elt/ciisrv/bin/config-service |ini:
TelemService |active:no |install:
Filebeat |active:yes |install:/usr/share/filebeat/bin/filebeat
Logstash |active:yes |install:/usr/share/logstash/bin/logstash
Kibana |active:yes |install:/usr/share/kibana/bin/kibana
Jaeger |active:no |install:
AlarmServices |active:no |install:
AlarmPlugin |active:no
AlarmConsumer |active:no
AlarmConverter |active:no
AlarmSupervisor |active:no
ConfigClient |ini:
Addresses of Services:
Config / Local-DB |access:yes |file:/home/eltdev/localdb
Config / ElasticSearch |access:yes |host:ciielastichost(IP:127.0.0.1)
Config / MinIO |access:no |host:ciihdfshost(IP:)
Config / Service |access:yes |host:ciiconfservicehost(IP:127.0.0.1) |host2:ciiconfservicehost2(IP:127.0.0.1)
OLDB / Redis |access:yes |host:localhost(IP:::1)
OLDB / MinIO |access:no |host:ciihdfshost(IP:)
Telem / Service |access:no |host:ciiarchivehost(IP:127.0.0.1)
Statistics of Services:
Config / Service |NrOfRequests: 2 |UpTime: 4 minutes
OLDB / Redis |total_connections_received:3 |rejected_connections:0
Telemetry / Service
status This is a feature-centric view, that basically tells you which features you have available. For example, “Blob Values” means that the Distributed File System MinIO (in previous versions: Hadoop) is available for storage of large values and binaries.
$ cii-services status
CII Services Tool (20210706)
Collecting information........
Status Summary:
OK Config from Central Server
OK Config from Local Files
-- Config with Blob Values
OK OLDB Normal Mode
-- OLDB with Blob Values
-- Telemetry
logs Privileged Operation. Run the command as root.
$ su -c "cii-services logs"
Password:
CII Services Tool (20210706)
=== ConfigService Journal ==============
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,979 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=StatusLogger
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,980 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=ContextSelector
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,981 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=Loggers,name=
Sep 07 15:05:28 elthlccd20 config-service[29961]: 2021-09-07 15:05:28,985 main DEBUG Registering MBean org.apache.logging.log4j2:type=764c12b6,component=Appenders,name=console
[...]
=== Logstash Journal ===================
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
Sep 07 15:32:11 elthlccd20 systemd[1]: Started logstash.
Sep 07 15:32:24 elthlccd20 logstash[33887]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Sep 07 15:32:25 elthlccd20 logstash[33887]: [2021-09-07T15:32:25,370][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.3"}
Sep 07 15:32:26 elthlccd20 logstash[33887]: [2021-09-07T15:32:26,183][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[...]
=== TelemService Journal ===============
-- Logs begin at Tue 2021-08-31 18:06:15 UTC, end at Tue 2021-09-07 15:32:55 UTC. --
-- No entries --
12.4.2. Start / Stop¶
Privileged Operations. You will be asked for the root password.
$ cii-services start
CII Services Tool (20210706)
Possible names: logKibana redis elasticsearch logLogstash configService logFilebeat, or a prefix thereof, or 'all'
$ cii-services start all
CII Services Tool (20210706)
About to start logKibana redis elasticsearch logLogstash configService logFilebeat ...
Password:
$ cii-services stop all
CII Services Tool (20210706)
About to stop logKibana redis elasticsearch logLogstash configService logFilebeat ...
Password:
12.5. Example¶
(base) elthlccd20 eltdev:~ 8 > cii-services info
CII Services Tool (20210706)
Collecting information........
Collecting information........................
Installations on this host:
ElasticSearch |active:no |install:/usr/share/elasticsearch/bin/elasticsearch
Redis |active:no |install:/opt/anaconda3/bin/redis-server
MinIO |active:no |install:/usr/local/bin/minio
ConfigService |active:no |install:/elt/ciisrv/bin/config-service |ini:
TelemService |active:no |install:
Filebeat |active:no |install:/usr/share/filebeat/bin/filebeat
Logstash |active:no |install:/usr/share/logstash/bin/logstash
Kibana |active:no |install:/usr/share/kibana/bin/kibana
Jaeger |active:no |install:
AlarmServices |active:no |install:
AlarmPlugin |active:no
AlarmConsumer |active:no
AlarmConverter |active:no
AlarmSupervisor |active:no
ConfigClient |ini:
Addresses of Services:
Config / Local-DB |access:no |file:/home/eltdev/localdb
Config / ElasticSearch |access:no |host:ciielastichost(IP:)
Config / MinIO |access:no |host:ciihdfshost(IP:)
Config / Service |access:no |host:ciiconfservicehost(IP:) |host2:ciiconfservicehost2(IP:)
OLDB / Redis |access:no |host:localhost(IP:::1)
OLDB / MinIO |access:no |host:ciihdfshost(IP:)
Telem / Service |access:no |host:ciiarchivehost(IP:)
Statistics of Services:
Config / Service
OLDB / Redis
Telemetry / Service
How to read this output:
This section reports some info about the tool itself
CII Services Tool (20210706)
Collecting information........
Collecting information........................
=> Tool version and progress dots. => If an error occurs while collecting information, we count the progress dots to spot the problem.
This section reports where applications on your host will find the CII services, and whether the cii-services tool was able to access them (= talk to them). The services may be running on your host (see next paragraph) or on another host.
Addresses of Services:
Config / Local-DB |access:no |file:/localdb
Config / ElasticSearch |access:no |host:ciielastichost(IP:)
Config / MinIO |access:no |host:ciihdfshost(IP:)
Config / Service |access:no |host:ciiconfservicehost(IP:)
OLDB / Redis |access:no |host:localhost(IP:::1)
OLDB / MinIO |access:no |host:ciihdfshost(IP:)
Telem / Service |access:no |host:ciiarchivehost(IP:)
=> The empty parentheses “(IP:)” tell us that hostname resolution is not configured. => This means applications running on your host are not able to locate the CII services. => You need to configure your host, see cii-postinstall
This section reports which CII services are installed on your host, and whether they are running.
Installations on this host:
ElasticSearch |active:no |install:/usr/share/elasticsearch/bin/elasticsearch
Redis |active:no |install:/opt/anaconda3/bin/redis-server
MinIO |active:no |install:/usr/local/bin/minio
ConfigService |active:no |install:/elt/ciisrv/bin/config-service |ini:
TelemService |active:no |install:
Filebeat |active:no |install:/usr/share/filebeat/bin/filebeat
Logstash |active:no |install:/usr/share/logstash/bin/logstash
Kibana |active:no |install:/usr/share/kibana/bin/kibana
Jaeger |active:no |install:
AlarmServices |active:no |install:
AlarmPlugin |active:no
AlarmConsumer |active:no
AlarmConverter |active:no
AlarmSupervisor |active:no
ConfigClient |ini:
=> Several services (not: Telemetry and Alarms) are installed on your host. None of them is running on your host. => If you plan to use CII services running on another host, this is fine. => Otherwise, you need to start the services, see above.
This section reports some statistics from the services.
Statistics of Services:
Config / Service
OLDB / Redis
Telemetry / Service
=> An empty output like this again indicates that the the services are not accessible, but you knew that already.
Preview: After configuring and starting the services, the output will change to something like this:
CII Services Tool (20210706)
Collecting information........
Collecting information........................
Installations on this host:
ElasticSearch |active:yes |install:/usr/share/elasticsearch/bin/elasticsearch
Redis |active:yes |install:/opt/anaconda3/bin/redis-server
MinIO |active:no |install:/usr/local/bin/minio
ConfigService |active:yes |install:/elt/ciisrv/bin/config-service |ini:
TelemService |active:no |install:
Filebeat |active:yes |install:/usr/share/filebeat/bin/filebeat
Logstash |active:yes |install:/usr/share/logstash/bin/logstash
Kibana |active:yes |install:/usr/share/kibana/bin/kibana
Jaeger |active:no |install:
AlarmServices |active:no |install:
AlarmPlugin |active:no
AlarmConsumer |active:no
AlarmConverter |active:no
AlarmSupervisor |active:no
ConfigClient |ini:
Addresses of Services:
Config / Local-DB |access:yes |file:/home/eltdev/localdb
Config / ElasticSearch |access:yes |host:ciielastichost(IP:127.0.0.1)
Config / MinIO |access:no |host:ciihdfshost(IP:)
Config / Service |access:yes |host:ciiconfservicehost(IP:127.0.0.1) |host2:ciiconfservicehost2(IP:127.0.0.1)
OLDB / Redis |access:yes |host:localhost(IP:::1)
OLDB / MinIO |access:no |host:ciihdfshost(IP:)
Telem / Service |access:no |host:ciiarchivehost(IP:127.0.0.1)
Statistics of Services:
Config / Service |NrOfRequests: 2 |UpTime: 4 minutes
OLDB / Redis |total_connections_received:3 |rejected_connections:0
Telemetry / Service