12. Service Management
Document ID: |
|
Revision: |
1.1 |
Last modification: |
February 29, 2024 |
Status: |
Released |
Repository: |
|
File: |
services.rst |
Project: |
ELT CII |
Owner: |
Marcus Schilling |
Revision |
Date |
Changed/ reviewed |
Section(s) |
Modification |
|---|---|---|---|---|
1.0 |
04.10.2021 |
mschilli |
All |
Created. |
1.1 |
29.02.2024 |
mschilli |
All |
CII v4: updated, Public doc |
Confidentiality
This document is classified as Public.
Scope
This document is a user manual for the service management of the ELT Core Integration Infrastructure software.
Audience
This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.
12.1. Overview
This document is a user manual for CII Services management tooling.
12.2. Introduction
This document describes the tools available for the management of CII Services
cii-postinstall - for setting up CII services on a host
cii-services - for starting / stopping / monitoring of the CII Services
12.3. cii-postinstall
You run the CII-PostInstall on a freshly installed DevEnv machine, or after an upgrade of the DevEnv. You choose a role for the host, and the utility will apply the corresponding configuration steps. For most configuration steps, root privileges are needed.
Usage:
$ /elt/ciisrv/postinstall/cii-postinstall <role>
Run the command without arguments, or with the “help” argument, to see its help.
12.3.1. Roles
The role “ownserver” configures the services for local development on a single machine:
$ /elt/ciisrv/postinstall/cii-postinstall role_ownserver
The role “groupserver” does a few more things than the one above. It runs the services with more resources and makes the services/databases accessible from other hosts:
$ /elt/ciisrv/postinstall/cii-postinstall role_groupserver
The role “groupclient” configures the host to use the services provided by a groupserver. For example, assuming host “elthlccd20” has been post-installed with the role “groupserver”, then, to let applications on your host use the CII Services on that groupserver via the groupserver’s non-deterministic network interface (alternatively, you can specify the IP-Address of the server’s network interface), you do:
$ /elt/ciisrv/postinstall/cii-postinstall role_groupclient elthlccd20-cnd
Switching Roles?
It is possible to run the utility again specifying a different role, but your mileage will vary: some role-switches will work, others not.
Redo post-install
It is possible to re-apply/repair/update the already selected role, by doing:
$ /elt/ciisrv/postinstall/cii-postinstall knownrole
Loading 'known role' for this host: role_groupclient elthlccd20-cnd
Update/Installation started, please wait....
12.3.2. Subcommands
A “role” is a collection of configuration steps, and those configuration steps can also be applied one-by-one, by executing them as a “subcommand”. To see the available subcommands, just pass “commands”:
$ /elt/ciisrv/postinstall/cii-postinstall commands
Available subcommands: role_ownserver role_groupserver role_groupclient knownrole logsink oldb rsyslog rsyslog_collect rsyslog_forward serverlocation help
12.4. cii-services
The cii-services utility lets you start/stop/monitor the CII services. For some operations, it requires root-privileges -> unless you are running it with root-privileges, it will show a password prompt when needed.
Usage:
$ cii-services
CII Services Tool (20240220)
Available subcommands: info logs start stop restart enable disable status(deprecated)
12.4.1. Monitoring/Status Check
info install
A deployment-centric view showing which services are running on the current host:
$ cii-services info install
CII Services Tool (20240220)
# install .................. .................
[oldb] config-client-ini |install:
[oldb] cii-oldb-default-redis |active:no |boot:n |install:
[oldb] cii-oldb-calc-daemon |active:no |boot:n |install:
[oldb] cii-oldb-calc-scheduler |active:no |boot:n |install:
[log] rsyslog |active:yes |boot:y |install:/usr/sbin/rsyslogd
[log] systemd-journald |active:yes |boot:y |install:/usr/lib/systemd/systemd-journald
[log] logrotate |active:yes |boot:y |install:/usr/sbin/logrotate
[trace] jaeger-all |active:no |boot:n |install:/usr/local/bin/jaeger-all-in-one
[telem] srv-telemetry |active:no |boot:n |install:
[alarm] cii_ias |active:no |boot:n |install:
[alarm] kafka |active:no |boot:n |install:
[alarm] kafka-zookeeper |active:no |boot:n |install:
[alarm] cii_alarm_mon |active:no |boot:n |install:
[deprec] filebeat |active:no |boot:n |install:
[deprec] logstash |active:no |boot:n |install:
[deprec] kibana |active:no |boot:n |install:
[deprec] elasticsearch |active:no |boot:n |install:
[deprec] minio |active:no |boot:n |install:
info access
A deployment-centric view showing which CII services are being used:
$ cii-services info access
CII Services Tool (20240220)
# discovering ........
# access ...
[oldb] config-redis |access:no |host:localhost |IP:127.0.0.1 |Port:6379
[oldb] redis-server |host: |IP: |Port:0
[oldb] pubsub-server |host: |IP: |Port:0
[oldb] calc-node
[trace] jaeger |access:no |host:localhost |IP:127.0.0.1 |Port:14269
[telem] telem |host:localhost |IP:127.0.0.1 |Port:9115
[alarm] alarm-ias |host:localhost |IP:127.0.0.1
[alarm] alarm-mon |access:n/a |host:localhost |IP:127.0.0.1 |Port:5602
info stats
Details about the usage of the services:
$ cii-services info stats
CII Services Tool (20240220)
# stats ....
config-redis |total_connections_received:81 |rejected_connections:0
redis-server |total_connections_received:82 |rejected_connections:0
jaeger
telem
info function
A feature-centric view, that tells you which features you have available:
$ cii-services info function
CII Services Tool (20240220)
# function ...
Log |functional:yes
OLDB DP |functional:yes
OLDB CE |functional:yes
IntCfg |functional:yes
logs
Shows journal logs of CII Services. Privileged Operation. Run the command as root:
$ su -c "cii-services logs"
Password:
CII Services Tool (20240220)
Service Logs ........
[cii-oldb-default-redis]
-- No entries --
[rsyslog]
Jan 29 20:20:23 eltmal28 rsyslogd[946]: imjournal: journal files changed, reloading... [v8.2306.0-1.fc38 try https://www.rsyslog.com/e/0 ]
Jan 29 20:20:24 eltmal28 rsyslogd[946]: imjournal: journal files changed, reloading... [v8.2306.0-1.fc38 try https://www.rsyslog.com/e/0 ]
Feb 04 00:00:23 eltmal28 systemd[1]: rsyslog.service: Sent signal SIGHUP to main process 946 (rsyslogd) on client request.
12.4.2. Start / Stop
Privileged Operations. You will be asked for the root password:
$ cii-services start
CII Services Tool (20240220)
Possible args: one or more groups: oldb, log, trace, all, or any unit names from the all group
$ cii-services start all
CII Services Tool (20210706)
Password:
$ cii-services stop all
CII Services Tool (20210706)
Password:
12.5. Example
How to read the tool’s output:
$ cii-services info
Section 1 reports some info about the tool itself:
CII Services Tool (20240220)
# install .................. .................
Tool version and progress dots. If an error occurs while collecting information, run the command with “-v” or “-vv”.
Section 2 reports which CII services are installed on your host, whether they are running, and whether they are automatically started at machine boot-up:
# install .................. .................
[oldb] config-client-ini |install:
[oldb] cii-oldb-default-redis |active:no |boot:n |install:/usr/bin/redis-server
[oldb] cii-oldb-calc-daemon |active:no |boot:n |install:/elt/ciisrv/bin/srv-oldb-calculation
[oldb] cii-oldb-calc-scheduler |active:no |boot:n |install:/elt/ciisrv/bin/srv-oldb-scheduler
[log] rsyslog |active:yes |boot:y |install:/usr/sbin/rsyslogd
[log] systemd-journald |active:yes |boot:y |install:/usr/lib/systemd/systemd-journald
[log] logrotate |active:yes |boot:y |install:/usr/sbin/logrotate
[trace] jaeger-all |active:no |boot:n |install:/usr/local/bin/jaeger-all-in-one
[telem] srv-telemetry |active:no |boot:n |install:
[alarm] cii_ias |active:no |boot:n |install:
[alarm] kafka |active:no |boot:n |install:
[alarm] kafka-zookeeper |active:no |boot:n |install:
[alarm] cii_alarm_mon |active:no |boot:n |install:
Several services (not: Telemetry and Alarms) are installed on your host. Only the log services are running on your host. Note the log services are recommended to run always. No other CII services are running on your host. If you plan to use CII services running on another host, this is fine. Otherwise, you need to start the services, see above.
Section 3 reports where applications on your host will find the CII services, and whether the cii-services tool was able to access them (= talk to them). The services may be running on your host (previous section) or on another host:
# access ...
[oldb] config-redis |access:yes |host:ciiconfservicehost |ip: |port:6379
[oldb] redis-server |host: |ip: |port:0
[oldb] pubsub-server |host: |ip: |port:0
[oldb] calc-node
[trace] jaeger |access:no |host:localhost |ip:127.0.0.1 |port:14269
[telem] telem |host:localhost |ip:127.0.0.1 |port:9115
[alarm] alarm-ias |host:localhost |ip:127.0.0.1
[alarm] alarm-mon |access:n/a |host:localhost |ip:127.0.0.1 |port:5602
The empty ip fields tell us that hostname resolution is not configured. This means applications running on your host are not able to locate the CII services. You need to configure your host, see cii-postinstall
Section 4 reports some statistics from the services:
# stats ....
config-redis
redis-server
jaeger
telem
An empty output like this again indicates that the the services are not accessible, but you knew that already.
Sections 5 reports whether the services are working:
# function ....
Log |functional:yes
OLDB DP |functional:no
OLDB CE |functional:no
IntCfg |functional:yes
Explanation:
Log : syslog logging (to /var/log/elt/elt.log)
OLDB DP: datapoint storage
OLDB CE: calculation engine
IntCfg : internal config-storage
Preview: After running post-install and cii-services start, the output will change to something like this:
$ cii-services info
CII Services Tool (20240220)
# install .................. .................
[oldb] config-client-ini |install:
[oldb] cii-oldb-default-redis |active:yes |boot:y |install:/usr/bin/redis-server
[oldb] cii-oldb-calc-daemon |active:yes |boot:y |install:/elt/ciisrv/bin/srv-oldb-calculation
[oldb] cii-oldb-calc-scheduler |active:yes |boot:y |install:/elt/ciisrv/bin/srv-oldb-scheduler
[log] rsyslog |active:yes |boot:y |install:/usr/sbin/rsyslogd
[log] systemd-journald |active:yes |boot:y |install:/usr/lib/systemd/systemd-journald
[log] logrotate |active:yes |boot:y |install:/usr/sbin/logrotate
[trace] jaeger-all |active:no |boot:y |install:/usr/local/bin/jaeger-all-in-one
[telem] srv-telemetry |active:no |boot:n |install:
[alarm] cii_ias |active:no |boot:n |install:
[alarm] kafka |active:no |boot:n |install:
[alarm] kafka-zookeeper |active:no |boot:n |install:
[alarm] cii_alarm_mon |active:no |boot:n |install:
[deprec] filebeat |active:no |boot:n |install:/usr/share/filebeat/bin/filebeat
[deprec] logstash |active:no |boot:n |install:/usr/share/logstash/bin/logstash
[deprec] kibana |active:no |boot:n |install:/usr/share/kibana/bin/kibana
[deprec] elasticsearch |active:no |boot:n |install:/usr/share/elasticsearch/bin/elasticsearch
[deprec] minio |active:no |boot:n |install:/usr/local/bin/minio
# discovering ........
# access ...
[oldb] config-redis |access:yes |host:ciiconfservicehost |ip:127.0.0.1 |port:6379
[oldb] redis-server |access:yes |host:localhost |ip:127.0.0.1 |port:6379
[oldb] pubsub-server |access:yes |host:localhost |ip:127.0.0.1 |port:6379
[oldb] calc-node |access:yes |host:localhost |ip:127.0.0.1 |port:9117
[trace] jaeger |access:no |host:localhost |ip:127.0.0.1 |port:14269
[telem] telem |host:localhost |ip:127.0.0.1 |port:9115
[alarm] alarm-ias |host:localhost |ip:127.0.0.1
[alarm] alarm-mon |access:n/a |host:localhost |ip:127.0.0.1 |port:5602
# stats ....
config-redis |total_connections_received:6 |rejected_connections:0
redis-server |total_connections_received:7 |rejected_connections:0
jaeger
telem
# function ...
Log |functional:yes
OLDB DP |functional:yes
OLDB CE |functional:no
IntCfg |functional:no