13. Internal Config System¶
Revision: |
1.8 |
Status: |
Released |
Repository: |
|
Project: |
ELT CII |
Folder: |
trunk/deliverables/phase7 |
Document ID: |
|
File: |
MAN-ELT_Configuration.docx |
Owner: |
Borut Terpinc |
Last modification: |
March 17, 2022 |
Created: |
March 16, 2019 |
Prepared by |
Reviewed by |
Approved by |
|---|---|---|
Borut Terpinc (CSL SWE) Marko Novak (CSL) |
Miha Vitorovič (CSL) |
Gregor Čuk (CSL) |
Revision |
Date |
Changed/rev iewed |
Section(s) |
Modificatio n |
|---|---|---|---|---|
0.1 |
10.04. 2019 |
bterpinc |
All |
Document creation |
0.2 |
15.04. 2019 |
manovak |
GUI |
GUI part added |
0.9 |
26.4. 2019 |
bterpinc |
All |
Update after internal review, release |
1.0 |
3.6. 2019 |
bterpinc |
6 |
Package support update |
1.1 |
19.7. 2019 |
bterpinc |
3 |
Added sample application section. |
1.2 |
11.9. 2019 |
bterpinc/jp ribosek |
All |
Updated after internal review. Added sample application workflow and search examples. |
1.3 |
25.9. 2019 |
bterpinc |
Appendix C |
Added sample code produced by the generators. |
1.4 |
09.10. 2019 |
bterpinc |
Introductio n 4 |
Added introductio n section Added simple custom metadata sample application |
1.5 |
12.02. 2020 |
bterpinc |
All |
General rewrite of the manual. Sections added and restructure d. |
1.6 |
20.02. 2020 |
kstampar |
All |
Update of cpp file names and added flags when generating python bindings. |
1.7 |
13.07. 2020 |
bterpinc |
Appendix C Appendix G Appendix H 5.4 2.1 |
Removed the MdInt64 duplicate. Added client settings Added service settings Added cache example Additional note on class versioning. |
1.8 |
17.03. 2022 |
mschilli |
Doc Title Overview |
Module is now only for CII-internal usage |
Confidentiality
This document is classified as a confidential document. As such, it or parts thereof must not be made accessible to anyone not listed in the Audience section, neither in electronic nor in any other form.
Scope
This document is manual for the configuration system used by applications in the ELT Core Integration Infrastructure Software.
Audience
This document is aimed at those Cosylab and ESO employees involved with the ELT Core Integration Infrastructure Software project, as well as other Users and Maintainers of the ELT Core Integration Infrastructure Software.
References
Cosylab, ELT CII Online Database, User Manual, CSL-DOC-19-176283, v 1.1
Cosylab, ELT Config, Error Handling transfer document, CSL-DOC-19-172644, Version 1.9
JsonPath, https://github.com/json-path/JsonPath
Elastic search query, https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax
13.1. Overview¶
This document is a manual for the CII-internal configuration system. It is used by CII to manage the metadata of the Online database, the capture configurations of the Telemetry service, and others.
13.2. Introduction¶
The Core Integration Infrastructure (CII) Internal Configuration system provides the mechanisms for distributing configuration data to the CII services software and to allow the collection of configuration data from these systems.
Figure 2‑1 Configuration interactions
Configuration data is stored as JSON documents, and supports three different storage modes:
Cache: Data is stored into client API local memory. This mode is not persistent.
Local DB: Data is stored into local data storage on client computers. This mode is persistent.
Remote DB: Data is stored in the Engineering Archive using the config service. As config service together with Engineering Archive storage provides a complete solution to access and store data to remote storage, it is also referred to as “remote DB”. This mode is persistent.
The configuration is a set of data and metadata. The types and structure of configuration is defined in classes. Metadata classes define the structure of config points and metadata, while configuration classes define the structure of a target configuration (configuration instance). Both configuration and metadata classes support inheritance.
The configuration and metadata classes are based on YAML structures used to generate language-specific classes. The structure of the YAML defines the class names, class member fields, and member types.
Configuration data is composed of metadata instances containing the metadata values and default config values, and target configuration containing config values of the target config points. The configuration instances and metadata instances are JSON data structures. Instances define how values of the actual data-bind to corresponding config and metadata classes.
Configuration instances are instances of the configuration classes which present the actual target configurations. Therefor “target configuration” and “configuration instance” have the same meaning.
Figure 2‑2 Configuration/metadata classes and instances
Figure 2‑2 shows the example of relationship between configuration system (configuration and metadata) classes and instances. The figure shows configuration classes in orange, target configurations in light blue and metadata instances in dark blue colour. The configuration classes MySimpleClass and MySimpleExtendedClass define the member fields (testInt, testIntArray, testString, testDouble) of chosen type.
More detailed description of the system can be found in the sections below.
13.2.1. Classes¶
Classes only present the structure of the configuration and thus cannot contain any values. Classes, in general, have the following properties:
Classes present the structure of the configuration.
Classes don’t contain any values.
Classes definitions are generated into selected programming language class bindings.
Changing the class structure forces recompilation of the client code.
Classes cannot be versioned, but they can be updated and deleted.
Classes are generated from remote storage or a corresponding local YAML file.
Classes cannot be versioned, but they support inheritance. In the standard workflow, classes cannot be deleted or updated. For defining a new class structure, a new derived class must be created. The new derived class must inherit from the root (configuration or metadata) class which is already a part of CII configuration. This root class has no members, so it doesn’t define any class structure. Members must be added to the derived class.
Note on class versioning: As classes present the structure and are a base for the class bindings, the versioning of the classes would present additional complexity to the configuration. Every change of the class version would mean the recompilation of the code. A part of a good design principle is to have fixed structures defined. Versioning of classes would break this concept, as every version could completely change the structure. As the instances are bound to the structure, this would also mean completely changing the instances under the existing TC URI. As the versioning of the classes is not supported, the structure can be extended by using the inheritance. Inheritance maintains the base structure, but adds the ability to add additional attributes to the configuration.
Deleting and updating the classes can only be used in the process of development and testing.
Figure 2‑3 Configuration and metadata classes relationship
Figure 2‑3 shows relationship diagram of config and metadata classes on basic example. The class MySimpleClass derives from the config base class (CiiConfigClass), which is the root class for all the Cii configuration classes. The MySimpleClass contains 3 member fields, testInt, testIntArray and testString. Field types are defined by metadata classes (MdInt32, MdInt32Array, MdString). For the list of the supported types refer to 2.1.2 and Appendix C.
Additionally, MySimpleExtendedClass derives from the MySimpleClass and adds additional field testDouble which is of type MdDouble. All other fields are inherited from the base class (MySimpleClass).
13.2.1.1. Configuration classes¶
Configuration classes are structures which define the type and member fields for generated specific language classes. They have the following properties:
Configuration classes present nodes in the config structure.
Configuration class member fields present the configuration data points – config points.
Config points can be metadata classes or references to another configuration class.
Configuration classes cannot contain complex types or list of complex types, the only exception is a reference to another class.
All configuration classes derive from basic class (CiiConfigClass).
All configuration classes must be stored in the elt.config.classes package.
13.2.1.2. Metadata classes¶
Metadata classes are structures which are used to generate language specific classes that contain metadata values. Metadata classes contain custom metadata fields and a special value field, which holds the actual value of the config point. The value field is added to the metadata classes for simplification and more straightforward usage of Java API. The actual value field is not serialized in metadata instances, but in configuration instances, as the value is not directly part of metadata. Usage of the value field in the metadata classes simplifies typing of metadata and config points, as the types for value, default value and metadata are all contained in one class.
Config points (configuration class member fields) are in most cases metadata classes. The naming convention for this classes follows the rule: Md + chosen name (e.g.: MdNumber). Metadata classes have the following properties:
Metadata classes can only contain a limited set of CiiBaseTypes (CII Basic Data Type Set), which are synced with corresponding language types.
Metadata classes cannot contain complex types.
Metadata member fields cannot contain big arrays. These are limited to the special file field, which only exist in MdArray and all its derivates.
Metadata classes support generic member types, so the type of desired members can be set according to the defined generic type of the class (as Java generics).
All metadata classes derive from basic class (MdBase).
All metadata classes must be stored in the elt.config.classes.meta package.
Set of basic metadata classes is pre-defined. These classes derive directly from MdBase and cannot be deleted or changed as they are part of the configuration system. Basic metadata classes also contain check functions. The check functions are executed when calling the setValue method. The exception is thrown when the value is not properly evaluated by the check function.
MdBase is a root metadata class for all metadata. All other metadata classes must derive from this class. Basic types are defined by metadata classes. The metadata types are shown in Figure 2‑4 and in the table in Appendix C.
Figure 2‑4 Metadata type system
To store array and matrix values, MdArray and MdMatrix derivates must be used.
The data for matrix is represented as a single array, with specified width (member field of the MdMatrix class), so matrix is represented as a continuous array of elements. The width attribute determines at which array element number the array should be cut into row to form a matrix. All matrices are row based; this means that the data is split into rows. When using default values, the width is set to 50.
Binary values are always stored as file references. Like for all the other types of data, the value for binary is set by using the setValue and retrieved by using the getValue method. On setValue method call, the data is written to a file and a reference is stored in the target configuration JSON file. On getValue method call, the file reference is resolved and data from the reference is retrieved. There is no limit specified for the binary data values.
Vector (array) and matrix values are stored as a string up to the limit of 100KB. All values exceeding this limit are stored as file references.
For more complex configurations, custom metadata classes can be defined. The classes can extend existing metadata classes.
The data of the config and metadata classes cannot be manipulated using the config client API. Classes and metadata classes data shall be centrally stored in the remote DB. The config GUI is used to the manipulate the classes on the remote DB. For the development and testing purposes the classes can also be stored on local developer or test machines as YAML definitions.
13.2.2. Instances¶
Instances are actual value holders for the corresponding configuration and metadata classes. Instances are entities of JSON structure stored in local or remote DB and bind data to corresponding class defined fields. Class and instances member fields must be linked by name, otherwise exception CiiConfigTcBindException is thrown.
13.2.2.1. Metadata instances¶
Metadata instances define metadata for the config point. Metadata instances have the following properties:
Metadata instances are identified by name. The names must be unique.
Metadata instances can be created, updated and deleted.
They can be stored to cache, local DB or remote DB. The asterisk “*” authority in metadata instances is not supported. The proper storage location for reading the reference directly form the TC is determined by the TC URI authority.
Config point metadata is configured by metadata instance. Configuration instances only hold references to metadata instances. As metadata instances are not directly part of configuration instances, they are stored as separate entities.
Metadata instances can be versioned.
Set of default (predefined) metadata instances is provided (Appendix C). These default metadata instances cannot be saved or updated.
When saveTargetConfigWithMetadata method is used, metadata instances are automatically saved with target configuration. For other types of metadata operations, API contains methods that support operations just with metadata instances. These methods enable saving, updating and deleting metadata instances.
Figure 2‑5 Metadata instances
Figure 2‑5 shows the relationship between the metadata classes and metadata instances. The metadata class MdInt32 which derives from the MdNumber and root metadata class MdBase defines the structure for the metadata instances. Three metadata instances are created based on the MdInt32 class. The ConfInt32Std is a basic predefined metadata instance and has no special values defined. The custom metadata instances ConfCustomInstance1 and ConfCustomInstance2 have custom values set. The values of the class member fields (defaultValue, minValue, maxValue, checked, etc.) can be set for these instances.
13.2.2.2. Configuration instance¶
Target configuration (configuration instance) contains actual configuration data for the specific configuration class. The configuration instance is identified with URI location. Each URI defines the location from which the configuration instance is obtained. The authority part of URI defines the storage location. The following authorities are supported:
Cache: ‘cii.config://cache/exampleconfig/c1’
Local DB: ‘cii.config://local/exampleconfig/c1’
Remote DB: ‘cii.config://remote/exampleconfig/c1’
All locations: “cii.config://*/exampleconfig/c1”
Note: URI locations that start with number are not allowed. Examples of not valid URIs: cii.config:///exampleconfig/1abc or cii.config:///1exampleconfig/abc
The asterisk “*” authority can be used for retrieving configuration instance. When saving the instances, it is better to provide a specific location. When “*” authority is used, target configuration is obtained in the following order: cache, local DB, remote DB. In case there is no data stored under specified URI in the specific storage location, the location is skipped.
If there is no explicit version specified for retrieving the target configuration, the last existing version stored in the specific storage location will be retrieved. These versions can differ between the storage locations. As the config service can be used also without the presence of the remote DB, an additional setting was added to the client, to control the exception case.
The setting exceptionWhenRemoteNotAccessible can be set. Appendix G contains details on how to apply this setting to the client.
The following rules apply when retrieving the Target configuration with the “*” authority is used:
exceptionWhenRemoteNotAccessible=true: If no remote DB is present – throw an exception.
exceptionWhenRemoteNotAccessible=false: If no remote is present, check local and cache.
When saving or updating the Target configuration using the “*” authority, the versions between the different storage locations can become out of sync. In general, the following rules apply:
saveTC(“*”): save always creates a new version.
updateTC(“*”): always update the last version.
updateTC(specificVersion) – updates specific version. It is always expected for a specific version on location (cache, local, remote) to exist. If it doesn’t exist an exception is thrown.
As the configuration can be used without the remote DB, the following specifics apply:
exceptionWhenRemoteNotAccessible=true:
saveTC(“*”): remote DB is version maintainer. Max version off all the locations is retrieved, using the rule (cache|local|remote) + 1. This version is used to make new cache|remote|local Target configuration. This way versions between the locations are always in sync, but version holes can exist.
updateTC(“*”): remote DB is version maintainer (the last version is retrieved from the remote DB). It is expected that the same version exists in local and cache and that this is the last version (important). If this is not the case, an exception is thrown.
updateTC(specificVersion): All versions are expected to exist. Updates for the specific version to all locations are made (cache|local|remote). If there is no specific version present in any of the DB, an exception is thrown.
- exceptionWhenRemoteNotAccessible=false:If remote DB is present, the behavior is the same as above. If not, the following actions are applied after the timeout:
saveTC(“*”): local DB is version maintainer. We get max version from (cache|local) + 1. We use this version to make a new cache|local TC.
updateTC(“*”): local DB is version maintainer (the last version is retrieved from the local DB). It is expected that the same version exists in cache and local and that this is the last version. If this is not the case, an exception is thrown.
updateTC(specificVersion): All versions are expected to exist. Updates for the specific version to these locations are made (cahce|local). If there is no specific version present in any of the DB, an exception is thrown.
To use the configuration in Local DB mode, YAML definition files for classes and values must be created. The config-tool shall be used to generate language-specific classes and to deploy configuration data into the local DB.
When working with Remote DB, config-gui shall be used to manipulate the data. The remote DB also contains the config and metadata class definitions.
A cache can be used to store the data into memory. From the operations point of view, the cache acts the same as the local DB or remote DB and it is transparent to the client. The major difference to the other storage location is that cache is isolated per process and will not provide any permanent storage. The major benefit of using the cache is the fast storage and retrieval of configuration data. The cache is cleared when the process terminates.
For the configuration instances that use the config point fields of MdArray and MdBinary class, the data is not stored as a part of the JSON, but it is stored to a large storage location. The data to the large storage location is written as raw data in the big-endian order.
13.3. Prerequisites¶
In order to use the configuration system, the following packages must be built and installed:
MAL, MAL ZPB (elt-mal)
Client APIs (client-APIs)
Config service (srv-config)
Config tools (config-tools)
Config GUI (config-gui)
Config service depends on the ElasticSearch. ElasticSearch server must be accessible to the config service. The default configuration assumes that ElasticSearch is available on http://ciielastichost:9200. Appendix H contains details on how to configure the config service.
For remote large data storage, the Hadoop HDFS system is used. HDFS system must be accessible in order to store large data files. The default configuration assumes that HDFS is accessible on http://ciihdfshost:9870/.
To use the remote database, the remote config service has to be installed and started.
See Error Handling and Configuration Transfer Document [2] for the instructions on how to install and configure the configuration system and its components. The document also contains a description of how to set the ElasticSearch and Hadoop hosts.
13.3.1. Local DB initialization¶
Local DB stores it’s data into the $INTROOT/localdb folder. To use the default metadata instances used in the examples, the initial predefined metadata instances must be deployed:
config-tool init
13.3.2. Includes/Imports¶
For basic usage of the Config API the user needs the CiiConfigClient class and all generated binding configuration classes that will be used. If any custom metadata classes are used, the imports of these classes are also needed. Next, the language specific URI class is needed. Java also needs import of all the checked exceptions used. Following sections present the code imports and includes:
13.3.2.1. Java imports¶
import elt.config.client.CiiConfigClient;
import elt.config.exceptions.CiiConfigInitializationError;
import elt.config.exceptions.CiiConfigNoTcException;
import elt.config.exceptions.CiiConfigWriteDisabledException;
import elt.config.exceptions.CiiConfigSaveException;
import elt.config.exceptions.CiiConfigWriteDisabledException;
import elt.error.CiiInvalidTypeException;
import elt.error.CiiInvalidURIException;
import java.net.URI;
import java.net.URISyntaxException;
13.3.2.2. CPP Includes¶
#include <ciiDataProviderFactory.hpp>
#include <ciiLocalDataProvider.hpp>
#include <ciiCacheDataProvider.hpp>
#include <ciiRemoteDataProvider.hpp>
#include <ciiConfigApi.hpp>
13.3.2.3. Python imports¶
import os
import elt.config
13.4. Basic usage¶
This section presents basic usage examples. The examples in this section can be found in the config-examples project (M*ySimpleClient.java, MySimpleClient.py, mySimpleClient.cpp*). Refer to the config-examples project for details about files, folder structure and the WAF wscript files configuration. The examples project is build using the standard WAF build procedure:
waf distclean configure build install
To run the examples, use the provided executables (run-java, cpp-config-sample-app, python MySimpleClient.py).
The examples show the usage of Config tool (6.1) and Config GUI (7). The Config tool only deploys/undeploys data from YAML files to local DB. Config tool is intended to be used in the development process, where developer can quickly change definitions in YAML files on local machine, while Config GUI will work with remote with all the defined TCs. Config tool cannot be used to deploy data to remote DB. This can however be done with usage of Client API (the data can be read from local DB and stored to remote DB). For this reason, the examples show usage of both Config tools and Config GUI in for the same example.
Note: In case examples are run directly from config-examples project, make sure that all the needed local DB configuration instance files are deployed. The deploy commands are listed in README.md file of the project.
13.4.1. Example description¶
---
configuration:
configClassesConfiguration:
classes:
- name: MySimpleClass
parent: CiiConfigClass
__comment__: Personal config test class
members:
- name: testIntValue
type: MdInt32
- name: testDoubleValue
type: MdDouble
- name: testStringValue
type: MdString
The types names and language mappings for all types can be found in Appendix A. The syntax of YAML schema is explained in Appendix E. Not all words are allowed for the field names of config or metadata classes. The table in Appendix D contains all the reserved words.
In the example we create a sample application. The application is placed in the root of config-examples project. To prepare the directory structure we create the following subdirectories: java, cpp, cpp-app, cpp-pybindings, python and YAML:
mkdir java
mkdir cpp
mkdir cpp-app
mkdir cpp-pybindings
mkdir python
mkdir yaml
13.4.2. Creating a simple configuration using YAML definitions¶
In the simple class definition, we defined the structure of the configuration class. We store this definition in YAML file named mySimpleClass.cdt.yaml in the yaml directory. Based on this YAML definition we must first generate the classes used by the application:
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o java/
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o cpp/ -l cpp
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o cpp-pybindings/ -l python -ws
The sample code generated by the config-tool can be found in Appendix I.
Once the class is prepared, we must also define target configuration YAML. The target configuration will use the generated class MySimpleClass and default metadata instances. List of default metadata instances with internal types is listed in Appendix C. The following TC YAML is prepared:
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MySimpleClass
fields:
testStringValue:
"@type": MdString
metadataInstance: ConfStringStd
metadataInstanceVersion: 1
value: 'My string defined value'
testIntValue:
"@type": MdInt32
metadataInstance: ConfInt32Std
metadataInstanceVersion: 1
value: 155
testDoubleValue:
"@type": MdDouble
metadataInstance: ConfDoubleStd
metadataInstanceVersion: 1
value: 155.33
We store the target configuration under the testTC.cdi.yaml file in yaml directory.
Note: Types defined in target configuration must be the same as types defined in the classes in order for the configuration to work properly. The depending on the language the JSON (de)serializers will throw exception with indication that (de)serialization could not be executed. For the types with embedded arrays, the types used in target configuration must have the vector types defined. Please refer to Appendix C for details.
To have the default metadata instances ready, we must first initialize the local DB:
config-tool init
Then we deploy the target configuration to local DB under cii.config://mytest/mysampletc:
config-tool deploy -i yaml/testTC.cdi.yaml -a cii.config://mytest/mysampletc
After the target config is deployed to local DB, we can start using the clients. For successful code builds, proper build dependencies must be set for each of the clients. Please check the config-examples project for details about the WAF dependencies.
Results of deploying testTC.cdi.yaml to local DB (JSON file) can be found in Listing 7‑5.
13.4.2.1. Java client¶
Listing 4‑1 Simple client Java sample code
public static void main(String[] args) throws Exception {
URI myUri = new URI("cii.config://*/mytest/mysampletc");
//Retrieve config from local or remote DB
try(CiiConfigClient client=CiiConfigClient.getInstance()){
MySimpleClass data = client.retrieveConfig(myUri).getData(MySimpleClass.class);
System.out.println("My double value:" + data.getTestDoubleValue().getValue());
System.out.println("My int value:" + data.getTestIntValue().getValue());
System.out.println("My string value:" + data.getTestStringValue().getValue());
//Store changed data into new TC location -- local DB
client.setWriteEnabled(true);
URI localUri = new URI("cii.config://local/mytest/mysampletc2");
data.getTestDoubleValue().setValue(121442242.224242424);
CiiConfigClient.getInstance().saveTargetConfig(localUri, data);
//Read changed data
MySimpleClass changedData = client. retrieveConfig(localUri).getData(MySimpleClass.class);
System.out.println("My double value:" + changedData.getTestDoubleValue().getValue());
System.out.println("My int value:" + changedData.getTestIntValue().getValue());
System.out.println("My string value:" + changedData.getTestStringValue().getValue());
}
}
}
13.4.2.2. CPP client¶
Listing 4‑2 contains a sample CPP client code. The code file is named mySimpleClient.cpp and stored in the cpp-app folder. The code follows the same flow as the Java code.
Listing 4‑2 Simple client CPP sample code
int main(int ac, char* av[]) {
::elt::config::classes::MySimpleClass mySimpleClassDataCh;
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MySimpleClass>(
elt::mal::Uri("cii.config://*/mytest/mysampletc"));
std::cout << "My int value:" << retrievedData->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedData->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedData->get_testStringValue() << std::endl;
//Store changed data into new TC location -- local DB
::elt::config::CiiConfigClient::setWriteEnabled(true);
mySimpleClassDataCh.set_testStringValue("Changed string value");
mySimpleClassDataCh.set_testIntValue(retrievedData->get_testIntValue());
mySimpleClassDataCh.set_testDoubleValue(retrievedData->get_testDoubleValue());
int version = elt::config::CiiConfigClient::saveTargetConfig(
elt::mal::Uri("cii.config://local/mytest/mysampletc2"), mySimpleClassDataCh);
//Read changed data
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedData2 =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MySimpleClass>(
elt::mal::Uri("cii.config://local/mytest/mysampletc2"));
std::cout << "My int value:" << retrievedData2->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedData2->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedData2->get_testStringValue() << std::endl;
}
13.4.2.3. Python client¶
Listing 4‑3 contains a sample Python code. The code file is named mySimpleClient.py and stored in the python folder. The code follows the same flow as the CPP code.
Listing 4‑3 Simple client Python sample code
import os
import elt.config
from MySimpleClassPyB import MySimpleClass
uri = elt.config.Uri("cii.config://*/mytest/mysampletc")
uri2 = elt.config.Uri("cii.config://local/mytest/mysampletc2")
client = elt.config.CiiConfigClient
#Retrieve config configuration
retrieved = client.retrieve_config(uri, -1)
print("My string value:" + retrieved.get_testStringValue())
print("My integer value: {}".format(retrieved.get_testIntValue()))
print("My double value: {}".format(retrieved.get_testDoubleValue()))
#Store changed data into new TC location -- local DB
client.set_write_enabled(True)
retrievedNew = MySimpleClass.get_new_node_instance()
retrievedNew.set_testStringValue("New test value")
retrievedNew.set_testIntValue(retrieved.get_testIntValue())
retrievedNew.set_testDoubleValue(retrieved.get_testDoubleValue())
version = client.save_target_config(uri2, retrievedNew)
retrieved2 = client.retrieve_config(uri2, -1)
print("My string value:" + retrieved2.get_testStringValue())
print("My integer value: {}".format(retrieved2.get_testIntValue()))
print("My double value: {}".format(retrieved2.get_testDoubleValue()))
13.4.3. Creating a simple configuration using GUI¶
In this section GUI is used to define classes and target configuration on remote DB.
We first start the configuration GUI:
config-gui
Make sure that GUI has write tick-box enabled (Figure 4‑1):
Figure 4‑1 Write enabled tick box
13.4.3.1. Creating a simple class in GUI¶
Click on the Configuration class tab, and click Add button to add a configuration class (Figure 4‑2). Select the CiiConfigClass to be the parent class, and name the class MySimpleClass.
Figure 4‑2 Adding configuration class in GUI
After the class is created, we add attributes to the class. To add attributes (members), we click on the Add attribute button and fill in the details of every class attribute (Figure 4‑3). We use the same names and types as in the YAML class definition in section 4.1.
Figure 4‑3 Adding class members in GUI
After we added all the attributes (testIntValue, testDoubleValue, testStringValue), definition changes must be confirmed and propagated to remote DB by clicking on the Commit button.
13.4.3.2. Creating a target configuration for simple class in GUI¶
To create a TC we click on the Target Config tab and then the Add button. We choose the proper Configuration class and define a URI suffix that corresponds to the URI used when deploying to the local DB, for example mytest/mysampletc. (Figure 4‑4)
Figure 4‑4 Creating target configuration
After the target configuration is created, we must refresh the tree by clicking the “Refresh button”. Next, we bind the metadata instances used by the TC to the member fields. To select the metadata instance, we click on the cells (Figure 4‑6) under Meta Data Instance column. For every class member field name, we choose metadata instance (Figure 4‑5). Predefined default metadata instances are used (ConfInt32Std, ConfDoubleStd, ConfStringStd).
Figure 4‑5 Target configuration definition in GUI
Figure 4‑6 Target configuration definition in GUI
After metadata instances are set, we set the values for every member (cells under Value column), and confirm the changes by clicking on the Commit button (Figure 4‑6).
13.4.3.3. Generating classes from the classes defined in GUI¶
To generate classes defined in GUI (stored in remote DB), the config tool is called without an input argument:
config-tool generate -o java/
config-tool generate -o cpp/ -l cpp
config-tool generate -o cpp-pybindings/ -l python -ws
To read the data with the client, we use the same code as shown in the previous example (Listing 4‑1, Listing 4‑2, Listing 4‑3). To read the data from the remote DB, the “remote” authority must be used. In case “*” authority is used, all data will first be read from the local DB. Only if the data is not present in local DB, the remote DB will be used.
13.4.4. CRUD operations¶
The following section presents the code for creating, reading, updating and deleting metadata and configuration instances. The examples for all languages follow the same flow, and can be easily understood from the code samples. The examples first show the operations on the metadata instance, followed by operation on target configuration. The TC is saved under the cii.config://remote/mytest/my_test_instance URI. Config API also supports the listing of config points stored under certain path. To get all the children, the getChildren method can be used. Refer to Appendix A for details about the API.
Note: When setting only one attribute of a configuration programmatically, the metadata for the other attributes are not set automatically. It is expected that all the fields of the configuration class are set by user.
The same MySimpleClass as it was used in section 4.2 is also used in the following examples.
13.4.4.1. Java CRUD¶
Listing 4‑4 Java metadata CRUD example
//Metadata CRUD example
final String mdiName = "MyCustomMetaInstanceName";
MyCustomMetaClass<Integer> customMetaClass = new MyCustomMetaClass<>(mdiName, "This is my custom metadata instance name",
123, false, "m/s", 9);
URI metadataLocal = new URI("cii.config://local");
//Save metadata under local DB with name MyCustomMetaInstanceName
int version2 = client.saveMetadata(metadataLocal, customMetaClass);
//Retrieve and update the same version
MyCustomMetaClass<Integer> customMetaClassChange = client.retrieveMetadata(metadataLocal, mdiName, -1, MyCustomMetaClass.class);
System.out.println("Metadata int:" + customMetaClassChange.getMetaOfInt());
customMetaClassChange.setMetaOfInt(15);
client.updateMetadata(metadataLocal, mdiName, version2, customMetaClassChange);
//delete the metadata
System.out.println("Deleting metadata: " + mdiName + ", version: " + version2);
client.deleteMetadata(metadataLocal, mdiName, version2);
Listing 4‑5 Java target config CRUD example
//TC CRUD example
//Create another instance of MySimple class and store it to remote
MySimpleClass newSimpleInstance = new MySimpleClass();
newSimpleInstance.setTestDoubleValueValue(12.0d);
newSimpleInstance.setTestIntValueValue(100);
newSimpleInstance.setTestStringValueValue("My test string");
URI newInstanceUri = new URI("cii.config://remote/mytest/my_test_instance");
int version1 = client.saveTargetConfig(newInstanceUri, newSimpleInstance);
//Retrieve and update the same version
MySimpleClass newSimpleInstanceChange = client.retrieveConfig(newInstanceUri).getData(MySimpleClass.class);
newSimpleInstanceChange.setTestStringValueValue("My test string changed");
client.updateConfig(newInstanceUri, version1, newSimpleInstanceChange);
System.out.println("My string value:" + newSimpleInstanceChange.getTestStringValue().getValue());
//Delete TC
client.deleteConfig(newInstanceUri, -1);
13.4.4.2. CPP CRUD¶
Listing 4‑6 CPP metadata CRUD example
//Metadata CRUD example
std::string mdiName("MyCustomMetaInstanceName");
std::shared_ptr<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>> customMetaClass(
elt::config::CiiDataPointMetadataFactory::getNewMetadataInstance<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>>(mdiName));
customMetaClass->set_metaOfBool(false);
customMetaClass->set_metaOfString("m/s");
customMetaClass->set_metaOfInt(9);
//Save metadata under local DB with name MyCustomMetaInstanceName
int version2 = elt::config::CiiConfigClient::saveMetadata("local", *customMetaClass);
//Retrieve and update the same version
std::shared_ptr<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>> customMetaClassChange =
std::dynamic_pointer_cast<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>>(elt::config::CiiConfigClient::retrieveMetadata("local", mdiName));
std::cout << "Metadata int:" << customMetaClassChange->get_metaOfInt() << std::endl;
customMetaClassChange->set_metaOfInt(15);
elt::config::CiiConfigClient::updateMetadata("local", version2, *customMetaClassChange);
//delete the metadata
std::cout << "Deleting metadata: " << mdiName << ", version: " << version2 << std::endl;
elt::config::CiiConfigClient::deleteMetadata("local", mdiName, version2);
Listing 4‑7 CPP target config CRUD example
//TC CRUD example
//Create another instance of MySimple class and store it to remote
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstance =
CiiNodeFactory::getNewNodeInstance<::elt::config::classes::MySimpleClass>();
newSimpleInstance->set_testStringValue("My test string");
newSimpleInstance->set_testIntValue(100);
newSimpleInstance->set_testDoubleValue(12.0);
elt::mal::Uri newInstanceUri("cii.config://remote/mytest/my_test_instance");
int version1 = elt::config::CiiConfigClient::saveTargetConfig(newInstanceUri, *newSimpleInstance);
//Retrieve and update the same version
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstanceChange =
elt::config::CiiConfigClient::getConfigData<::elt::config::classes::MySimpleClass>(newInstanceUri);
newSimpleInstanceChange->set_testStringValue("My test string changed");
elt::config::CiiConfigClient::updateConfig(newInstanceUri, version1, *newSimpleInstanceChange);
std::cout << "My string value:" << newSimpleInstanceChange->get_testStringValue() << std::endl;
//Delete TC
elt::config::CiiConfigClient::deleteConfig(newInstanceUri, -1);
13.4.4.3. Python CRUD¶
Listing 4‑8 Python metadata CRUD example
#Metadata CRUD example
mdiName = "MyCustomMetaInstanceName"
customMetaClass = MyCustomMetaClassINT32.get_new_metadata_instance(mdiName)
customMetaClass.set_metaOfBool(False);
customMetaClass.set_metaOfString("m/s");
customMetaClass.set_metaOfInt(9);
#Save metadata under local DB with name MyCustomMetaInstanceName
version2 = client.save_metadata("local", customMetaClass);
#Retrieve and update the same version
customMetaClassChange = client.retrieve_metadata("local", mdiName)
customMetaClassChange = MyCustomMetaClassINT32.cast(customMetaClassChange)
#Cannot print as wrong type is returned.
print("Metadata int: %i" % customMetaClassChange.get_metaOfInt())
customMetaClassChange.set_metaOfInt(15)
print("Metadata int changed: %i" % customMetaClassChange.get_metaOfInt())
client.update_metadata("local", version2, customMetaClassChange);
#delete the metadata
print("Deleting metadata: %s, version: %i" % (mdiName, version2))
client.delete_metadata("local", mdiName, version2);
Listing 4‑9 Python target config CRUD example
#TC CRUD example
#Create another instance of MySimple class and store it to remote
newSimpleInstance = MySimpleClass.get_new_node_instance()
newSimpleInstance.set_testStringValue("My test string")
newSimpleInstance.set_testIntValue(100)
newSimpleInstance.set_testDoubleValue(12.0)
newInstanceUri = elt.config.Uri("cii.config://remote/mytest/my_test_instance")
version1 = client.save_target_config(newInstanceUri, newSimpleInstance)
#Retrieve and update the same version
newSimpleInstanceChange = client.retrieve_config(newInstanceUri)
newSimpleInstanceChange.set_testStringValue("My test string changed");
client.update_config(newInstanceUri, version1, newSimpleInstanceChange);
print("My integer value: {}".format(newSimpleInstanceChange.get_testStringValue()))
#Delete TC
client.delete_config(newInstanceUri, -1);
13.4.5. Using the cache operations¶
The following examples present the usage of cache operations. From the operations point of view, the cache acts the same as the local DB or remote DB and it is transparent to the client. The cache is not permanent storage, as it lives with the process. The cache can be used for inter-process fast storing/retrieval of Target Configurations.
The example first copies Target Configuration from the remote DB (cii.config://remote/mytest/my_test_instance), then it stores it to the cache. Then the values are changed and stored back to same location in cache (cii.config://cache/mytest/my_test_instance). Finally, these values are stored back to the original remote DB location.
In the example, the same path is used for remote DB and cache. If the same path is used, the Target Configuration can be retrieved using the “*” authority. It will first check the cache, then the local and remote DB. The same Configuration class data can be stored to any URI, as there is no limitation for cache to be on the same path as remote and local DB.
13.4.5.1. Java cache example¶
Listing 4‑10 Java cache example
//The cache workflow -- data in cache only lives as long as the process is alive
//Retrieve the TC from the remote, and store it to cache.
URI newInstanceUri = new
URI("cii.config://remote/mytest/my_test_instance");
URI cacheInstance = new URI("cii.config://cache/mytest/my_test_instance");
client.saveTargetConfig(cacheInstance,
client.retrieveConfig(newInstanceUri).getData(MySimpleClass.class));
MySimpleClass dataFromCache =
client.retrieveConfig(cacheInstance).getData(MySimpleClass.class);
System.out.println("Original cache data: " +
dataFromCache.getTestIntValueValue() + ", " +
dataFromCache.getTestStringValueValue());
//manipulate cache data
dataFromCache.setTestIntValueValue(15);
dataFromCache.setTestStringValueValue("My string value");
System.out.println("Changed cache data: " +
dataFromCache.getTestIntValueValue() + ", " +
dataFromCache.getTestStringValueValue());
//store data into cache
client.saveTargetConfig(cacheInstance, dataFromCache);
//retrieve data form cache and store it to remote.
MySimpleClass dataFromCacheChanged =
client.retrieveConfig(cacheInstance).getData(MySimpleClass.class);
System.out.println("Retrieved changed cache data: " +
dataFromCacheChanged.getTestIntValueValue() + ", " +
dataFromCacheChanged.getTestStringValueValue());
//save the changed data to remote
client.saveTargetConfig(newInstanceUri, dataFromCacheChanged);
//Delete TC
client.deleteConfig(newInstanceUri, -1);
13.4.5.2. CPP Cache¶
Listing 4‑11 CPP cache example
//The cache workflow -- data in cache only lives as long as the process is alive
//Retrieve the TC from the remote, and store it to cache.
elt::mal::Uri newInstanceUri
("cii.config://remote/mytest/my_test_instance");
elt::mal::Uri cacheInstance("cii.config://cache/mytest/my_test_instance");
//as there are no metadata instances in cache, they must be saved
elt::config::CiiConfigClient::SaveTargetConfig(cacheInstance,
*elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(newInstanceUri));
std::shared_ptr<::elt::config::classes::MySimpleClass> dataFromCache =
elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(cacheInstance);
std::cout << "Original cache data: " << dataFromCache->get_testIntValue()
<< ", " << dataFromCache->get_testStringValue() << std::endl;
//manipulate cache data
dataFromCache->set_testIntValue(15);
dataFromCache->set_testStringValue("My string value");
//store data into cache
elt::config::CiiConfigClient::SaveTargetConfig(cacheInstance, *dataFromCache);
//retrieve data form cache and store it to remote.
std::shared_ptr<::elt::config::classes::MySimpleClass> dataFromCacheChanged =
elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(cacheInstance);
std::cout << "Changed cache data: " << dataFromCacheChanged->get_testIntValue() << ", " << dataFromCacheChanged->get_testStringValue() << std::endl;
//save the changed data to remote
elt::config::CiiConfigClient::SaveTargetConfig(newInstanceUri, *dataFromCacheChanged);
//Delete TC
elt::config::CiiConfigClient::DeleteConfig(newInstanceUri, -1);
13.4.5.3. Python cache example¶
Listing 4‑12 Python cache example
#The cache workflow -- data in cache only lives as long as the process is alive
#Retrieve the TC from the remote, and store it to cache.
newInstanceUri = elt.config.Uri("cii.config://remote/mytest/my_test_instance")
cacheInstance = elt.config.Uri("cii.config://cache/mytest/my_test_instance")
client.save_target_config(cacheInstance, client.retrieve_config(newInstanceUri))
dataFromCache = client.retrieve_config(cacheInstance)
print("Original cache data: %i, %s" % (dataFromCache.get_testIntValue(), dataFromCache.get_testStringValue()))
#manipulate cache data
dataFromCache.set_testStringValue("My string value")
dataFromCache.set_testIntValue(15)
#store data into cache
client.save_target_config(cacheInstance, dataFromCache)
#retrieve data form cache and store it to remote.
dataFromCacheChanged = client.retrieve_config(cacheInstance)
print("Changed cache data: %i, %s" % (dataFromCacheChanged.get_testIntValue(), dataFromCacheChanged.get_testStringValue()))
#save the changed data to remote
client.save_target_config(newInstanceUri, dataFromCacheChanged)
#Delete TC
client.delete_config(newInstanceUri, version1)
13.5. Advanced usage¶
This section presents advanced usage examples. The examples in this section are suffixed with advanced in the config-examples project (M*yAdvancedClient.java, MyAdvancedClient.py, myAdvancedClient.cpp*). The examples project is build using the standard WAF build procedure:
waf distclean configure build install
To run the examples, use the provided executables (run-java-advanced, cpp-config-sample-app-advanced, python MyAdvancedClient.py).
Note: In a case when examples are run directly from config-examples project, make sure that all the needed local DB configuration instance files are deployed. The deploy commands are listed in Readme.md file of the project.
13.5.1. Custom metadata class¶
In this example we create a custom metadata class, which will be used as a member field in the configuration class. Listing 5‑1 shows the YAML code of the metadata class definition. We name the custom metadata class MyCustomMetaClass and save the metadata class definition as myCustomMetadata.cdt.yaml in the folder named yaml. The class derives from the MdNumber. This class has generic type set to T. This means that the actual type of the configuration class member field can be set at the class configuration class definition.
Listing 5‑1 MyCustomMetaClass definition
---
configuration:
metadataClassesConfiguration:
classes:
- name: MyCustomMetaClass
parent: MdNumber
__comment__: Custom metadata class
genericType: T
members:
- name: metaOfBool
type: BOOLEAN
- name: metaOfString
type: STRING
- name: metaOfInt
type: INT32
The class has 3 metadata fields defined: metaOfBool (type Boolean), metaOfString (type string), metaOfInt (type int32). All other metadata fields that can be used by the metadata instances (minValue, maxValue, checked, defaultValue, …) are already defined by the MdNumber and its parent MdBase.
Next, we create a configuration class, which will use this custom metadata class. We name the class MyClassWithCustomMeta and save the configuration class definition as myClassWithCustomMeta.cdt.yaml in the folder named yaml. The class derives from the parent configuration class CiiConfigClass and has only one Config point defined: intPointWithCustomMeta. As the MyCustomMetaClass definition has the type defined as T, we need to set the generic type of the member field to proper type. We set the intPointWithCustomMeta to be INT32. Listing 5‑2 shows the YAML definition of the class.
Listing 5‑2 MyClassWithCustomMeta definition
---
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithCustomMeta
parent: CiiConfigClass
__comment__: Class with custom metadata
members:
- name: intPointWithCustomMeta
type: MyCustomMetaClass
genericType: INT32
After the configuration classes are defined, we must create YAML files which will define the metadata instance and TC values. Listing 5‑3 shows the YAML definition which defines the values of the custom metadata instance member fields for the MyCustomMetaClass. We name the definition myCustomMDI.mdi.yaml and save it in the yaml folder. The definition sets the values and sets the type. As metadata default value can be of any number type, we must also define the type with @genType directive in the metadata instances definition file.
Listing 5‑3 Definition of custom metadata instance (myCustomMDI)
metadata:
instances:
- "@type": MyCustomMetaClass
"@name": myCustomMDI
"@genType": INT32
comment: "Integer MDI with values"
metaOfBool: "true"
metaOfString: "The string metadata example"
metaOfInt: "12"
Finally, we must define the TC values file. We name this file simpleCustomMetaTC.cdi.yaml and save it in the yaml folder. The file contains the values for linking the metadata instance myCustomMDI to the config point intPointWithCustomMeta. The file also sets the value of the point to 13. Listing 5‑4 shows the file definition.
Listing 5‑4 Definition of target config with custom meta data (simpleCustomMetaTC)
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithCustomMeta
fields:
intPointWithCustomMeta:
"@type": MyCustomMetaClass
"@genType": INT32
metadataInstance: myCustomMDI
metadataInstanceVersion: 1
value: 13
Once all the classes are defined, we must generate the metadata binding classes. We generate the classes for all three programming languages with the config tool:
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o java/
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o cpp/ -l cpp
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o cpp-pybindings/ -l python
Next, we generate the configuration classes with our custom metadata:
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o java/
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o cpp/ -l cpp
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o cpp-pybindings/ -l python -ws
After the language bindings for the configuration and metadata classes are generated, the produced code files can be built. Please note that config-tool generator will not generate all the WAF wscript files. For the proper build, the wscript files with dependencies have to be added to the java, cpp, and cpp-pybindings directory. The python bindings will assume, that CPP code and python bindings are located in cpp and cpp-pybindings directory respectively. The config tool provides option “-ws” to generate wscript file for generated python binding modules.
To retrieve the data from the local DB, the YAML value files have to be deployed. The following example shows the CLI commands to deploy the metadata instance values and the configuration instance values. The target configuration is deployed under the cii.config://mytest/myCustomMetaTC:
config-tool deploy -i yaml/myCustomMDI.mdi.yaml -t metadata
config-tool deploy -i yaml/simpleCustomMetaTC.cdi.yaml -a cii.config://mytest/my_custom_meta_tc
Listing 5‑5 Java custom metadata class code example
public class mySimpleClient{
public static void main(String[] args) throws Exception {
URI myUri = new URI("cii.config://local/mytest/my_custom_meta_tc");
try(CiiConfigClient client=CiiConfigClient.getInstance()){
MyClassWithCustomMeta data = client.retrieveConfig(myUri).getData(MyClassWithCustomMeta.class);
System.out.println("The value:" + data.getIntPointWithCustomMeta().getValue());
System.out.println("Metadata one:" + data.getIntPointWithCustomMeta().getMetaOfBool());
System.out.println("Metadata two:" + data.getIntPointWithCustomMeta().getMetaOfString());
System.out.println("Metadata three:" + data.getIntPointWithCustomMeta().getMetaOfInt());
}
}
}
Listing 5‑6 CPP custom metadata class code example
int main(int ac, char* av[]) {
::elt::config::classes::MyClassWithCustomMeta MyClassWithCustomMetaCh;
std::shared_ptr<::elt::config::classes::MyClassWithCustomMeta> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MyClassWithCustomMeta>(
elt::mal::Uri("cii.config://local/mytest/my_custom_meta_tc"));
std::cout << "The value:" << retrievedData->get_intPointWithCustomMeta() << std::endl;
std::cout << "Metadata one:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfBool() << std::endl;
std::cout << "Metadata two:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfString() << std::endl;
std::cout << "Metadata three:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfInt() << std::endl;
}
Listing 5‑7 Python custom metadata class code example
uriCm = elt.config.Uri("cii.config://*/mytest/my_custom_meta_tc")
#Retrieve config configuration
retrievedCm = client.retrieve_config(uriCm, -1)
print("The value: %i" % retrievedCm.get_stringPointWithCustomMeta())
print("The first metadata value : %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfBool())
print("The second metadata value: %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfString())
print("The third metadata value: %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfInt())
13.5.2. Using referenced classes¶
In this example we create a class that has a field which links a referenced class. This means that the filed will act as a placeholder for all class members of the class that is defined as referenced. The members can be accessed using the “.” dot keyword in the instance YAML definition. The class which we are refereeing to must already exist or be defined in the same YAML definition. The class uses isRef directive to identify the reference type in the configuration class YAML definition.
In the example we name the class MyClassWithRef. The following class definition YAML is prepared (configClassWithRef.cdt.yaml):
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithRef
parent: CiiConfigClass
__comment__: Configuration class with referenced class
members:
- name: device1
type: MySimpleClass
isRef: true
The configuration expects that referenced class, MySimpleClass is already present in the source code. To create the same reference field in GUI, the Generic Type “complex” must be used (Figure 5‑1).
Figure 5‑1 Adding referenced class in GUI
Once the class is prepared, we must define target configuration YAML. The target configuration will use the generated class MyClassWithRef. The following TC YAML is prepared (testRefTc.cdi.yaml):
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithRef
fields:
device1.testStringValue:
"@type": MySimpleClass.MdString
metadataInstance: ConfStringStd
metadataInstanceVersion: 1
value: 'Ref string defined value'
device1.testIntValue:
"@type": MySimpleClass.MdInt32
metadataInstance: ConfInt32Std
metadataInstanceVersion: 1
value: 100
device1.testDoubleValue:
"@type": MySimpleClass.MdDouble
metadataInstance: ConfDoubleStd
metadataInstanceVersion: 1
value: 200.33
When using the referenced classes, target configuration YAML fields and class types must be separated with “.” dot. For example: To set the value of testDobuleValue from the referenced field device1, the device1.testDoubleValue is used for the field value definition. Also, both types must be set, the class type and the metadata type (MySimpleClass.MdDouble).
After the class is generated (chapter 4.2 shows the example), we deploy the target configuration under the cii.config://mytest/myreftc:
config-tool deploy -i yaml/testRefTc.cdi.yaml -a cii.config://mytest/myreftc
The following sections show usage examples of referenced class in all languages.
13.5.2.1. Java referenced type¶
Listing 5‑8 Java referenced type example
/*** Usage of the referenced class */
CiiConfigClient client=CiiConfigClient.getInstance();
URI myUri = new URI("cii.config://*/mytest/myreftc");
//Retrieve config from local or remote DB
MyClassWithRef data=
client.retrieveConfig(myUri).getData(MyClassWithRef.class);
//Retrieve referenced class
MySimpleClass dataFromRef = data.getDevice1();
System.out.println("My double value:" +
dataFromRef.getTestDoubleValueValue());
System.out.println("My int value:" +
dataFromRef.getTestIntValue().getValue());
System.out.println("My string value:" +
dataFromRef.getTestStringValue().getValue());
13.5.2.2. CPP referenced type¶
Listing 5‑9 CPP referenced type example
/*** Usage of the referenced class */
std::shared_ptr<::elt::config::classes::MyClassWithRef> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MyClassWithRef>(elt::mal::Uri("cii.config://*/mytest/myreftc"));
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedDataRef = retrievedData->get_device1();
std::cout << "My int value:" << retrievedDataRef->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedDataRef->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedDataRef->get_testStringValue() << std::endl;
13.5.2.3. Python referenced type¶
Listing 5‑10 Python referenced type example
# Usage of the referenced class
uri = elt.config.Uri("cii.config://*/mytest/myreftc")
client = elt.config.CiiConfigClient
#Retrieve config configuration
retrieved = client.retrieve_config(uri, -1)
retFromRef = retrieved.get_device1()
print("My string value:" + retFromRef.get_testStringValue())
print("My integer value: {}".format(retFromRef.get_testIntValue()))
print("My double value: {}".format(retFromRef.get_testDoubleValue()))
13.5.3. Using check values¶
The metadata classes support a check for setting minimum and maximum values on number types. The limits can be set by setting the values of the minLimit (setMinLimit), maxLimit (setMaxLimit) metadata instance member fields. The check is disabled by default. To enable the checks, the flag checked (setChecked) has to be set to true.
The minLimit uses “value >= minLimit” check logic, while maxLimit uses “value < maxLimit” check logic. In case the value exceeds the limit, the CiiConfigLimitOutOfBoundsException exception is thrown. The check is executed when setting the value using the API.
The MdString class also supports allowed values check. This check verifies that the values for config points are part of a set of allowed values. The array of allowed values has to be set. In case the value is not part of the allowed values, the CiiConfigAllowedValuesException is thrown.
The following examples show the usage of min and max limits. The usage of allowed values follows the same principle.
13.5.3.1. Java checked functions¶
Listing 5‑11 Java checked functions example
/**
* Using the check functions
*/
try {
MdInt32 checkedIntClass = MdInt32.getDefaultMDI();
checkedIntClass.setMinLimit(1);
checkedIntClass.setMaxLimit(200);
checkedIntClass.setChecked(true);
//value in limits
checkedIntClass.setValue(50);
//value out of limits
checkedIntClass.setValue(200);
MySimpleClass testClass = new MySimpleClass();
testClass.setTestIntValue(checkedIntClass);
testClass.setTestDoubleValueValue(12.0d);
testClass.setTestStringValueValue("My test string");
} catch (CiiConfigLimitOutOfBoundsException e) {
System.out.println(e.getMessage());
}
13.5.3.2. CPP checked functions¶
Listing 5‑12 CPP checked functions example
/**
* Using the check functions
*/
std::shared_ptr<MdNumber<std::int32_t>> checkedIntClass =
std::dynamic_pointer_cast<elt::config::MdNumber<std::int32_t>>(
CiiDataPointMetadataFactory::getMetadataInstance(elt::common::CiiBasicDataType::INT32));
checkedIntClass->set_checked(true);
checkedIntClass->set_min_limit(1);
checkedIntClass->set_max_limit(200);
try{
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstance =
CiiNodeFactory::getNewNodeInstance<::elt::config::classes::MySimpleClass>();
//value in limits
newSimpleInstance->set_testIntValue(110, checkedIntClass);
//value out of limits
newSimpleInstance->set_testIntValue(201, checkedIntClass);
newSimpleInstance->set_testStringValue("My test string");
newSimpleInstance->set_testDoubleValue(12.0);
}
catch(const elt::config::CiiConfigLimitOutOfBoundsException& e) {
std::cout << e.what() << std::endl;
}
13.5.3.3. Python checked functions¶
Listing 5‑13 Python checked functions example
#Using the check functions
# this has to be called
checkedIntClass = elt.config.MdInt32.get_new_metadata_instance('')
checkedIntClass.set_checked(True)
checkedIntClass.set_min_limit(1)
checkedIntClass.set_max_limit(200)
try:
newSimpleInstance = MySimpleClass.get_new_node_instance()
newSimpleInstance.set_testStringValue("My test string")
#value in limits
newSimpleInstance.set_testIntValue(10, checkedIntClass)
#value out of limits
newSimpleInstance.set_testIntValue(202, checkedIntClass)
newSimpleInstance.set_testDoubleValue(12.0)
except elt.config.CiiConfigLimitOutOfBoundsException as cix:
print(cix)
13.6. Additional information¶
13.6.1. Config tool¶
The configuration tool (config-tool) is used to generate classes and to deploy and undeploy the YAML defined configurations to/from the local DB JSON structure. The tool also supports class generation from remote DB.
The config-tool has the following usage pattern: config-tool <operation> <options>.
To use the predefined simple metadata classes (ConfInt32Std, ConfStringStd, etc.), the classes must be deployed in the local DB. The config-tool init operation supports this. To initialize the local DB use:
config-tool init
For remote DB the config-initES.sh must be run. Please refer to [2] for instructions.
13.6.1.1. Class generation¶
The tool supports programming language configuration and metadata class files generation. The files are generated from YAML files (optionally JSON). To generate classes the generate option must be used. The following options are supported:
lang: cpp, java, python. Usage: -l <arg>. Default: java
type: config, metadata. Usage: -t <arg>. Default: config
input file. Usage: -i <arg>
output directory. Usage: -o <arg>
input file is in JSON. Usage: -j
generate wscript files for python bindings. Usage: -ws
The argument “o” is mandatory, other arguments are optional. When generating metadata classes, the type “t” must be properly set. To generate the classes from the remote DB, run the tool without “i” argument.
It is not allowed to mix the metadata and configuration classes in one YAML file. As configuration classes change frequently and metadata classes will not change much, the metadata and configuration class generation are separated. The “t” argument forces the user to explicitly know that one is actually generating metadata classes.
The class generation tool has no support for checking the names of existing classes and field names of parent classes. It is up to the user, to properly check the existing fields of the parent classes. However, the tool will check for the reserved words for field names which also contain the field names of the predefined classes. Appendix D contains the list of reserved words.
13.6.1.2. Deployment and undeployment¶
The tool supports deployment and undeployment of instances files to/from local DB. To deploy the instances the deploy option must be used. The following options are supported:
address: target configuration URI. Option: -a <arg>
input file. Option: -i <arg>
type: config, metadata. Option: -t <arg> Default: config
input file is in JSON. Option: -j
version. Option: -v <arg>
The arguments “a” and “i” are mandatory. When deploying metadata instances the type “t” must be properly set. If no version “-v” is specified the instances are always deployed under version 1.
To undeploy instances, the undeploy option must be used. The following options are supported:
address: target configuration URI. Option: -a <arg>
instancenames: Metadata instance names to be deleted,
separated with “,”. Option: -n <args>
type: config, metadata. Option: -t <arg> Default: config
input file is in JSON. Option: -j
version to be deleted. Option: -v <arg>
The version “v” must be explicitly specified (for safety reasons). To delete all versions, use -1.
The argument “n” must be used with type metadata. In this case the argument “a” is ignored.
To get a list of all available operations, run the:
config-tool -h
To get help on the chosen operation, run config-tool <operation> -h. Example:
config-tool undeploy -h
13.6.2. Generic types¶
Metadata classes partly support generic type inference. The generic type can be used in all the metadata classes, but the actual type used must be defined in the configuration class definition. There is no limit for the actual type to be defined before in the hierarchy of metadata. When the generic type is fixed, there is no need to define the type in the configuration class definition. Example of these classes are all classes with defined type (MdString, MdInt32). These classes have the actual type defined in the metadata class definition, so the type cannot be defined in the configuration class definition. To set the type of the metadata class to be generic, the T keyword must be used (no other keywords are allowed). The keyword tells the metadata class definition that the actual type will be set by the configuration class. Using the defined generic types in the metadata class definition (e.g.: INT32) will determine the generic type of the class. If the type is fixed in the metadata class hierarchy, the genericType attribute of configuration class must be left out or set to genericType: NONE.
Listing 6‑1 shows example of custom generic metadata class (MyCustomMetaClass) definition in YAML. Listing 6‑2 shows use of the custom generic metadata class in configuration class YAML definition (MyClassWithCustomMeta). The type INT32 is set in the configuration class definition.
Listing 6‑1 Generic type metadata class
configuration:
metadataClassesConfiguration:
classes:
- name: MyCustomMetaClass
parent: MdNumber
__comment__: Custom metadata class
genericType: T
members:
- name: metaOfBool
type: BOOLEAN
- name: metaOfString
type: STRING
- name: metaOfInt
type: INT32
Listing 6‑2 Usage of generic type metadata class
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithCustomMeta
parent: CiiConfigClass
__comment__: Class with custom metadata
members:
- name: stringPointWithCustomMeta
type: MyCustomMetaClass
genericType: INT32
CII Configuration provides an option to have generic type metadata classes and to set the type of the config point metadata class in the configuration class definition. This option can be used for classes, however the instances definition YAML files must have generic type specified for serializers to properly determine the type of data provided.
Types defined in instances must be the same as types defined in the classes in order for configuration to work properly. Listing 6‑3 and Listing 6‑4 show the examples of instances YAML definition files.
Listing 6‑3 Config instance when generic type is used
metadata:
instances:
- "@type": MyCustomMetaClass
"@name": myCustomMDI
"@genType": INT32
comment: "String MDI with values"
metaOfBool: "true"
metaOfString: "The string metadata example"
metaOfInt: "12"
Listing 6‑4 Configuration instances with generic type YAML
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithCustomMeta
fields:
stringPointWithCustomMeta:
"@type": MyCustomMetaClass
"@genType": INT32
metadataInstance: myCustomMDI
metadataInstanceVersion: 1
value: 13
13.6.3. Java search example¶
Listing 6‑5 shows the Java example for configuration search. The same API can also be used for CPP and Python. A different search language is used for local and remote DB:
When searching in local database, a json path expression should be used: https://github.com/json-path/JsonPath
When searching in remote database, an Elasticsearch query DSL should be used: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax
Searching with both languages is shown in the example.
Listing 6‑5 Java search example
// remote search example
CiiConfigClient client = CiiConfigClient.getInstance();
// search or configuration with name 'temperatureSensor'
String configurationName = "temperatureSensor";
final String searchExpression = String.format("data.name:%s", configurationName);
try {
List<CiiTargetConfig> foundConfigurations = client
.searchConfigRepo(ApiSearch.REMOTE, searchExpression);
} catch (CiiConfigNoTcException e) {
// handle exception in case there was no such configuration
} catch (CiiConfigSearchException e) {
// handle exception in case an error occurred while searching for configuration
}
// local search example
CiiConfigClient client = CiiConfigClient.getInstance();
// search locally for configuration with name 'temperatureSensor'
String configurationName = "temperatureSensor";
final String search = String.format("$..[?(@.name == '%s')]", configurationName);
try {
List<CiiTargetConfig> foundConfigurations = client.searchConfigRepo(ApiSearch.LOCAL, search);
} catch (CiiConfigNoTcException e) {
// handle exception in case there was no such configuration
} catch (CiiConfigSearchException e) {
// handle exception in case an error occurred while searching for configuration
}
13.6.4. Large data – binary files¶
The MdArray and MdBinary types support storing of large data. The CII Configuration will store all binary data (MdBinary) as file references. All the arrays (MdArray) exceeding 100kb (this limit is hard coded) will be stored as binary files and only references to files will be passed to predefined metadata field “file”. Data is stored as a raw data in big-endian order.
Configuration instance YAML definition supports deployment of the raw data files. To properly deploy the file, it must be placed in the subfolder named “files”. The name of the file must be set in the file field of the config point. In Listing 6‑6 the picture1 config point will use the file MyBigData.raw as file reference. When the instance is deployed the file will be copied to local DB or remote large data storage.
Listing 6‑6 Configuration instance field with file
picture1:
"@type": MdBinary
metadataInstance: ConfBinaryStd
metadataInstanceVersion: 1
value:
file: MyBigData.raw
13.6.5. Extending metadata classes with additional logic¶
Cii Configuration provides built-in check functions (5.3), but has no direct support for extending the metadata classes with additional logic. The only way to add additional check functions is to modify the source code of the CII Configuration. For such an action, deep knowledge of internals is needed. As it is not intended for the end user to modify the source code, here only some short guidelines are provided.
To add additional logic in Java, the logic has to be put into one of the existing metadata classes. The logic has to be programmed in a separate private method, and then added to existing setValue method. The recommended classes for additional logic are MdNumber, MdArray, MdString. The logic method can also be put in MdBase (as protected), but it has to be called in setValue method of the MdNumber, MdArray or MdString classes.
For CPP additional logic has to be added as a private method in the MdBase.hpp file. The custom logic private method then has to be added to the existing validate method. As the method must support different types of an input value, it has to be a templated method.
As the Python bindings to CPP are used, no additional work is needed for the Python.
13.6.6. Accessing statistics using the JMX¶
Config service exposes basic statistic using the Java MBeans JMX interface. Statistics can be obtained using various JMX tools. The following example shows usage of jconsole. First start the jconsole with:
jconsole&
Connect to the Config service process:
Figure 6‑1 Jconsole JMX connection
After connection is established, the console with statistics is displayed. The console displays heap usage, CPU usage, number of threads and similar java statistics. The specific Config service statistics is exposed under the elt.config.service.stat namespace.
To obtain statistics click on the MBeans tab and browse the elt.config.service.stat namespace:
Figure 6‑2 Jconsole configuration service statistics
13.7. GUI¶
13.7.1. General¶
Configuration GUI is a graphical editor application that can be used for browsing, querying and editing the configuration and metadata classes, target configurations and metadata instances. None of the configuration data is stored locally, everything is being sent to the remote configuration service. Config GUI is thus intended to work with the remote DB only.
Configuration GUI can be executed by issuing the following command from the shell:
config-gui
13.7.2. Configuration GUI settings¶
By default, the configuration GUI will look at ciiconfservicehost for config service and at http://ciihdfshost:9870/webhdfs/v1/esoLs for HDFS. For logging, default settings of CII logging will be used. Optionally, the executable takes 2 arguments:
path_to_application_config_file: a path to the file that contains the settings for the Configuration GUI application. Usually, the configuration file for the Configuration GUI application is named app-config.ini. A detailed description of the application configuration file can be found in Appendix F.
path_to_log_config_file: a path to the file that contains the settings for the log4cplus logging library. If this argument is not provided, the basic log4cplus configuration is used. Usually, the configuration file for the log4cplus library is named log4cplus.properties. Example value for the attribute: ../log4cplus.properties.
To display the help, “-h” parameter shall be used.
13.7.3. Main Window¶
The main window is displayed when we execute the Configuration GUI. The application screenshot is shown in Figure 7‑1. As we can see from the figure, the functionality of the Configuration GUI is organized into 4 groups, each group having its own tab:
Target Config: this tab is used for viewing and managing configurations. It is described in more detail in section 7.5.
Configuration class: this tab is used for viewing and managing configuration classes. It is described in more detail in section 7.6.
Metadata Class: this tab is used for viewing and managing metadata classes. It is described in more detail in section 7.7.
Metadata Instance: this tab is used for viewing and managing metadata instances. It is described in more detail in section 7.8.
Figure 7‑1 Main window.
Every tab is divided into 2 parts. The left part displays the list of entities (i.e. configurations, configuration classes, metadata instances, metadata classes) retrieved from the remote service and the button for adding new entities. In some cases, also the delete button is present. The user can select entities by clicking them. A list of entries can be refreshed by clicking on the Refresh button.
The right part is used for displaying the data about the selected entity: name, parent, list of attributes, etc. For some tabs, the right part of the GUI also contains the controls for selecting and managing entity versions.
13.7.3.1. Filtering the values¶
For easier searching of the entries, a search filter is implemented. The filer can be used for searching the classes and instances. The filter is presented as input box on each of the tabs (Figure 7‑2). Filter can also be used in the dialogs (Figure 7‑3). The user can input a part of the entire and all the entries whose name doesn’t contain the string input are filtered out (the string comparison is case insensitive). In order to display all the entries again, the user needs to clear the filtering text.
Figure 7‑2 Filter input box location on each tab
Figure 7‑3 Filter input box location on a dialog
13.7.5. Target Config Tab¶
The Target Config tab enables the user to do the following actions:
Navigate and view the target configurations in the remote database.
Control configuration values – manage configuration control operations.
Create target configurations on the desired URI endpoints.
View revision history of target configurations and increase versions.
Filter the navigation tree (section 7.3.1)
13.7.5.1. Selecting configuration¶
The user selects a configuration by clicking the leaf item in the configuration hierarchy tree (Figure 7‑4). If the outer tree node is clicked or Refresh button is pressed, the configuration is deselected. The configurations are organized in the tree structure based on their URI paths (2.2.2).
When a configuration is selected, the following data is loaded in the right part of the GUI:
URI: the unique URI that identifies the given configuration.
Version: the most recent version number of the configuration,
Target Configuration class: the assigned configuration class. It determines the list of attributes the configuration contains.
The list of configuration attributes and their values: the list of attributes is comprised of the configuration class’s attributes and the attributes of all its parents’ classes.
Figure 7‑4 Selecting configuration.
13.7.5.2. Adding new configuration entity¶
The adding of a new configuration entity is initiated by clicking the Add button below the hierarchy tree. The button is enabled only in the “Write Enabled” mode.
When the “Add” button is clicked, a dialog for inserting configuration data is displayed (Figure 7‑5). The dialog contains the following fields:
URI prefix of TC: a read-only field that contains the URI prefix (cii.config:///*).
URI suffix of TC: the input field where the user can input the URI suffix. The URI suffix actually defines the TC endpoint. The URI suffix can contain multiple “/” characters: in such case, multiple intermediate nodes will be created in the tree hierarchy. The final URI of the configuration is formed by concatenating the URI prefix and suffix. For example URI suffix /test_node/test_point will create TC under the cii.config:///test_node/test_point*.
Configuration class: here, the user selects the configuration class for the newly created configuration.
After the user clicks the OK button, the dialog is closed and a new configuration is created on the remote server.
Figure 7‑5 Adding new configuration entity.
13.7.5.3. Setting configuration value¶
The setting of the value for the configuration attribute is performed using one of the input data dialogs (Figure 7‑6) that are displayed when clicking the Value cell of the given attribute.
There are 3 types of input data dialogs:
Primitive data input dialog: this dialog is used for entering the majority of data, except for vector and binary data. It contains a simple input field where the user can input the data value.
- Vector data input dialog: this dialog is used for inputting vector data and matrices (see organisation of matrices in 2.1.2). The dialog contains the input field where the user can input the content for smaller vectors. Compared to primitive data input dialog, this dialog enables input of larger data. The vector input data must be formatted as JSON arrays. Figure 7‑7 shows the example of vector input data type.Besides the input field, the dialog also contains the “Upload File” button which can be used for uploading the file containing the vector elements to the remote HDFS server. After the successful upload of the file, its URL is inserted in the input field. Below the upload button there is the “Download” button that can be used to download the file from the remote HDFS server.
Binary data input dialog: The dialog contains the “Upload File” button which can be used for uploading the binary file to the large storage server. After the successful upload of the file, file name is inserted in the input field. Below the upload button there is the “Download” button that can be used to download the file from the large storage server.
After the user clicks the “OK” button, the dialog is closed, however the updates aren’t stored on the remote DB yet. They are only displayed in the attribute table. In order to trigger the updating on the remote server, the user needs to click the Commit button.
Note: When inputting vector data as a file, it is treated the same way as binary data input. All files must contain raw data in big-endian ordering.
Figure 7‑6 Setting configuration attribute value.
Figure 7‑7 List box input for arrays.
13.7.5.4. Setting metadata instance and version¶
The setting of the metadata instance and a version for the configuration attribute is performed using the dialog that is displayed when clicking the Meta Data Instance cell of the given attribute. In the dialog, the user selects the metadata instance and its version. After the user clicks the “OK” button, the dialog is closed, however, the updates aren’t stored on the remote server yet. They are only displayed in the attribute table. In order to trigger the updating on the remote server, the user needs to click the Commit button.
13.7.5.5. Version management¶
The switching between different versions of the configuration is performed by using the Version combo box in the right section of the tab. When selecting a particular version number, the data of the corresponding configuration version will be displayed.
The user can also increase the version of the configuration by clicking the Increase Version button. When doing this, the most recent version gets copied on the remote server, the version number gets increased and the new latest version gets loaded in the GUI.
13.7.6. Configuration class Tab¶
The Configuration class tab enables the user to do the following actions:
Create, delete, view and browse configuration classes.
Define the structure and member fields of configuration classes.
Adding additional member fields to the configuration class.
Reference another configuration class (using the complex type).
Filter the navigation tree (section 7.3.1)
13.7.6.1. Selecting configuration class¶
The user selects a configuration class by clicking the item in the configuration class hierarchy tree (Figure 7‑8). The configuration classes are organized in the tree structure based on their “parent” attribute.
When a configuration class is selected, the following data is loaded in the right part of the GUI:
Name: the name of the configuration class.
Parent Class: the name of the configuration class’s parent class. This attribute is empty for root configuration classes.
the list of configuration class attributes: the list of attributes is comprised only of the selected configuration class’s attributes. In order to see the configuration class’s parents’ attributes, the parent class has to be selected in the hierarchy tree.
Figure 7‑8 Selecting configuration class.
13.7.6.2. Adding new configuration class entity¶
Adding a new configuration class entity is initiated by clicking the Add button below the hierarchy tree. The button is only enabled in the “Write Enabled” mode.
When the “Add” button is clicked, a dialog for inserting a configuration class data is displayed (Figure 7‑9). The dialog contains the following fields:
Name: the name of the newly-created configuration class.
Parent Configuration Class: here, the user selects the parent configuration class for the newly-created configuration class.
After the user clicks the “OK” button, the dialog is closed and a new configuration class is created on the remote server.
Figure 7‑9 Adding new configuration class entity.
13.7.6.3. Adding member fields to the configuration class¶
Adding of new member field (configuration class attribute) is initiated by clicking the Add attribute button below the attributes table. The button is only enabled in the “Write Enabled” mode.
When the “Add attribute” button is clicked, a dialog for inserting a configuration class attribute data is displayed (Figure 7‑10). The dialog contains the following fields:
Name: the name of the newly-created member field.
Generic Type: here, the user selects generic type for the member field. This attribute is optional: for the non-generic member fields the user should select the “none” option from the combo box. When the type of the member field is a reference to another configuration class, this option must be set to complex (Figure 7‑11).
Metadata Class: here, the user selects the metadata class for the newly created member field.
After the user clicks the “OK” button, the dialog is closed. However, the updates aren’t stored on the remote server yet. They are only displayed in the attribute table. In order to trigger the updating on the remote server, the user needs to click the Commit button.
Figure 7‑10 Adding a new configuration class attribute.
Figure 7‑11 Adding referenced configuration class attribute.
13.7.6.4. Delete configuration class¶
To delete the configuration class, first make sure that Dangerous action is ticked under the Options menu (Figure 7‑12). After this is enabled, click on the desired configuration class and press Delete button. A warning is displayed. Click OK to finally delete the class.
Figure 7‑12 Deleting configuration class attribute.
13.7.7. Metadata Class Tab¶
The Metadata Class tab enables the user to do the following actions:
Create, delete, view and browse metadata classes.
Define the structure and member fields of metadata classes.
Filter the navigation tree (section 7.3.1)
13.7.7.1. Selecting metadata class¶
The user selects a metadata class by clicking the item in the metadata class hierarchy tree (
Figure 7‑13). The metadata classes are organized in the tree structure based on their “parent” attribute.
When a metadata class is selected, the following data is loaded in the right part of the GUI:
Name: the name of the metadata class.
Parent Class: the name of the metadata class’s parent class. This attribute is empty for root metadata classes.
Generic Type: the generic type for the metadata class.
the list of metadata class attributes: the list of attributes is comprised only of the selected metadata class’s attributes. In order to see the metadata parents’ attributes, the parent class has to be selected in the hierarchy tree.
Figure 7‑13 Selecting metadata class.
13.7.7.2. Adding new metadata class entity¶
Adding of a new metadata class entity is initiated by clicking the Add button below the hierarchy tree. The button is only enabled in the “Write Enabled” mode.
When the “Add” button is clicked, a dialog for inserting a metadata class data is displayed (Figure 7‑14). The dialog contains the following fields:
Name: the name of the newly-created metadata class.
Parent Metadata Class: here, the user selects the parent metadata class for the newly-created metadata class.
Generic Type: here, the user selects a generic type for the newly-created metadata class. This attribute is optional: for instances whose metadata class is non-generic the “none” value should be selected.
After the user clicks the “OK” button, the dialog is closed and a new metadata class is created on the remote server.
Figure 7‑14 Adding metadata class entity.
13.7.7.3. Adding metadata class attribute¶
Adding of new metadata class attribute is initiated by clicking the Add attribute button below the attributes table. The button is only enabled in the “Write Enabled” mode.
When the “Add attribute” button is clicked, a dialog for inserting a metadata class attribute data is displayed (Figure 7‑15). The dialog contains the following fields:
Name: the name of the newly-created metadata class attribute.
Type: the type of the newly-created metadata class attribute. The user is able to choose from a pre-defined list of basic types.
After the user clicks the “OK” button, the dialog is closed, however, the updates aren’t stored on the remote server yet. They are only displayed in the attribute table. In order to trigger the updating on the remote server, the user needs to click the Commit button.
Figure 7‑15 Adding metadata class attribute.
13.7.7.4. Delete metadata class¶
To delete the metadata class, first make sure that Dangerous action is ticked under Options menu (Figure 7‑12). After this is enabled, click on the desired metadata class and press Delete button. The warning is displayed. Click OK to finally delete the class.
13.7.8. Metadata Instance Tab¶
The Metadata Instance tab enables the user to do the following actions:
Define instances of metadata classes.
Assign metadata values to metadata member fields.
Update metadata instances values.
View revision history of metadata instances and increase versions.
Filter the navigation list.
13.7.8.1. Selecting metadata instance¶
User selects a metadata instance by clicking the item in the metadata instance list (
Figure 7‑16).
For the easier searching of metadata instances, search and dropdown filters are implemented:
With the search filter, the user can input a part of the name and all the metadata instances whose name doesn’t contain the string input are filtered out (the string comparison is case insensitive). In order to display all the metadata instances again, the user needs to clear the filtering text.
In the dropdown filter there is a list of all metadata classes. When the user choses one of the metadata classes from the dropdown filter only the metadata instances with that metadata class stay on the list, the others get filtered out. In order to display all the metadata instances again, the user needs to choose the empty option in the dropdown filter.
When a metadata instance is selected, the following data is loaded in the right part of the GUI:
Name: the name of the metadata instance.
Version: the most recent version of the metadata instance.
Metadata Class: the metadata class assigned to a given metadata instance. It determines the list of attributes the metadata instance contains.
Generic Type: the generic type for the metadata instance. The generic type for the metadata instance will be automatically retrieved from the metadata class generic type.
The list of metadata instance attributes, their values and their types: the list of attributes is comprised of the metadata class’s attributes and the attributes of all its parents’ classes.
Figure 7‑16 Selecting metadata instance.
13.7.8.2. Adding new metadata instance¶
The adding of new metadata instance entity is initiated by clicking the Add button below the list of instances. The button is only enabled in the “Write Enabled” mode.
When the “Add” button is clicked, a dialog for inserting a metadata instance data is displayed (Figure 7‑17). The dialog contains the following fields:
Name: the name of the newly-created metadata instance.
Metadata Class: here, the user selects a metadata class for the newly-created metadata instance.
After the user clicks the “OK” button, the dialog is closed and a new metadata instance is created on the remote server.
Figure 7‑17 Adding new metadata instance.
13.7.9. Setting metadata instance value¶
The setting of the value for the metadata instance attribute is performed using the primitive data input dialog (Figure 7‑18) that is displayed when clicking the Value cell of the given attribute. This dialog contains a simple input field where the user can input the data value.
After the user clicks the “OK” button, the dialog is closed, however, the updates aren’t stored on the remote server, yet. They are only displayed in the attribute table. In order to trigger the updating on the remote server, the user needs to click the Commit button.
Figure 7‑18 Setting metadata instance attribute value.
13.7.9.1. Version management¶
The switching between different versions of the metadata instances is performed by using the Version combo box in the right section of the tab. When selecting a particular version number, the data of the corresponding metadata instance version will be displayed.
The user can also increase the version of the metadata instance by clicking the Increase Version button. When doing this, the most recent version gets copied on the remote server, the version number gets increased and the new latest version gets loaded in the GUI.
13.7.10. Notify OLDB button¶
OLDB [1] internally uses the CII Configuration remote DB (target configuration and metadata instances) for its configuration. When configuration for OLDB changes, the calculation nodes and scheduler service must be notified about the change to use the changed values. When using OLDB client API, the services are automatically notified. But if the configuration settings are changed using the config GUI, the services are not automatically notified, as the configuration has no dependency on the OLDB. To overcome this problem, Notify OLDB button was added. This button can be used to notify OLDB services about the changes in the target configuration or metadata. Proper target configuration or metadata instance must be selected and button Notify OLDB clicked to trigger the notification. The usage of this button only makes sense on the config points (URI cii.config://*/oldb) and metadata instances (prefixed with Oldb) used by OLDB.
Figure 7‑19 Notify OLDB button
13.8. Config client API listing¶
Config client API provides methods for the CRUD operations of configuration and metadata instances. After the listing with methods description, the source code of specific programming language API is provided (A.1,A.2,A.3). Additional methods for searching the configurations are also present. The following methods are provided:
getInstance() |
|---|
This is Java only method that returns instance of Config client API |
saveTargetConfig(uri, rootNode) |
|---|
Saves the configuration class to the specified URI location (cache, local, remote) and returns a version that was assigned to the saved configuration. Used programming language instance of configuration class must be set to rootNode parameter. |
saveTargetConfigWithMetadata(uri, rootNode) |
|---|
Saves the configuration class to the specified URI location (cache, local, remote) and returns a version that was assigned to the saved configuration. The method will also save all the metadata instances that are programmatically created and used in the programming language configuration class instance. Used programming language instance of configuration class must be set to rootNode parameter. |
retrieveConfig(uri) | retrieveConfig(uri, version) |
|---|
Retrieves the chosen version of target configuration from the specified location (determined by the uri). If the version is set to -1, the last target configuration will be retrieved (configuration with the highest version that is stored at the specified location. The overload with no version will return the latest version. |
updateConfig(uri, rootNode) | updateConfig(uri, version, rootNode) |
|---|
Updates target configuration at the specified URI with the data from the Configuration class set to the rootNode parameter. Overload with no version updates the last known configuration version. |
deleteConfig(uri, version) |
|---|
Deletes exact version of target config at defined URI location. If the version is set to -1, all configurations of the given URI will be deleted. |
saveMetadata(host, metadataInstance) |
|---|
Saves new metadata instance into the specified location. Parameter host determines the location under which metadata will be saved to. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
retrieveMetadata(host, instanceName) | retrieveMetadata(host, instanceName, version) |
|---|
Retrieves specific version of the metadata from the chosen location saved under the chosen instanceName. Parameter host determines the location where metadata will be retrieved from. The overload without version will return the last saved version. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
updateMetadata(host, instanceName, version) |
|---|
Updates specific version of the metadata with instanceName, under the chosen host location. Parameter host determines the location in which metadata will be updated. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
deleteMetadata (host, instanceName, version) |
|---|
Deletes specific version of the metadata with instanceName, under the chosen host location. Parameter host determines the location where metadata will be deleted from In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
searchConfigRepo (host, query) |
|---|
Searches for target configurations that can be found using the chosen search expression. Depending on the location of the searches different search expressions should be used. For local searches a search expression should be written in JSON path format [3]. When searching in remote database a search expression should be written in elastic search query DSL format [4]. Host: In Java the first part of the path that determines the location must be provided (cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“local”, “remote”). |
getChildren (uri) |
|---|
The method returns all the children config point URIs under the certain path. The method will return list of URIs with the config points. |
setWriteEnabled (enabled) | isWriteEnabled() |
|---|
Sets or checks the write enabled state flag. If write is enabled we can save, update and delete target configurations. If write is not enabled we can only retrieve the data. |
13.8.1. Java API code¶
public static synchronized CiiConfigClient getInstance() throws CiiConfigInitializationError { ... }
public List<CiiConfigStatus> getConfigStatus(URI uri) throws CiiInvalidURIException { ... }
public int saveTargetConfig(URI uri, CiiConfigClass rootNode)
throws CiiConfigSaveException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public int saveTargetConfigWithMetadata(URI uri, CiiConfigClass rootNode)
throws CiiConfigWriteDisabledException, CiiInvalidURIException, CiiConfigSaveException { ... }
public void updateConfig(URI uri, CiiConfigClass rootNode)
throws CiiConfigNoTcException, CiiConfigUpdateException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public void updateConfig(URI uri, int version, CiiConfigClass rootNode)
throws CiiConfigNoTcException, CiiConfigUpdateException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public CiiTargetConfig retrieveConfig(URI uri)
throws CiiConfigNoTcException, CiiInvalidURIException { ... }
public CiiTargetConfig retrieveConfig(URI uri, int version)
throws CiiConfigNoTcException, CiiInvalidURIException { ... }
public void deleteConfig(URI uri, int version)
throws CiiConfigDeleteException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public List<CiiTargetConfig> searchConfigRepo(ApiSearch searchPath, String searchExp)
throws CiiConfigNoTcException, CiiConfigSearchException { ... }
public List<String> remoteIndexSearch(String index, String query)
throws CiiConfigSearchException { ... }
public <T extends MdBase> T retrieveMetadata(URI searchLocation,
String metadataInstanceName, int version, Class<T> clazz)
throws IOException, CiiInvalidURIException { ... }
public <T extends MdBase> int saveMetadata(URI saveLocation, T metadataInstance)
throws CiiConfigSaveException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public void deleteMetadata(URI deleteLocation, String metadataInstanceName, int version)
throws CiiConfigWriteDisabledException, CiiInvalidURIException, CiiConfigDeleteException { ... }
public List<URI> getChildren(URI uri) throws CiiInvalidURIException, CiiConfigSearchException { ... }
public void setWriteEnabled(boolean writeEnabled) { ... }
public boolean isWriteEnabled() { ... }
13.8.2. CPP API code¶
static int saveTargetConfig(const elt::mal::Uri& uri, const CiiConfigClass& root_node);
static int saveTargetConfigWithMetadata(const elt::mal::Uri& uri, CiiConfigClass& root_node);
static std::list<std::shared_ptr<CiiDataPointMetadataBase>> getTargetConfigMetadata(CiiConfigurationBase& root_node);
static int saveMetadata(const std::string& host, CiiDataPointMetadataBase& metadata);
static std::shared_ptr<CiiConfigClass> retrieveConfig(
const elt::mal::Uri& uri, int version = -1);
static std::shared_ptr<CiiDataPointMetadataBase> retrieveMetadata(
const std::string& host, const std::string& instance_name, int version = -1);
static void updateConfig (const elt::mal::Uri& uri, int version,
const CiiConfigClass& targetConfig);
static void updateMetadata(const std::string& host, const CiiDataPointMetadataBase& metadata);
static void updateMetadata(const std::string& host, int version, const CiiDataPointMetadataBase& metadata);
static void deleteConfig (const elt::mal::Uri& uri, int version = -1);
static void deleteMetadata (const std::string& host, const std::string& instance_name, int version = -1);
template <typename T>
static std::shared_ptr<T> getConfigData(const elt::mal::Uri& uri, int version = -1) {
return std::dynamic_pointer_cast<T>(retrieveConfig(uri, version));
}
template <typename T>
static std::shared_ptr<T> getMetadata(const std::string& host, const std::string& instance_name, int version = -1) {
return std::dynamic_pointer_cast<T>(retrieveMetadata(host, instance_name, version));
}
static const std::list<elt::mal::Uri> getChildren(const elt::mal::Uri& parent_uri);
static const std::list<std::shared_ptr<CiiConfigClass>> searchConfigRepo(const std::string& host, const std::string& query);
static void setWriteEnabled(bool enabled);
static bool isWriteEnabled();
13.8.3. Python bindings code¶
py::class_<elt::config::CiiConfigClient, std::shared_ptr<elt::config::CiiConfigClient>>(m, "CiiConfigClient",
"Configration Client API")
.def_static("save_target_config", &elt::config::CiiConfigClient::saveTargetConfig,
"Save a new version of configuration class")
.def_static("save_target_config_with_metadata", &elt::config::CiiConfigClient::saveTargetConfigWithMetadata,
"Save configuration node and the metadata object it references")
.def_static("retrieve_config", &elt::config::CiiConfigClient::retrieveConfig,
"Retrieve a configuration object previously saved via saveTargetConfig")
.def_static("retrieve_config",
[](const ::elt::mal::Uri &uri) { return elt::config::CiiConfigClient::retrieveConfig(uri, -1); },
"Retrieve a configuration object previously saved via saveTargetConfig")
.def_static("update_config", &elt::config::CiiConfigClient::updateConfig,
"Overwrite previously saved configuration or save it using a specific version")
.def_static("delete_config", &elt::config::CiiConfigClient::deleteConfig,
"Delete a configuration object")
.def_static("delete_config", [](const elt::mal::Uri &uri) {
elt::config::CiiConfigClient::deleteConfig(uri);
}, "Delete a configuration object")
.def_static("get_config_data", &elt::config::CiiConfigClient::getConfigData<elt::config::CiiConfigClass>,
"Retrieve a configuration object")
.def_static("save_metadata", &elt::config::CiiConfigClient::saveMetadata,
"Save a new version of a metadata instance")
.def_static("update_metadata", &elt::config::CiiConfigClient::updateMetadata,
"Retrieve a metadata object previously saved via saveMetadata()")
.def_static("delete_metadata", &elt::config::CiiConfigClient::deleteMetadata,
"Delete a metadata object")
.def_static("delete_metadata", [](const std::string &host, const std::string& instance_name) {
::elt::config::CiiConfigClient::deleteMetadata(host, instance_name);
}, "Delete a metadata object")
.def_static("retrieve_metadata", &elt::config::CiiConfigClient::retrieveMetadata,
"Retrieve a metadata object previously saved via saveMetadata")
.def_static("retrieve_metadata", [](const std::string &host, const std::string &instance_name) {
return elt::config::CiiConfigClient::retrieveMetadata(host, instance_name);
}, "Retrieve a metadata object previously saved via saveMetaData")
.def_static("search_config_repo", &elt::config::CiiConfigClient::searchConfigRepo,
"Search a data provider for specific configurations")
.def_static("set_write_enabled", &elt::config::CiiConfigClient::setWriteEnabled,
"Enables or disables the possibility of writing into the configuration databases")
.def_static("is_write_enabled", &elt::config::CiiConfigClient::isWriteEnabled,
"Returns True if the API allows writing into configuration databases, false otherwise")
.def_static("get_children", &elt::config::CiiConfigClient::getChildren,
"Query children nodes")
13.9. Config point value type to language types mapping¶
Below is the mapping table between config point types and data types of all programming languages that the configuration can use.
Table 7‑1 Config point types mapping table
ELT CII BASIC TYPE |
Java |
CPP |
Python |
|
|---|---|---|---|---|
INT32 |
Integer |
std::int32_t |
int |
|
INT64 |
Long |
std::int64_t |
int |
|
UINT8 |
Integer |
std::uint8_t |
int |
|
UINT16 |
Long |
std::uint16_t |
int |
|
UINT32 |
Long |
std::uint32_t |
int |
|
SINGLE |
Float |
float |
float |
|
DOUBLE |
Double |
double |
float |
|
BOOLEAN |
Boolean |
bool |
bool |
|
STRING |
String |
std::string |
str |
|
BINARY |
byte[] |
std::vector<std::uint8_t> |
bytearray |
13.10. Default metadata instance mapping¶
Table 7‑2 shows the default metadata instance and class type mappings. The columns “metadata instance @genType” and “class genericType” are CII Basic types. These columns are used in YAML class definition for metadata classes and metadata instances. The fields are described in Appendix E.
Table 7‑2 Default metadata mappings
Metadata class name |
Metadata instance @genType |
Class genericType |
Is Array |
Default metadata instance |
|---|---|---|---|---|
MdInt32 |
INT32 |
INT32 |
F |
ConfInt32St d |
MdInt64 |
INT64 |
INT64 |
F |
ConfInt64St d |
MdFloat |
SINGLE |
SINGLE |
F |
ConfFloatSt d |
MdDouble |
DOUBLE |
DOUBLE |
F |
ConfDoubleS td |
MdBool |
BOOLEAN |
BOOLEAN |
F |
ConfBoolStd |
MdString |
STRING |
STRING |
F |
ConfStringS td |
MdStringArr ay |
VECTOR_STRI NG |
STRING |
T |
ConfStringA rrayStd |
MdInt32Arra y |
VECTOR_INT3 2 |
INT32 |
T |
ConfInt32Ar rayStd |
MdFloatArra y |
VECTOR_SING LE |
SINGLE |
T |
ConfFloatAr rayStd |
MdDoubleArr ay |
VECTOR_DOUB LE |
DOUBLE |
T |
ConfDoubleA rrayStd |
MdInt32Matr ix |
MATRIX2D_IN T32 |
INT32 |
T |
ConfInt32Ma trixStd |
MdFloatMatr ix |
MATRIX2D_SI NGLE |
SINGLE |
T |
ConfFloatMa trixStd |
MdDoubleMat rix |
MATRIX2D_DO UBLE |
DOUBLE |
T |
ConfDoubleM atrixStd |
MdBinary |
BINARY |
BINARY |
F |
ConfBinaryS td |
13.11. Class definition reserved words¶
Table 7‑3 shows list of words that cannot be used as field names in config or metadata class definitions.
Table 7‑3 List of words not allowed for config and metadata class names
Field name |
|---|
name |
uri |
defaultValue |
value |
comment |
ciiType |
genType |
metadataChanged |
minLimit |
maxLimit |
allowedValues |
size |
field |
file |
genericType |
parent |
__comment__ |
isRef |
13.12. JSON/YAML Schema¶
Schemas are defined for classes and instances. All local and remote DB JSON files must be in line with the defined schemas. Target configuration and instances schemas are used for value bindings. Class schemas are used for class generation. For local and remote DB, YAML structures are translated into JSON structures with nested elements.
The following schema descriptions are indented in the same manner as the actual YAML files. The indentation follows the YAML files indentation structure.
All class schemas start with the starting structure, which is presented only here and not listed in the subchapters. This structure contains the following elements: “field: type, required/optional; description”. The definition is as follows:
configuration: string, required; defines the root for all configuration schemas
configClassesConfiguration or metadataClassesConfiguration string, required; defines the type of configuration schema.
13.12.1. Configuration class schema¶
Configuration class schema (field: type, required/optional; description):
classes: array, required; List of classes to be generated.
name: string, required; Name of the configuration class.
parent: string, required; Parent configuration class name.
- __comment__: string, optional; Class comment. This comment ischanged to programming language specific comment onbindings generation.
members: array, required; member fields of classes (config points)
name: string, required; config point name.
type: string, required; config point metadata class type.
genericType: string, optional; generic type of config point (CII data type)
isRef: boolean, optional; True if this member is a reference to another class.
13.12.2. Metadata class schema¶
Metadata class schema (field: type, required; description):
classes: array, required; List of metadata classes to be generated.
name: string, required; Name of the metadata class.
parent: string, required; Parent metadata class name.
- __comment__: string, optional; Metadata class comment. This comment ischanged to programming language specific comment onbindings generation.
members: array, required; member fields of metadata classes
name: string, required; metadata member field name.
type: string, required; CII data type of metadata class.
13.12.3. Configuration instance schema¶
Target configuration (configuration instance) schema has the following starting structure:
config: string, required; Root node.
instance: string, required; Definition of config instance.
The starting structure is followed by the schema structure for the details. The details schema is child node of instance (field: type, required/optional; description):
__comment__: string, optional; YAML/JSON comment for the target configuration.
data: Object, required; Data object.
@type: string, required; Configuration class type.
fields: string, required; List of all members that have the same member name and type as defined in the corresponding class definition. Member name and value pairs are added to YAML (“field”: “value”):
@type: string, required; Metadata class type.
metadataInstance: string, required; Metadata instance name.
metadataInstanceVersion: string, optional; Version of metadata instance. Last existing version is used if the version is not specified.
value: string | number | boolean | array, required; the actual value of config point.
@genType: string, optional; generic type used in the instance.
To address the fields of referenced classes, special name rule is used. The name is a concatenation of reference instance name followed by the reference field name, separated by a dot “.” character is used. The same logic is used when one reference class references another class.
Examples:
Configuration class contains referenced adFlowClass. The “adFlow1.sensorsSize” will address the sensorsSize field in the adFlow1 referenced class instance.
Configuration class contains referenced adFlowClass, which references currentFlowClass. The “adFlow1.currFlow1.running” will address the running field in the currFlow1 referenced class instance, which is referenced from adFlow1.
13.12.4. Metadata instances schema¶
Metadata instances schema has the following starting structure:
metadata: string, required; Root node.
instances: string, required; Definition of metadata instance.
The starting structure is followed by the schema structure for the details. The details are child nodes of instances node: (field: type, required; description):
@name: string, required; Name of metadata instance.
@type: string, required; Metadata class of metadata instance.
@genType: string, required; Generic type used by the instance. This type must be the same as the defined metadata class type.
List of all member names that are in line with metadata class. Member name and value pair are added to YAML (“field”: “value”):
field: string, required; metadata class field name.
value: string | number | boolean | array, required; the actual value of metadata field.
13.13. GUI configuration file¶
The configuration file for the Configuration GUI application contains the information needed to connect to the remote configuration service (i.e. the data source) and file storage (i.e. HDFS service). The configuration file contains 4 sections:
DataSource: this section contains all the data needed to connect to the primary configuration service. The hostname attribute provides the address of the service (in a form of an IP address or a host name) and the port attribute provides a port on which the service listens for incoming connections.
DataSourceBackup: this section contains the same parameters as DataSource, only the values are set to the ones of the backup configuration service. In case you don’t have a backup configuration service, insert the values of the primary configuration service into this section.
HdfsService: this section contains only the urlPrefix attribute. This prefix is used when uploading files to the HDFS service. This is a REST endpoint to which Configuration GUI sends files using the HTTP POST calls.
HdfsServiceBackup: this section contains the same parameters as HdfsService, only the values are set to the ones of the backup file storage. In case you don’t have backup file storage, insert the values of the primary file storage into this section.
The following is an example of the GUI configuration file:
# configuration for a config gui
[DataSource]
hostname = ciiconfservicehost
port = 9116
[DataSourceBackup]
hostname = ciiconfservicehost2
port = 9116
[HdfsService]
urlPrefix = http://ciihdfshost:9870/webhdfs/v1/esoLs
[HdfsServiceBackup]
urlPrefix = http://ciihdfshost:9871/webhdfs/v1/esoLs
13.14. Configuration client settings¶
Configuration client API needs some internal setting to work properly. These settings are key/value pairs. Default settings are a part of the client but can be overridden. The following settings are defined:
configLocalDbRoot – root folder where all the local DB JSON files are stored.
configServiceAddress – URI address and port. E.g : zpb.rr://ciiconfservicehost:9116/
configServiceBackupAddress - (Optional) URI address and port. E.g.: zpb.rr://ciiconfservicehost2:9116/
largeStorageAddress - URI address and port. E.g.: http://ciihdfshost:9087
largeStorageBackupAddress - (Optional) URI address and port. E.g.: http://ciihdfshost2:9870
largeStorageEndpointPath – large storage server root path for the data. Default: /esoLs
exceptionWhenRemoteNotAccessible - Exception is thrown by the Config client if the remote DB is not accessible. In case this setting is false, remote DB will be ignored when the “*” is used. Default: false.
connectionTimeout – Timeout for the connection to the remote DB in seconds. Default: 30.** **
Default value of configLocalDbRoot is defined by the following procedure:
If CONFIG_LOCAL_DB environment variable is set, this variable will define the location.
If CONFIG_LOCAL_DB environment variable is not set, it will check for existence of INTROOT environment variable. If this variable is set, then location fallback $INTROOT/localdb.
If none of the variables is set, the default location will be /localdb.
To override the default values, path to config.ini file can be specified using the CONFIG_CLIENT_INI environment variable. The following listing shows a simple example:
configServiceAdress:http://configServer.elt.eso:12345
largeStorageAdress:http://hdfs.elt.eso:9087
13.15. Configuration service settings¶
Configuration service needs some settings to work properly. Settings have predefined default values. The configuration file can be created to override predefined values. The file must contain key/value pair settings needed by the configuration service client. The following settings are defined:
metadata_class_index – default name of the Elasticsearch index, which contains metadata classes. Default value: metadata_class.
metadata_instance_index – default name of the ElasticSearch index, which contains metadata instances. Default value: metadata_instance.
configuration_class_index– default name of the ElasticSearch index, which contains config classes. Default value: configuration_class.
configuration_instance_indices – list of ES indices that contain target configurations. Default value: configuration_instance.
configServiceAddress – address to which the service is bound to. Default: zpb.rr://0.0.0.0:9116/
serviceClientEndpointPath – endpoint URI used by the Config clients. Default: configuration/service/clientApi
serviceControlEndpointPath – endpoint URI used by the GUI. Default: configuration/service/control
config_cache_eviction_in_seconds – Config service internal cache eviction time for Target configurations. Default: 100
metadata_cache_eviction_in_seconds – Config service internal cache eviction time for metadata. Default: 100
max_search_size – the maximum number of documents that can be retrieved from Elasticsearch at once. Default value: 10000.
elastic_URI – address and port to the ElasticSearch service. Default value: http://ciielastichost:9200.
elastic_backup_URI – Optional; address and port to the ElasticSearch backup service. Default value: http://ciielastichost2:9200.
elastic_refresh_policy – Index refresh policy used by the ElasticSearch. Default: None.
largeStorageAddress – address of the large storage server. Default: http://ciihdfshost:9870
largeStorageBackupAddress – backup address of the large storage server. Default: http://ciihdfshost:9870
largeStorageEndpointPath – large storage server root path for the data. Default: /esoLs
The path to configservice.ini file must be set, to override default values. This is done by using the CONFIG_SERVICE_INI environment variable. The following listing shows the example of the configservice.ini file.
metadata_class_index:metadata_class
metadata_instance_index:metadata_instance
configuration_class_index:configuration_class
configuration_instance_index: configuration_instance
max_search_size:10000
elastic_URI: 'http://localhost:9200'
elastic_backup_URI: 'http://localhost:9200'
elastic_refresh_policy: NONE
13.16. Code produced by the generators¶
Listing 7‑1 Java generated code by the config tool
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an
* Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
package elt.config.classes;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectWriter;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.core.JsonProcessingException;
import java.util.List;
import elt.error.CiiInvalidTypeException;
import elt.config.exceptions.CiiConfigException;
import elt.config.CiiConfigClass;
import elt.config.classes.meta.MdInt32;
import elt.config.classes.meta.MdDouble;
import elt.config.classes.meta.MdString;
//Personal config test class
public class MySimpleClass extends CiiConfigClass {
private MdInt32 testIntValue;
private MdDouble testDoubleValue;
private MdString testStringValue;
//default constructor is needed by the ser/deser engine.
public MySimpleClass(){
super();
this.testIntValue = MdInt32.getDefaultMDI();
this.testDoubleValue = MdDouble.getDefaultMDI();
this.testStringValue = MdString.getDefaultMDI();
}
public MySimpleClass(String name,
MdInt32 testIntValue,
MdDouble testDoubleValue,
MdString testStringValue) {
super(name);
this.testIntValue = testIntValue;
this.testDoubleValue = testDoubleValue;
this.testStringValue = testStringValue;
}
public MySimpleClass(String name,
int testIntValue,
double testDoubleValue,
String testStringValue) throws CiiInvalidTypeException {
super(name);
this.testIntValue = new MdInt32(testIntValue);
this.testDoubleValue = new MdDouble(testDoubleValue);
this.testStringValue = new MdString(testStringValue);
}
public MdInt32 getTestIntValue() {
return testIntValue;
}
public void setTestIntValue(MdInt32 testIntValue) {
this.testIntValue = testIntValue;
}
@JsonIgnore
public int getTestIntValueValue() {
return testIntValue.getValue();
}
@JsonIgnore
public void setTestIntValueValue(int testIntValue) throws CiiConfigException {
this.testIntValue.setValue(testIntValue);
}
public MdDouble getTestDoubleValue() {
return testDoubleValue;
}
public void setTestDoubleValue(MdDouble testDoubleValue) {
this.testDoubleValue = testDoubleValue;
}
@JsonIgnore
public double getTestDoubleValueValue() {
return testDoubleValue.getValue();
}
@JsonIgnore
public void setTestDoubleValueValue(double testDoubleValue) throws CiiConfigException {
this.testDoubleValue.setValue(testDoubleValue);
}
public MdString getTestStringValue() {
return testStringValue;
}
public void setTestStringValue(MdString testStringValue) {
this.testStringValue = testStringValue;
}
@JsonIgnore
public String getTestStringValueValue() {
return testStringValue.getValue();
}
@JsonIgnore
public void setTestStringValueValue(String testStringValue) throws CiiConfigException {
this.testStringValue.setValue(testStringValue);
}
public String toString() {
try {
ObjectWriter writer = new ObjectMapper().writer().withDefaultPrettyPrinter();
return writer.writeValueAsString(this);
} catch (JsonProcessingException e) {
e.printStackTrace();
return "";
}
}
}
Listing 7‑2 HPP generated code from by config tool
#ifndef CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
#define CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
#include <ciiNodeContainer.hpp>
#include <ciiNodeFactory.hpp>
#include <string>
#include <vector>
using namespace elt::config;
namespace elt{
namespace config{
namespace classes{
class MySimpleClass : public ::elt::config::CiiConfigClass {
public:
DECLARE_NODE_CLASS();
DECLARE_NODE_FIELD_METADATA(MdInt32, testIntValue);
DECLARE_NODE_FIELD_METADATA(MdDouble, testDoubleValue);
DECLARE_NODE_FIELD_METADATA(MdString, testStringValue);
};
// class MySimpleClass
} //namespace elt
} //namespace config
} //namespace classes
#endif // CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
Listing 7‑3 CPP generated code from the config tool
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
#include <ciiNodeFactory.hpp>
#include "mySimpleClass.hpp"
namespace elt{
namespace config{
namespace classes{
IMPLEMENT_NODE_CLASS(MySimpleClass, MySimpleClass)
} //namespace elt
} //namespace config
} //namespace classes
Listing 7‑4 Python generated code from the config tool
py::class_<MySimpleClass, std::shared_ptr<MySimpleClass>,
::elt::config::CiiConfigClass>(m,"MySimpleClass")
.def_static("getNewNodeInstance", []() {
return ::elt::config::CiiNodeFactory::getNewNodeInstance<MySimpleClass>();
})
.def("get_testIntValue", &MySimpleClass::get_testIntValue)
.def("set_testIntValue", (void (MySimpleClass::*)(const CiiCPIntegerDataClass::data_type_t&))
&MySimpleClass::set_testIntValue)
.def("set_testIntValue", (void (MySimpleClass::*)(const CiiCPIntegerDataClass::data_type_t&,
const std::shared_ptr<CiiCPIntegerDataClass>&))
&MySimpleClass::set_testIntValue)
.def("get_testDoubleValue", &MySimpleClass::get_testDoubleValue)
.def("set_testDoubleValue", (void (MySimpleClass::*)(const CiiCPDoubleDataClass::data_type_t&))
&MySimpleClass::set_testDoubleValue)
.def("set_testDoubleValue", (void (MySimpleClass::*)(const CiiCPDoubleDataClass::data_type_t&,
const std::shared_ptr<CiiCPDoubleDataClass>&))
&MySimpleClass::set_testDoubleValue)
.def("get_testStringValue", &MySimpleClass::get_testStringValue)
.def("set_testStringValue", (void (MySimpleClass::*)(const MdString::data_type_t&))
&MySimpleClass::set_testStringValue)
.def("set_testStringValue", (void (MySimpleClass::*)(const MdString::data_type_t&,
const std::shared_ptr<MdString>&))
&MySimpleClass::set_testStringValue)
;
}
Listing 7‑5 Deployed JSON file from the config-tool
{
"__comment__" : "Configuration for cryo cooler",
"data" : {
"@type" : "MySimpleClass",
"testStringValue" : {
"@type" : "MdString",
"metadataInstance" : "ConfStringStd",
"metadataInstanceVersion" : 1,
"value" : "My string defined value",
"@genType" : "STRING"
},
"testIntValue" : {
"@type" : "MdNumber",
"metadataInstance" : "ConfInt32Std",
"metadataInstanceVersion" : 1,
"@genType" : "INT32",
"value" : 155
},
"testDoubleValue" : {
"@type" : "MdNumber",
"metadataInstance" : "ConfFloatStd",
"metadataInstanceVersion" : 1,
"@genType" : "SINGLE",
"value" : 155.33
}
}
}