13. Internal Config System
Document ID: |
|
Revision: |
1.11 |
Last modification: |
March 18, 2024 |
Status: |
Released |
Repository: |
|
File: |
config.rst |
Project: |
ELT CII |
Owner: |
Marcus Schilling |
Revision |
Date |
Changed/ reviewed |
Section(s) |
Modification |
|---|---|---|---|---|
0.1 |
10.04. 2019 |
bterpinc |
All |
Document creation |
0.2 |
15.04. 2019 |
manovak |
GUI |
GUI part added |
0.9 |
26.4. 2019 |
bterpinc |
All |
Update after internal review, release |
1.0 |
3.6. 2019 |
bterpinc |
6 |
Package support update |
1.1 |
19.7. 2019 |
bterpinc |
3 |
Added sample application section. |
1.2 |
11.9. 2019 |
bterpinc/jp ribosek |
All |
Updated after internal review. Added sample application workflow and search examples. |
1.3 |
25.9. 2019 |
bterpinc |
Appendix C |
Added sample code produced by the generators. |
1.4 |
09.10. 2019 |
bterpinc |
Introductio n 4 |
Added introductio n section Added simple custom metadata sample application |
1.5 |
12.02. 2020 |
bterpinc |
All |
General rewrite of the manual. Sections added and restructure d. |
1.6 |
20.02. 2020 |
kstampar |
All |
Update of cpp file names and added flags when generating python bindings. |
1.7 |
13.07. 2020 |
bterpinc |
Appendix C Appendix G Appendix H 5.4 2.1 |
Removed the MdInt64 duplicate. Added client settings Added service settings Added cache example Additional note on class versioning. |
1.8 |
17.03. 2022 |
mschilli |
Doc Title Overview |
Module is now only for CII-internal usage |
1.9 |
24.08. 2023 |
jrepinc |
7 |
Added YAML config type mapping |
1.10 |
05.09.2023 |
mschilli |
7 |
Removed GUI and out- dated info |
1.11 |
18.03.2024 |
mschilli |
0 |
CII v4: Public doc |
Confidentiality
This document is classified as Public.
Scope
This document is a manual for the internal configuration system of the ELT Core Integration Infrastructure software.
Audience
This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.
References
https://www.eso.org/~eltmgr/CII/latest/manuals/html/docs/oldb.html
https://www.eso.org/~eltmgr/CII/latest/manuals/html/docs/error_api.html
JsonPath, https://github.com/json-path/JsonPath
13.1. Overview
This document is a manual for the CII-internal configuration system. It is used by CII to manage the metadata of the Online database, the capture configurations of the Telemetry service, and others.
13.2. Introduction
The Core Integration Infrastructure (CII) Internal Configuration system provides the mechanisms for distributing configuration data to the CII services software and to allow the collection of configuration data from these systems.
Figure 2‑1 Configuration interactions
Configuration data is stored as JSON documents, and supports three different storage modes:
Cache: Data is stored into client API local memory. This mode is not persistent.
Local DB: Data is stored into local data storage on client computers. This mode is persistent.
Remote DB: Data is stored in the Engineering Archive using the config service. As config service together with Engineering Archive storage provides a complete solution to access and store data to remote storage, it is also referred to as “remote DB”. This mode is persistent.
The configuration is a set of data and metadata. The types and structure of configuration is defined in classes. Metadata classes define the structure of config points and metadata, while configuration classes define the structure of a target configuration (configuration instance). Both configuration and metadata classes support inheritance.
The configuration and metadata classes are based on YAML structures used to generate language-specific classes. The structure of the YAML defines the class names, class member fields, and member types.
Configuration data is composed of metadata instances containing the metadata values and default config values, and target configuration containing config values of the target config points. The configuration instances and metadata instances are JSON data structures. Instances define how values of the actual data-bind to corresponding config and metadata classes.
Configuration instances are instances of the configuration classes which present the actual target configurations. Therefor “target configuration” and “configuration instance” have the same meaning.
Figure 2‑2 Configuration/metadata classes and instances
Figure 2‑2 shows the example of relationship between configuration system (configuration and metadata) classes and instances. The figure shows configuration classes in orange, target configurations in light blue and metadata instances in dark blue colour. The configuration classes MySimpleClass and MySimpleExtendedClass define the member fields (testInt, testIntArray, testString, testDouble) of chosen type.
More detailed description of the system can be found in the sections below.
13.2.1. Classes
Classes only present the structure of the configuration and thus cannot contain any values. Classes, in general, have the following properties:
Classes present the structure of the configuration.
Classes don’t contain any values.
Classes definitions are generated into selected programming language class bindings.
Changing the class structure forces recompilation of the client code.
Classes cannot be versioned, but they can be updated and deleted.
Classes are generated from remote storage or a corresponding local YAML file.
Classes cannot be versioned, but they support inheritance. In the standard workflow, classes cannot be deleted or updated. For defining a new class structure, a new derived class must be created. The new derived class must inherit from the root (configuration or metadata) class which is already a part of CII configuration. This root class has no members, so it doesn’t define any class structure. Members must be added to the derived class.
Note on class versioning: As classes present the structure and are a base for the class bindings, the versioning of the classes would present additional complexity to the configuration. Every change of the class version would mean the recompilation of the code. A part of a good design principle is to have fixed structures defined. Versioning of classes would break this concept, as every version could completely change the structure. As the instances are bound to the structure, this would also mean completely changing the instances under the existing TC URI. As the versioning of the classes is not supported, the structure can be extended by using the inheritance. Inheritance maintains the base structure, but adds the ability to add additional attributes to the configuration.
Deleting and updating the classes can only be used in the process of development and testing.
Figure 2‑3 Configuration and metadata classes relationship
Figure 2‑3 shows relationship diagram of config and metadata classes on basic example. The class MySimpleClass derives from the config base class (CiiConfigClass), which is the root class for all the Cii configuration classes. The MySimpleClass contains 3 member fields, testInt, testIntArray and testString. Field types are defined by metadata classes (MdInt32, MdInt32Array, MdString). For the list of the supported types refer to 2.1.2 and Appendix C.
Additionally, MySimpleExtendedClass derives from the MySimpleClass and adds additional field testDouble which is of type MdDouble. All other fields are inherited from the base class (MySimpleClass).
13.2.1.1. Configuration classes
Configuration classes are structures which define the type and member fields for generated specific language classes. They have the following properties:
Configuration classes present nodes in the config structure.
Configuration class member fields present the configuration data points – config points.
Config points can be metadata classes or references to another configuration class.
Configuration classes cannot contain complex types or list of complex types, the only exception is a reference to another class.
All configuration classes derive from basic class (CiiConfigClass).
All configuration classes must be stored in the elt.config.classes package.
13.2.1.2. Metadata classes
Metadata classes are structures which are used to generate language specific classes that contain metadata values. Metadata classes contain custom metadata fields and a special value field, which holds the actual value of the config point. The value field is added to the metadata classes for simplification and more straightforward usage of Java API. The actual value field is not serialized in metadata instances, but in configuration instances, as the value is not directly part of metadata. Usage of the value field in the metadata classes simplifies typing of metadata and config points, as the types for value, default value and metadata are all contained in one class.
Config points (configuration class member fields) are in most cases metadata classes. The naming convention for this classes follows the rule: Md + chosen name (e.g.: MdNumber). Metadata classes have the following properties:
Metadata classes can only contain a limited set of CiiBaseTypes (CII Basic Data Type Set), which are synced with corresponding language types.
Metadata classes cannot contain complex types.
Metadata member fields cannot contain big arrays. These are limited to the special file field, which only exist in MdArray and all its derivates.
Metadata classes support generic member types, so the type of desired members can be set according to the defined generic type of the class (as Java generics).
All metadata classes derive from basic class (MdBase).
All metadata classes must be stored in the elt.config.classes.meta package.
Set of basic metadata classes is pre-defined. These classes derive directly from MdBase and cannot be deleted or changed as they are part of the configuration system. Basic metadata classes also contain check functions. The check functions are executed when calling the setValue method. The exception is thrown when the value is not properly evaluated by the check function.
MdBase is a root metadata class for all metadata. All other metadata classes must derive from this class. Basic types are defined by metadata classes. The metadata types are shown in Figure 2‑4 and in the table in Appendix C.
Figure 2‑4 Metadata type system
To store array and matrix values, MdArray and MdMatrix derivates must be used.
The data for matrix is represented as a single array, with specified width (member field of the MdMatrix class), so matrix is represented as a continuous array of elements. The width attribute determines at which array element number the array should be cut into row to form a matrix. All matrices are row based; this means that the data is split into rows. When using default values, the width is set to 50.
Binary values are always stored as file references. Like for all the other types of data, the value for binary is set by using the setValue and retrieved by using the getValue method. On setValue method call, the data is written to a file and a reference is stored in the target configuration JSON file. On getValue method call, the file reference is resolved and data from the reference is retrieved. There is no limit specified for the binary data values.
Vector (array) and matrix values are stored as a string up to the limit of 100KB. All values exceeding this limit are stored as file references.
For more complex configurations, custom metadata classes can be defined. The classes can extend existing metadata classes.
The data of the config and metadata classes cannot be manipulated using the config client API. Classes and metadata classes data shall be centrally stored in the remote DB. The config GUI is used to the manipulate the classes on the remote DB. For the development and testing purposes the classes can also be stored on local developer or test machines as YAML definitions.
13.2.2. Instances
Instances are actual value holders for the corresponding configuration and metadata classes. Instances are entities of JSON structure stored in local or remote DB and bind data to corresponding class defined fields. Class and instances member fields must be linked by name, otherwise exception CiiConfigTcBindException is thrown.
13.2.2.1. Metadata instances
Metadata instances define metadata for the config point. Metadata instances have the following properties:
Metadata instances are identified by name. The names must be unique.
Metadata instances can be created, updated and deleted.
They can be stored to cache, local DB or remote DB. The asterisk “*” authority in metadata instances is not supported. The proper storage location for reading the reference directly form the TC is determined by the TC URI authority.
Config point metadata is configured by metadata instance. Configuration instances only hold references to metadata instances. As metadata instances are not directly part of configuration instances, they are stored as separate entities.
Metadata instances can be versioned.
Set of default (predefined) metadata instances is provided (Appendix C). These default metadata instances cannot be saved or updated.
When saveTargetConfigWithMetadata method is used, metadata instances are automatically saved with target configuration. For other types of metadata operations, API contains methods that support operations just with metadata instances. These methods enable saving, updating and deleting metadata instances.
Figure 2‑5 Metadata instances
Figure 2‑5 shows the relationship between the metadata classes and metadata instances. The metadata class MdInt32 which derives from the MdNumber and root metadata class MdBase defines the structure for the metadata instances. Three metadata instances are created based on the MdInt32 class. The ConfInt32Std is a basic predefined metadata instance and has no special values defined. The custom metadata instances ConfCustomInstance1 and ConfCustomInstance2 have custom values set. The values of the class member fields (defaultValue, minValue, maxValue, checked, etc.) can be set for these instances.
13.2.2.2. Configuration instance
Target configuration (configuration instance) contains actual configuration data for the specific configuration class. The configuration instance is identified with URI location. Each URI defines the location from which the configuration instance is obtained. The authority part of URI defines the storage location. The following authorities are supported:
Cache: ‘cii.config://cache/exampleconfig/c1’
Local DB: ‘cii.config://local/exampleconfig/c1’
Remote DB: ‘cii.config://remote/exampleconfig/c1’
All locations: “cii.config://*/exampleconfig/c1”
Note: URI locations that start with number are not allowed. Examples of not valid URIs: cii.config:///exampleconfig/1abc or cii.config:///1exampleconfig/abc
The asterisk “*” authority can be used for retrieving configuration instance. When saving the instances, it is better to provide a specific location. When “*” authority is used, target configuration is obtained in the following order: cache, local DB, remote DB. In case there is no data stored under specified URI in the specific storage location, the location is skipped.
If there is no explicit version specified for retrieving the target configuration, the last existing version stored in the specific storage location will be retrieved. These versions can differ between the storage locations. As the config service can be used also without the presence of the remote DB, an additional setting was added to the client, to control the exception case.
The setting exceptionWhenRemoteNotAccessible can be set. Appendix G contains details on how to apply this setting to the client.
The following rules apply when retrieving the Target configuration with the “*” authority is used:
exceptionWhenRemoteNotAccessible=true: If no remote DB is present – throw an exception.
exceptionWhenRemoteNotAccessible=false: If no remote is present, check local and cache.
When saving or updating the Target configuration using the “*” authority, the versions between the different storage locations can become out of sync. In general, the following rules apply:
saveTC(“*”): save always creates a new version.
updateTC(“*”): always update the last version.
updateTC(specificVersion) – updates specific version. It is always expected for a specific version on location (cache, local, remote) to exist. If it doesn’t exist an exception is thrown.
As the configuration can be used without the remote DB, the following specifics apply:
exceptionWhenRemoteNotAccessible=true:
saveTC(“*”): remote DB is version maintainer. Max version off all the locations is retrieved, using the rule (cache|local|remote) + 1. This version is used to make new cache|remote|local Target configuration. This way versions between the locations are always in sync, but version holes can exist.
updateTC(“*”): remote DB is version maintainer (the last version is retrieved from the remote DB). It is expected that the same version exists in local and cache and that this is the last version (important). If this is not the case, an exception is thrown.
updateTC(specificVersion): All versions are expected to exist. Updates for the specific version to all locations are made (cache|local|remote). If there is no specific version present in any of the DB, an exception is thrown.
- exceptionWhenRemoteNotAccessible=false:If remote DB is present, the behavior is the same as above. If not, the following actions are applied after the timeout:
saveTC(“*”): local DB is version maintainer. We get max version from (cache|local) + 1. We use this version to make a new cache|local TC.
updateTC(“*”): local DB is version maintainer (the last version is retrieved from the local DB). It is expected that the same version exists in cache and local and that this is the last version. If this is not the case, an exception is thrown.
updateTC(specificVersion): All versions are expected to exist. Updates for the specific version to these locations are made (cahce|local). If there is no specific version present in any of the DB, an exception is thrown.
To use the configuration in Local DB mode, YAML definition files for classes and values must be created. The config-tool shall be used to generate language-specific classes and to deploy configuration data into the local DB.
When working with Remote DB, config-gui shall be used to manipulate the data. The remote DB also contains the config and metadata class definitions.
A cache can be used to store the data into memory. From the operations point of view, the cache acts the same as the local DB or remote DB and it is transparent to the client. The major difference to the other storage location is that cache is isolated per process and will not provide any permanent storage. The major benefit of using the cache is the fast storage and retrieval of configuration data. The cache is cleared when the process terminates.
For the configuration instances that use the config point fields of MdArray and MdBinary class, the data is not stored as a part of the JSON, but it is stored to a large storage location. The data to the large storage location is written as raw data in the big-endian order.
13.3. Prerequisites
In order to use the configuration system, the following packages must be built and installed:
MAL, MAL ZPB (elt-mal)
Client APIs (client-APIs)
Config service (srv-config)
Config tools (config-tools)
Config GUI (config-gui)
Config service depends on the ElasticSearch. ElasticSearch server must be accessible to the config service. The default configuration assumes that ElasticSearch is available on http://ciielastichost:9200. Appendix H contains details on how to configure the config service.
For remote large data storage, the Hadoop HDFS system is used. HDFS system must be accessible in order to store large data files. The default configuration assumes that HDFS is accessible on http://ciihdfshost:9870/.
To use the remote database, the remote config service has to be installed and started.
See Error Handling and Configuration Transfer Document [2] for the instructions on how to install and configure the configuration system and its components. The document also contains a description of how to set the ElasticSearch and Hadoop hosts.
13.3.1. Local DB initialization
Local DB stores it’s data into the $INTROOT/localdb folder. To use the default metadata instances used in the examples, the initial predefined metadata instances must be deployed:
config-tool init
13.3.2. Includes/Imports
For basic usage of the Config API the user needs the CiiConfigClient class and all generated binding configuration classes that will be used. If any custom metadata classes are used, the imports of these classes are also needed. Next, the language specific URI class is needed. Java also needs import of all the checked exceptions used. Following sections present the code imports and includes:
13.3.2.1. Java imports
import elt.config.client.CiiConfigClient;
import elt.config.exceptions.CiiConfigInitializationError;
import elt.config.exceptions.CiiConfigNoTcException;
import elt.config.exceptions.CiiConfigWriteDisabledException;
import elt.config.exceptions.CiiConfigSaveException;
import elt.config.exceptions.CiiConfigWriteDisabledException;
import elt.error.CiiInvalidTypeException;
import elt.error.CiiInvalidURIException;
import java.net.URI;
import java.net.URISyntaxException;
13.3.2.2. CPP Includes
#include <ciiDataProviderFactory.hpp>
#include <ciiLocalDataProvider.hpp>
#include <ciiCacheDataProvider.hpp>
#include <ciiRemoteDataProvider.hpp>
#include <ciiConfigApi.hpp>
13.3.2.3. Python imports
import os
import elt.config
13.4. Basic usage
This section presents basic usage examples. The examples in this section can be found in the config-examples project (M*ySimpleClient.java, MySimpleClient.py, mySimpleClient.cpp*). Refer to the config-examples project for details about files, folder structure and the WAF wscript files configuration. The examples project is build using the standard WAF build procedure:
waf distclean configure build install
To run the examples, use the provided executables (run-java, cpp-config-sample-app, python MySimpleClient.py).
The examples show the usage of Config tool (6.1) and Config GUI (7). The Config tool only deploys/undeploys data from YAML files to local DB. Config tool is intended to be used in the development process, where developer can quickly change definitions in YAML files on local machine, while Config GUI will work with remote with all the defined TCs. Config tool cannot be used to deploy data to remote DB. This can however be done with usage of Client API (the data can be read from local DB and stored to remote DB). For this reason, the examples show usage of both Config tools and Config GUI in for the same example.
Note: In case examples are run directly from config-examples project, make sure that all the needed local DB configuration instance files are deployed. The deploy commands are listed in README.md file of the project.
13.4.1. Example description
---
configuration:
configClassesConfiguration:
classes:
- name: MySimpleClass
parent: CiiConfigClass
__comment__: Personal config test class
members:
- name: testIntValue
type: MdInt32
- name: testDoubleValue
type: MdDouble
- name: testStringValue
type: MdString
The types names and language mappings for all types can be found in Appendix A. The syntax of YAML schema is explained in Appendix E. Not all words are allowed for the field names of config or metadata classes. The table in Appendix D contains all the reserved words.
In the example we create a sample application. The application is placed in the root of config-examples project. To prepare the directory structure we create the following subdirectories: java, cpp, cpp-app, cpp-pybindings, python and YAML:
mkdir java
mkdir cpp
mkdir cpp-app
mkdir cpp-pybindings
mkdir python
mkdir yaml
13.4.2. Creating a simple configuration using YAML definitions
In the simple class definition, we defined the structure of the configuration class. We store this definition in YAML file named mySimpleClass.cdt.yaml in the yaml directory. Based on this YAML definition we must first generate the classes used by the application:
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o java/
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o cpp/ -l cpp
config-tool generate -i yaml/mySimpleClass.cdt.yaml -o cpp-pybindings/ -l python -ws
The sample code generated by the config-tool can be found in Appendix I.
Once the class is prepared, we must also define target configuration YAML. The target configuration will use the generated class MySimpleClass and default metadata instances. List of default metadata instances with internal types is listed in Appendix C. The following TC YAML is prepared:
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MySimpleClass
fields:
testStringValue:
"@type": MdString
metadataInstance: ConfStringStd
metadataInstanceVersion: 1
value: 'My string defined value'
testIntValue:
"@type": MdInt32
metadataInstance: ConfInt32Std
metadataInstanceVersion: 1
value: 155
testDoubleValue:
"@type": MdDouble
metadataInstance: ConfDoubleStd
metadataInstanceVersion: 1
value: 155.33
We store the target configuration under the testTC.cdi.yaml file in yaml directory.
Note: Types defined in target configuration must be the same as types defined in the classes in order for the configuration to work properly. The depending on the language the JSON (de)serializers will throw exception with indication that (de)serialization could not be executed. For the types with embedded arrays, the types used in target configuration must have the vector types defined. Please refer to Appendix C for details.
To have the default metadata instances ready, we must first initialize the local DB:
config-tool init
Then we deploy the target configuration to local DB under cii.config://mytest/mysampletc:
config-tool deploy -i yaml/testTC.cdi.yaml -a cii.config://mytest/mysampletc
After the target config is deployed to local DB, we can start using the clients. For successful code builds, proper build dependencies must be set for each of the clients. Please check the config-examples project for details about the WAF dependencies.
Results of deploying testTC.cdi.yaml to local DB (JSON file) can be found in Listing 7‑5.
13.4.2.1. Java client
Listing 4‑1 Simple client Java sample code
public static void main(String[] args) throws Exception {
URI myUri = new URI("cii.config://*/mytest/mysampletc");
//Retrieve config from local or remote DB
try(CiiConfigClient client=CiiConfigClient.getInstance()){
MySimpleClass data = client.retrieveConfig(myUri).getData(MySimpleClass.class);
System.out.println("My double value:" + data.getTestDoubleValue().getValue());
System.out.println("My int value:" + data.getTestIntValue().getValue());
System.out.println("My string value:" + data.getTestStringValue().getValue());
//Store changed data into new TC location -- local DB
client.setWriteEnabled(true);
URI localUri = new URI("cii.config://local/mytest/mysampletc2");
data.getTestDoubleValue().setValue(121442242.224242424);
CiiConfigClient.getInstance().saveTargetConfig(localUri, data);
//Read changed data
MySimpleClass changedData = client. retrieveConfig(localUri).getData(MySimpleClass.class);
System.out.println("My double value:" + changedData.getTestDoubleValue().getValue());
System.out.println("My int value:" + changedData.getTestIntValue().getValue());
System.out.println("My string value:" + changedData.getTestStringValue().getValue());
}
}
}
13.4.2.2. CPP client
Listing 4‑2 contains a sample CPP client code. The code file is named mySimpleClient.cpp and stored in the cpp-app folder. The code follows the same flow as the Java code.
Listing 4‑2 Simple client CPP sample code
int main(int ac, char* av[]) {
::elt::config::classes::MySimpleClass mySimpleClassDataCh;
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MySimpleClass>(
elt::mal::Uri("cii.config://*/mytest/mysampletc"));
std::cout << "My int value:" << retrievedData->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedData->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedData->get_testStringValue() << std::endl;
//Store changed data into new TC location -- local DB
::elt::config::CiiConfigClient::setWriteEnabled(true);
mySimpleClassDataCh.set_testStringValue("Changed string value");
mySimpleClassDataCh.set_testIntValue(retrievedData->get_testIntValue());
mySimpleClassDataCh.set_testDoubleValue(retrievedData->get_testDoubleValue());
int version = elt::config::CiiConfigClient::saveTargetConfig(
elt::mal::Uri("cii.config://local/mytest/mysampletc2"), mySimpleClassDataCh);
//Read changed data
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedData2 =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MySimpleClass>(
elt::mal::Uri("cii.config://local/mytest/mysampletc2"));
std::cout << "My int value:" << retrievedData2->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedData2->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedData2->get_testStringValue() << std::endl;
}
13.4.2.3. Python client
Listing 4‑3 contains a sample Python code. The code file is named mySimpleClient.py and stored in the python folder. The code follows the same flow as the CPP code.
Listing 4‑3 Simple client Python sample code
import os
import elt.config
from MySimpleClassPyB import MySimpleClass
uri = elt.config.Uri("cii.config://*/mytest/mysampletc")
uri2 = elt.config.Uri("cii.config://local/mytest/mysampletc2")
client = elt.config.CiiConfigClient
#Retrieve config configuration
retrieved = client.retrieve_config(uri, -1)
print("My string value:" + retrieved.get_testStringValue())
print("My integer value: {}".format(retrieved.get_testIntValue()))
print("My double value: {}".format(retrieved.get_testDoubleValue()))
#Store changed data into new TC location -- local DB
client.set_write_enabled(True)
retrievedNew = MySimpleClass.get_new_node_instance()
retrievedNew.set_testStringValue("New test value")
retrievedNew.set_testIntValue(retrieved.get_testIntValue())
retrievedNew.set_testDoubleValue(retrieved.get_testDoubleValue())
version = client.save_target_config(uri2, retrievedNew)
retrieved2 = client.retrieve_config(uri2, -1)
print("My string value:" + retrieved2.get_testStringValue())
print("My integer value: {}".format(retrieved2.get_testIntValue()))
print("My double value: {}".format(retrieved2.get_testDoubleValue()))
13.4.3. Creating a simple configuration using GUI
In this section GUI is used to define classes and target configuration on remote DB.
We first start the configuration GUI:
config-gui
Make sure that GUI has write tick-box enabled (Figure 4‑1):
Figure 4‑1 Write enabled tick box
13.4.3.1. Creating a simple class in GUI
Click on the Configuration class tab, and click Add button to add a configuration class (Figure 4‑2). Select the CiiConfigClass to be the parent class, and name the class MySimpleClass.
Figure 4‑2 Adding configuration class in GUI
After the class is created, we add attributes to the class. To add attributes (members), we click on the Add attribute button and fill in the details of every class attribute (Figure 4‑3). We use the same names and types as in the YAML class definition in section 4.1.
Figure 4‑3 Adding class members in GUI
After we added all the attributes (testIntValue, testDoubleValue, testStringValue), definition changes must be confirmed and propagated to remote DB by clicking on the Commit button.
13.4.3.2. Creating a target configuration for simple class in GUI
To create a TC we click on the Target Config tab and then the Add button. We choose the proper Configuration class and define a URI suffix that corresponds to the URI used when deploying to the local DB, for example mytest/mysampletc. (Figure 4‑4)
Figure 4‑4 Creating target configuration
After the target configuration is created, we must refresh the tree by clicking the “Refresh button”. Next, we bind the metadata instances used by the TC to the member fields. To select the metadata instance, we click on the cells (Figure 4‑6) under Meta Data Instance column. For every class member field name, we choose metadata instance (Figure 4‑5). Predefined default metadata instances are used (ConfInt32Std, ConfDoubleStd, ConfStringStd).
Figure 4‑5 Target configuration definition in GUI
Figure 4‑6 Target configuration definition in GUI
After metadata instances are set, we set the values for every member (cells under Value column), and confirm the changes by clicking on the Commit button (Figure 4‑6).
13.4.3.3. Generating classes from the classes defined in GUI
To generate classes defined in GUI (stored in remote DB), the config tool is called without an input argument:
config-tool generate -o java/
config-tool generate -o cpp/ -l cpp
config-tool generate -o cpp-pybindings/ -l python -ws
To read the data with the client, we use the same code as shown in the previous example (Listing 4‑1, Listing 4‑2, Listing 4‑3). To read the data from the remote DB, the “remote” authority must be used. In case “*” authority is used, all data will first be read from the local DB. Only if the data is not present in local DB, the remote DB will be used.
13.4.4. CRUD operations
The following section presents the code for creating, reading, updating and deleting metadata and configuration instances. The examples for all languages follow the same flow, and can be easily understood from the code samples. The examples first show the operations on the metadata instance, followed by operation on target configuration. The TC is saved under the cii.config://remote/mytest/my_test_instance URI. Config API also supports the listing of config points stored under certain path. To get all the children, the getChildren method can be used. Refer to Appendix A for details about the API.
Note: When setting only one attribute of a configuration programmatically, the metadata for the other attributes are not set automatically. It is expected that all the fields of the configuration class are set by user.
The same MySimpleClass as it was used in section 4.2 is also used in the following examples.
13.4.4.1. Java CRUD
Listing 4‑4 Java metadata CRUD example
//Metadata CRUD example
final String mdiName = "MyCustomMetaInstanceName";
MyCustomMetaClass<Integer> customMetaClass = new MyCustomMetaClass<>(mdiName, "This is my custom metadata instance name",
123, false, "m/s", 9);
URI metadataLocal = new URI("cii.config://local");
//Save metadata under local DB with name MyCustomMetaInstanceName
int version2 = client.saveMetadata(metadataLocal, customMetaClass);
//Retrieve and update the same version
MyCustomMetaClass<Integer> customMetaClassChange = client.retrieveMetadata(metadataLocal, mdiName, -1, MyCustomMetaClass.class);
System.out.println("Metadata int:" + customMetaClassChange.getMetaOfInt());
customMetaClassChange.setMetaOfInt(15);
client.updateMetadata(metadataLocal, mdiName, version2, customMetaClassChange);
//delete the metadata
System.out.println("Deleting metadata: " + mdiName + ", version: " + version2);
client.deleteMetadata(metadataLocal, mdiName, version2);
Listing 4‑5 Java target config CRUD example
//TC CRUD example
//Create another instance of MySimple class and store it to remote
MySimpleClass newSimpleInstance = new MySimpleClass();
newSimpleInstance.setTestDoubleValueValue(12.0d);
newSimpleInstance.setTestIntValueValue(100);
newSimpleInstance.setTestStringValueValue("My test string");
URI newInstanceUri = new URI("cii.config://remote/mytest/my_test_instance");
int version1 = client.saveTargetConfig(newInstanceUri, newSimpleInstance);
//Retrieve and update the same version
MySimpleClass newSimpleInstanceChange = client.retrieveConfig(newInstanceUri).getData(MySimpleClass.class);
newSimpleInstanceChange.setTestStringValueValue("My test string changed");
client.updateConfig(newInstanceUri, version1, newSimpleInstanceChange);
System.out.println("My string value:" + newSimpleInstanceChange.getTestStringValue().getValue());
//Delete TC
client.deleteConfig(newInstanceUri, -1);
13.4.4.2. CPP CRUD
Listing 4‑6 CPP metadata CRUD example
//Metadata CRUD example
std::string mdiName("MyCustomMetaInstanceName");
std::shared_ptr<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>> customMetaClass(
elt::config::CiiDataPointMetadataFactory::getNewMetadataInstance<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>>(mdiName));
customMetaClass->set_metaOfBool(false);
customMetaClass->set_metaOfString("m/s");
customMetaClass->set_metaOfInt(9);
//Save metadata under local DB with name MyCustomMetaInstanceName
int version2 = elt::config::CiiConfigClient::saveMetadata("local", *customMetaClass);
//Retrieve and update the same version
std::shared_ptr<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>> customMetaClassChange =
std::dynamic_pointer_cast<elt::config::classes::meta::MyCustomMetaClass<std::int32_t>>(elt::config::CiiConfigClient::retrieveMetadata("local", mdiName));
std::cout << "Metadata int:" << customMetaClassChange->get_metaOfInt() << std::endl;
customMetaClassChange->set_metaOfInt(15);
elt::config::CiiConfigClient::updateMetadata("local", version2, *customMetaClassChange);
//delete the metadata
std::cout << "Deleting metadata: " << mdiName << ", version: " << version2 << std::endl;
elt::config::CiiConfigClient::deleteMetadata("local", mdiName, version2);
Listing 4‑7 CPP target config CRUD example
//TC CRUD example
//Create another instance of MySimple class and store it to remote
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstance =
CiiNodeFactory::getNewNodeInstance<::elt::config::classes::MySimpleClass>();
newSimpleInstance->set_testStringValue("My test string");
newSimpleInstance->set_testIntValue(100);
newSimpleInstance->set_testDoubleValue(12.0);
elt::mal::Uri newInstanceUri("cii.config://remote/mytest/my_test_instance");
int version1 = elt::config::CiiConfigClient::saveTargetConfig(newInstanceUri, *newSimpleInstance);
//Retrieve and update the same version
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstanceChange =
elt::config::CiiConfigClient::getConfigData<::elt::config::classes::MySimpleClass>(newInstanceUri);
newSimpleInstanceChange->set_testStringValue("My test string changed");
elt::config::CiiConfigClient::updateConfig(newInstanceUri, version1, *newSimpleInstanceChange);
std::cout << "My string value:" << newSimpleInstanceChange->get_testStringValue() << std::endl;
//Delete TC
elt::config::CiiConfigClient::deleteConfig(newInstanceUri, -1);
13.4.4.3. Python CRUD
Listing 4‑8 Python metadata CRUD example
#Metadata CRUD example
mdiName = "MyCustomMetaInstanceName"
customMetaClass = MyCustomMetaClassINT32.get_new_metadata_instance(mdiName)
customMetaClass.set_metaOfBool(False);
customMetaClass.set_metaOfString("m/s");
customMetaClass.set_metaOfInt(9);
#Save metadata under local DB with name MyCustomMetaInstanceName
version2 = client.save_metadata("local", customMetaClass);
#Retrieve and update the same version
customMetaClassChange = client.retrieve_metadata("local", mdiName)
customMetaClassChange = MyCustomMetaClassINT32.cast(customMetaClassChange)
#Cannot print as wrong type is returned.
print("Metadata int: %i" % customMetaClassChange.get_metaOfInt())
customMetaClassChange.set_metaOfInt(15)
print("Metadata int changed: %i" % customMetaClassChange.get_metaOfInt())
client.update_metadata("local", version2, customMetaClassChange);
#delete the metadata
print("Deleting metadata: %s, version: %i" % (mdiName, version2))
client.delete_metadata("local", mdiName, version2);
Listing 4‑9 Python target config CRUD example
#TC CRUD example
#Create another instance of MySimple class and store it to remote
newSimpleInstance = MySimpleClass.get_new_node_instance()
newSimpleInstance.set_testStringValue("My test string")
newSimpleInstance.set_testIntValue(100)
newSimpleInstance.set_testDoubleValue(12.0)
newInstanceUri = elt.config.Uri("cii.config://remote/mytest/my_test_instance")
version1 = client.save_target_config(newInstanceUri, newSimpleInstance)
#Retrieve and update the same version
newSimpleInstanceChange = client.retrieve_config(newInstanceUri)
newSimpleInstanceChange.set_testStringValue("My test string changed");
client.update_config(newInstanceUri, version1, newSimpleInstanceChange);
print("My integer value: {}".format(newSimpleInstanceChange.get_testStringValue()))
#Delete TC
client.delete_config(newInstanceUri, -1);
13.4.5. Using the cache operations
The following examples present the usage of cache operations. From the operations point of view, the cache acts the same as the local DB or remote DB and it is transparent to the client. The cache is not permanent storage, as it lives with the process. The cache can be used for inter-process fast storing/retrieval of Target Configurations.
The example first copies Target Configuration from the remote DB (cii.config://remote/mytest/my_test_instance), then it stores it to the cache. Then the values are changed and stored back to same location in cache (cii.config://cache/mytest/my_test_instance). Finally, these values are stored back to the original remote DB location.
In the example, the same path is used for remote DB and cache. If the same path is used, the Target Configuration can be retrieved using the “*” authority. It will first check the cache, then the local and remote DB. The same Configuration class data can be stored to any URI, as there is no limitation for cache to be on the same path as remote and local DB.
13.4.5.1. Java cache example
Listing 4‑10 Java cache example
//The cache workflow -- data in cache only lives as long as the process is alive
//Retrieve the TC from the remote, and store it to cache.
URI newInstanceUri = new
URI("cii.config://remote/mytest/my_test_instance");
URI cacheInstance = new URI("cii.config://cache/mytest/my_test_instance");
client.saveTargetConfig(cacheInstance,
client.retrieveConfig(newInstanceUri).getData(MySimpleClass.class));
MySimpleClass dataFromCache =
client.retrieveConfig(cacheInstance).getData(MySimpleClass.class);
System.out.println("Original cache data: " +
dataFromCache.getTestIntValueValue() + ", " +
dataFromCache.getTestStringValueValue());
//manipulate cache data
dataFromCache.setTestIntValueValue(15);
dataFromCache.setTestStringValueValue("My string value");
System.out.println("Changed cache data: " +
dataFromCache.getTestIntValueValue() + ", " +
dataFromCache.getTestStringValueValue());
//store data into cache
client.saveTargetConfig(cacheInstance, dataFromCache);
//retrieve data form cache and store it to remote.
MySimpleClass dataFromCacheChanged =
client.retrieveConfig(cacheInstance).getData(MySimpleClass.class);
System.out.println("Retrieved changed cache data: " +
dataFromCacheChanged.getTestIntValueValue() + ", " +
dataFromCacheChanged.getTestStringValueValue());
//save the changed data to remote
client.saveTargetConfig(newInstanceUri, dataFromCacheChanged);
//Delete TC
client.deleteConfig(newInstanceUri, -1);
13.4.5.2. CPP Cache
Listing 4‑11 CPP cache example
//The cache workflow -- data in cache only lives as long as the process is alive
//Retrieve the TC from the remote, and store it to cache.
elt::mal::Uri newInstanceUri
("cii.config://remote/mytest/my_test_instance");
elt::mal::Uri cacheInstance("cii.config://cache/mytest/my_test_instance");
//as there are no metadata instances in cache, they must be saved
elt::config::CiiConfigClient::SaveTargetConfig(cacheInstance,
*elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(newInstanceUri));
std::shared_ptr<::elt::config::classes::MySimpleClass> dataFromCache =
elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(cacheInstance);
std::cout << "Original cache data: " << dataFromCache->get_testIntValue()
<< ", " << dataFromCache->get_testStringValue() << std::endl;
//manipulate cache data
dataFromCache->set_testIntValue(15);
dataFromCache->set_testStringValue("My string value");
//store data into cache
elt::config::CiiConfigClient::SaveTargetConfig(cacheInstance, *dataFromCache);
//retrieve data form cache and store it to remote.
std::shared_ptr<::elt::config::classes::MySimpleClass> dataFromCacheChanged =
elt::config::CiiConfigClient::GetConfigData<::elt::config::classes::
MySimpleClass>(cacheInstance);
std::cout << "Changed cache data: " << dataFromCacheChanged->get_testIntValue() << ", " << dataFromCacheChanged->get_testStringValue() << std::endl;
//save the changed data to remote
elt::config::CiiConfigClient::SaveTargetConfig(newInstanceUri, *dataFromCacheChanged);
//Delete TC
elt::config::CiiConfigClient::DeleteConfig(newInstanceUri, -1);
13.4.5.3. Python cache example
Listing 4‑12 Python cache example
#The cache workflow -- data in cache only lives as long as the process is alive
#Retrieve the TC from the remote, and store it to cache.
newInstanceUri = elt.config.Uri("cii.config://remote/mytest/my_test_instance")
cacheInstance = elt.config.Uri("cii.config://cache/mytest/my_test_instance")
client.save_target_config(cacheInstance, client.retrieve_config(newInstanceUri))
dataFromCache = client.retrieve_config(cacheInstance)
print("Original cache data: %i, %s" % (dataFromCache.get_testIntValue(), dataFromCache.get_testStringValue()))
#manipulate cache data
dataFromCache.set_testStringValue("My string value")
dataFromCache.set_testIntValue(15)
#store data into cache
client.save_target_config(cacheInstance, dataFromCache)
#retrieve data form cache and store it to remote.
dataFromCacheChanged = client.retrieve_config(cacheInstance)
print("Changed cache data: %i, %s" % (dataFromCacheChanged.get_testIntValue(), dataFromCacheChanged.get_testStringValue()))
#save the changed data to remote
client.save_target_config(newInstanceUri, dataFromCacheChanged)
#Delete TC
client.delete_config(newInstanceUri, version1)
13.5. Advanced usage
This section presents advanced usage examples. The examples in this section are suffixed with advanced in the config-examples project (M*yAdvancedClient.java, MyAdvancedClient.py, myAdvancedClient.cpp*). The examples project is build using the standard WAF build procedure:
waf distclean configure build install
To run the examples, use the provided executables (run-java-advanced, cpp-config-sample-app-advanced, python MyAdvancedClient.py).
Note: In a case when examples are run directly from config-examples project, make sure that all the needed local DB configuration instance files are deployed. The deploy commands are listed in Readme.md file of the project.
13.5.1. Custom metadata class
In this example we create a custom metadata class, which will be used as a member field in the configuration class. Listing 5‑1 shows the YAML code of the metadata class definition. We name the custom metadata class MyCustomMetaClass and save the metadata class definition as myCustomMetadata.cdt.yaml in the folder named yaml. The class derives from the MdNumber. This class has generic type set to T. This means that the actual type of the configuration class member field can be set at the class configuration class definition.
Listing 5‑1 MyCustomMetaClass definition
---
configuration:
metadataClassesConfiguration:
classes:
- name: MyCustomMetaClass
parent: MdNumber
__comment__: Custom metadata class
genericType: T
members:
- name: metaOfBool
type: BOOLEAN
- name: metaOfString
type: STRING
- name: metaOfInt
type: INT32
The class has 3 metadata fields defined: metaOfBool (type Boolean), metaOfString (type string), metaOfInt (type int32). All other metadata fields that can be used by the metadata instances (minValue, maxValue, checked, defaultValue, …) are already defined by the MdNumber and its parent MdBase.
Next, we create a configuration class, which will use this custom metadata class. We name the class MyClassWithCustomMeta and save the configuration class definition as myClassWithCustomMeta.cdt.yaml in the folder named yaml. The class derives from the parent configuration class CiiConfigClass and has only one Config point defined: intPointWithCustomMeta. As the MyCustomMetaClass definition has the type defined as T, we need to set the generic type of the member field to proper type. We set the intPointWithCustomMeta to be INT32. Listing 5‑2 shows the YAML definition of the class.
Listing 5‑2 MyClassWithCustomMeta definition
---
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithCustomMeta
parent: CiiConfigClass
__comment__: Class with custom metadata
members:
- name: intPointWithCustomMeta
type: MyCustomMetaClass
genericType: INT32
After the configuration classes are defined, we must create YAML files which will define the metadata instance and TC values. Listing 5‑3 shows the YAML definition which defines the values of the custom metadata instance member fields for the MyCustomMetaClass. We name the definition myCustomMDI.mdi.yaml and save it in the yaml folder. The definition sets the values and sets the type. As metadata default value can be of any number type, we must also define the type with @genType directive in the metadata instances definition file.
Listing 5‑3 Definition of custom metadata instance (myCustomMDI)
metadata:
instances:
- "@type": MyCustomMetaClass
"@name": myCustomMDI
"@genType": INT32
comment: "Integer MDI with values"
metaOfBool: "true"
metaOfString: "The string metadata example"
metaOfInt: "12"
Finally, we must define the TC values file. We name this file simpleCustomMetaTC.cdi.yaml and save it in the yaml folder. The file contains the values for linking the metadata instance myCustomMDI to the config point intPointWithCustomMeta. The file also sets the value of the point to 13. Listing 5‑4 shows the file definition.
Listing 5‑4 Definition of target config with custom meta data (simpleCustomMetaTC)
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithCustomMeta
fields:
intPointWithCustomMeta:
"@type": MyCustomMetaClass
"@genType": INT32
metadataInstance: myCustomMDI
metadataInstanceVersion: 1
value: 13
Once all the classes are defined, we must generate the metadata binding classes. We generate the classes for all three programming languages with the config tool:
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o java/
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o cpp/ -l cpp
config-tool generate -i yaml/myCustomMetadata.mdt.yaml -t metadata -o cpp-pybindings/ -l python
Next, we generate the configuration classes with our custom metadata:
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o java/
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o cpp/ -l cpp
config-tool generate -i yaml/myClassWithCustomMeta.cdt.yaml -o cpp-pybindings/ -l python -ws
After the language bindings for the configuration and metadata classes are generated, the produced code files can be built. Please note that config-tool generator will not generate all the WAF wscript files. For the proper build, the wscript files with dependencies have to be added to the java, cpp, and cpp-pybindings directory. The python bindings will assume, that CPP code and python bindings are located in cpp and cpp-pybindings directory respectively. The config tool provides option “-ws” to generate wscript file for generated python binding modules.
To retrieve the data from the local DB, the YAML value files have to be deployed. The following example shows the CLI commands to deploy the metadata instance values and the configuration instance values. The target configuration is deployed under the cii.config://mytest/myCustomMetaTC:
config-tool deploy -i yaml/myCustomMDI.mdi.yaml -t metadata
config-tool deploy -i yaml/simpleCustomMetaTC.cdi.yaml -a cii.config://mytest/my_custom_meta_tc
Listing 5‑5 Java custom metadata class code example
public class mySimpleClient{
public static void main(String[] args) throws Exception {
URI myUri = new URI("cii.config://local/mytest/my_custom_meta_tc");
try(CiiConfigClient client=CiiConfigClient.getInstance()){
MyClassWithCustomMeta data = client.retrieveConfig(myUri).getData(MyClassWithCustomMeta.class);
System.out.println("The value:" + data.getIntPointWithCustomMeta().getValue());
System.out.println("Metadata one:" + data.getIntPointWithCustomMeta().getMetaOfBool());
System.out.println("Metadata two:" + data.getIntPointWithCustomMeta().getMetaOfString());
System.out.println("Metadata three:" + data.getIntPointWithCustomMeta().getMetaOfInt());
}
}
}
Listing 5‑6 CPP custom metadata class code example
int main(int ac, char* av[]) {
::elt::config::classes::MyClassWithCustomMeta MyClassWithCustomMetaCh;
std::shared_ptr<::elt::config::classes::MyClassWithCustomMeta> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MyClassWithCustomMeta>(
elt::mal::Uri("cii.config://local/mytest/my_custom_meta_tc"));
std::cout << "The value:" << retrievedData->get_intPointWithCustomMeta() << std::endl;
std::cout << "Metadata one:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfBool() << std::endl;
std::cout << "Metadata two:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfString() << std::endl;
std::cout << "Metadata three:" << retrievedData->get_metadata_intPointWithCustomMeta()->get_metaOfInt() << std::endl;
}
Listing 5‑7 Python custom metadata class code example
uriCm = elt.config.Uri("cii.config://*/mytest/my_custom_meta_tc")
#Retrieve config configuration
retrievedCm = client.retrieve_config(uriCm, -1)
print("The value: %i" % retrievedCm.get_stringPointWithCustomMeta())
print("The first metadata value : %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfBool())
print("The second metadata value: %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfString())
print("The third metadata value: %s" % retrievedCm.get_metadata_stringPointWithCustomMeta().get_metaOfInt())
13.5.2. Using referenced classes
In this example we create a class that has a field which links a referenced class. This means that the filed will act as a placeholder for all class members of the class that is defined as referenced. The members can be accessed using the “.” dot keyword in the instance YAML definition. The class which we are refereeing to must already exist or be defined in the same YAML definition. The class uses isRef directive to identify the reference type in the configuration class YAML definition.
In the example we name the class MyClassWithRef. The following class definition YAML is prepared (configClassWithRef.cdt.yaml):
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithRef
parent: CiiConfigClass
__comment__: Configuration class with referenced class
members:
- name: device1
type: MySimpleClass
isRef: true
The configuration expects that referenced class, MySimpleClass is already present in the source code. To create the same reference field in GUI, the Generic Type “complex” must be used (Figure 5‑1).
Figure 5‑1 Adding referenced class in GUI
Once the class is prepared, we must define target configuration YAML. The target configuration will use the generated class MyClassWithRef. The following TC YAML is prepared (testRefTc.cdi.yaml):
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithRef
fields:
device1.testStringValue:
"@type": MySimpleClass.MdString
metadataInstance: ConfStringStd
metadataInstanceVersion: 1
value: 'Ref string defined value'
device1.testIntValue:
"@type": MySimpleClass.MdInt32
metadataInstance: ConfInt32Std
metadataInstanceVersion: 1
value: 100
device1.testDoubleValue:
"@type": MySimpleClass.MdDouble
metadataInstance: ConfDoubleStd
metadataInstanceVersion: 1
value: 200.33
When using the referenced classes, target configuration YAML fields and class types must be separated with “.” dot. For example: To set the value of testDobuleValue from the referenced field device1, the device1.testDoubleValue is used for the field value definition. Also, both types must be set, the class type and the metadata type (MySimpleClass.MdDouble).
After the class is generated (chapter 4.2 shows the example), we deploy the target configuration under the cii.config://mytest/myreftc:
config-tool deploy -i yaml/testRefTc.cdi.yaml -a cii.config://mytest/myreftc
The following sections show usage examples of referenced class in all languages.
13.5.2.1. Java referenced type
Listing 5‑8 Java referenced type example
/*** Usage of the referenced class */
CiiConfigClient client=CiiConfigClient.getInstance();
URI myUri = new URI("cii.config://*/mytest/myreftc");
//Retrieve config from local or remote DB
MyClassWithRef data=
client.retrieveConfig(myUri).getData(MyClassWithRef.class);
//Retrieve referenced class
MySimpleClass dataFromRef = data.getDevice1();
System.out.println("My double value:" +
dataFromRef.getTestDoubleValueValue());
System.out.println("My int value:" +
dataFromRef.getTestIntValue().getValue());
System.out.println("My string value:" +
dataFromRef.getTestStringValue().getValue());
13.5.2.2. CPP referenced type
Listing 5‑9 CPP referenced type example
/*** Usage of the referenced class */
std::shared_ptr<::elt::config::classes::MyClassWithRef> retrievedData =
elt::config::CiiConfigClient::getConfigData<
::elt::config::classes::MyClassWithRef>(elt::mal::Uri("cii.config://*/mytest/myreftc"));
std::shared_ptr<::elt::config::classes::MySimpleClass> retrievedDataRef = retrievedData->get_device1();
std::cout << "My int value:" << retrievedDataRef->get_testIntValue() << std::endl;
std::cout << "My double value:" << retrievedDataRef->get_testDoubleValue() << std::endl;
std::cout << "My string value:" << retrievedDataRef->get_testStringValue() << std::endl;
13.5.2.3. Python referenced type
Listing 5‑10 Python referenced type example
# Usage of the referenced class
uri = elt.config.Uri("cii.config://*/mytest/myreftc")
client = elt.config.CiiConfigClient
#Retrieve config configuration
retrieved = client.retrieve_config(uri, -1)
retFromRef = retrieved.get_device1()
print("My string value:" + retFromRef.get_testStringValue())
print("My integer value: {}".format(retFromRef.get_testIntValue()))
print("My double value: {}".format(retFromRef.get_testDoubleValue()))
13.5.3. Using check values
The metadata classes support a check for setting minimum and maximum values on number types. The limits can be set by setting the values of the minLimit (setMinLimit), maxLimit (setMaxLimit) metadata instance member fields. The check is disabled by default. To enable the checks, the flag checked (setChecked) has to be set to true.
The minLimit uses “value >= minLimit” check logic, while maxLimit uses “value < maxLimit” check logic. In case the value exceeds the limit, the CiiConfigLimitOutOfBoundsException exception is thrown. The check is executed when setting the value using the API.
The MdString class also supports allowed values check. This check verifies that the values for config points are part of a set of allowed values. The array of allowed values has to be set. In case the value is not part of the allowed values, the CiiConfigAllowedValuesException is thrown.
The following examples show the usage of min and max limits. The usage of allowed values follows the same principle.
13.5.3.1. Java checked functions
Listing 5‑11 Java checked functions example
/**
* Using the check functions
*/
try {
MdInt32 checkedIntClass = MdInt32.getDefaultMDI();
checkedIntClass.setMinLimit(1);
checkedIntClass.setMaxLimit(200);
checkedIntClass.setChecked(true);
//value in limits
checkedIntClass.setValue(50);
//value out of limits
checkedIntClass.setValue(200);
MySimpleClass testClass = new MySimpleClass();
testClass.setTestIntValue(checkedIntClass);
testClass.setTestDoubleValueValue(12.0d);
testClass.setTestStringValueValue("My test string");
} catch (CiiConfigLimitOutOfBoundsException e) {
System.out.println(e.getMessage());
}
13.5.3.2. CPP checked functions
Listing 5‑12 CPP checked functions example
/**
* Using the check functions
*/
std::shared_ptr<MdNumber<std::int32_t>> checkedIntClass =
std::dynamic_pointer_cast<elt::config::MdNumber<std::int32_t>>(
CiiDataPointMetadataFactory::getMetadataInstance(elt::common::CiiBasicDataType::INT32));
checkedIntClass->set_checked(true);
checkedIntClass->set_min_limit(1);
checkedIntClass->set_max_limit(200);
try{
std::shared_ptr<::elt::config::classes::MySimpleClass> newSimpleInstance =
CiiNodeFactory::getNewNodeInstance<::elt::config::classes::MySimpleClass>();
//value in limits
newSimpleInstance->set_testIntValue(110, checkedIntClass);
//value out of limits
newSimpleInstance->set_testIntValue(201, checkedIntClass);
newSimpleInstance->set_testStringValue("My test string");
newSimpleInstance->set_testDoubleValue(12.0);
}
catch(const elt::config::CiiConfigLimitOutOfBoundsException& e) {
std::cout << e.what() << std::endl;
}
13.5.3.3. Python checked functions
Listing 5‑13 Python checked functions example
#Using the check functions
# this has to be called
checkedIntClass = elt.config.MdInt32.get_new_metadata_instance('')
checkedIntClass.set_checked(True)
checkedIntClass.set_min_limit(1)
checkedIntClass.set_max_limit(200)
try:
newSimpleInstance = MySimpleClass.get_new_node_instance()
newSimpleInstance.set_testStringValue("My test string")
#value in limits
newSimpleInstance.set_testIntValue(10, checkedIntClass)
#value out of limits
newSimpleInstance.set_testIntValue(202, checkedIntClass)
newSimpleInstance.set_testDoubleValue(12.0)
except elt.config.CiiConfigLimitOutOfBoundsException as cix:
print(cix)
13.6. Additional information
13.6.1. Config tool
The configuration tool (config-tool) is used to generate classes and to deploy and undeploy the YAML defined configurations to/from the local DB JSON structure. The tool also supports class generation from remote DB.
The config-tool has the following usage pattern: config-tool <operation> <options>.
To use the predefined simple metadata classes (ConfInt32Std, ConfStringStd, etc.), the classes must be deployed in the local DB. The config-tool init operation supports this. To initialize the local DB use:
config-tool init
For remote DB the config-initES.sh must be run. Please refer to [2] for instructions.
13.6.1.1. Class generation
The tool supports programming language configuration and metadata class files generation. The files are generated from YAML files (optionally JSON). To generate classes the generate option must be used. The following options are supported:
lang: cpp, java, python. Usage: -l <arg>. Default: java
type: config, metadata. Usage: -t <arg>. Default: config
input file. Usage: -i <arg>
output directory. Usage: -o <arg>
input file is in JSON. Usage: -j
generate wscript files for python bindings. Usage: -ws
The argument “o” is mandatory, other arguments are optional. When generating metadata classes, the type “t” must be properly set. To generate the classes from the remote DB, run the tool without “i” argument.
It is not allowed to mix the metadata and configuration classes in one YAML file. As configuration classes change frequently and metadata classes will not change much, the metadata and configuration class generation are separated. The “t” argument forces the user to explicitly know that one is actually generating metadata classes.
The class generation tool has no support for checking the names of existing classes and field names of parent classes. It is up to the user, to properly check the existing fields of the parent classes. However, the tool will check for the reserved words for field names which also contain the field names of the predefined classes. Appendix D contains the list of reserved words.
13.6.1.2. Deployment and undeployment
The tool supports deployment and undeployment of instances files to/from local DB. To deploy the instances the deploy option must be used. The following options are supported:
address: target configuration URI. Option: -a <arg>
input file. Option: -i <arg>
type: config, metadata. Option: -t <arg> Default: config
input file is in JSON. Option: -j
version. Option: -v <arg>
The arguments “a” and “i” are mandatory. When deploying metadata instances the type “t” must be properly set. If no version “-v” is specified the instances are always deployed under version 1.
To undeploy instances, the undeploy option must be used. The following options are supported:
address: target configuration URI. Option: -a <arg>
instancenames: Metadata instance names to be deleted,
separated with “,”. Option: -n <args>
type: config, metadata. Option: -t <arg> Default: config
input file is in JSON. Option: -j
version to be deleted. Option: -v <arg>
The version “v” must be explicitly specified (for safety reasons). To delete all versions, use -1.
The argument “n” must be used with type metadata. In this case the argument “a” is ignored.
To get a list of all available operations, run the:
config-tool -h
To get help on the chosen operation, run config-tool <operation> -h. Example:
config-tool undeploy -h
13.6.2. Generic types
Metadata classes partly support generic type inference. The generic type can be used in all the metadata classes, but the actual type used must be defined in the configuration class definition. There is no limit for the actual type to be defined before in the hierarchy of metadata. When the generic type is fixed, there is no need to define the type in the configuration class definition. Example of these classes are all classes with defined type (MdString, MdInt32). These classes have the actual type defined in the metadata class definition, so the type cannot be defined in the configuration class definition. To set the type of the metadata class to be generic, the T keyword must be used (no other keywords are allowed). The keyword tells the metadata class definition that the actual type will be set by the configuration class. Using the defined generic types in the metadata class definition (e.g.: INT32) will determine the generic type of the class. If the type is fixed in the metadata class hierarchy, the genericType attribute of configuration class must be left out or set to genericType: NONE.
Listing 6‑1 shows example of custom generic metadata class (MyCustomMetaClass) definition in YAML. Listing 6‑2 shows use of the custom generic metadata class in configuration class YAML definition (MyClassWithCustomMeta). The type INT32 is set in the configuration class definition.
Listing 6‑1 Generic type metadata class
configuration:
metadataClassesConfiguration:
classes:
- name: MyCustomMetaClass
parent: MdNumber
__comment__: Custom metadata class
genericType: T
members:
- name: metaOfBool
type: BOOLEAN
- name: metaOfString
type: STRING
- name: metaOfInt
type: INT32
Listing 6‑2 Usage of generic type metadata class
configuration:
configClassesConfiguration:
classes:
- name: MyClassWithCustomMeta
parent: CiiConfigClass
__comment__: Class with custom metadata
members:
- name: stringPointWithCustomMeta
type: MyCustomMetaClass
genericType: INT32
CII Configuration provides an option to have generic type metadata classes and to set the type of the config point metadata class in the configuration class definition. This option can be used for classes, however the instances definition YAML files must have generic type specified for serializers to properly determine the type of data provided.
Types defined in instances must be the same as types defined in the classes in order for configuration to work properly. Listing 6‑3 and Listing 6‑4 show the examples of instances YAML definition files.
Listing 6‑3 Config instance when generic type is used
metadata:
instances:
- "@type": MyCustomMetaClass
"@name": myCustomMDI
"@genType": INT32
comment: "String MDI with values"
metaOfBool: "true"
metaOfString: "The string metadata example"
metaOfInt: "12"
Listing 6‑4 Configuration instances with generic type YAML
config:
instance:
__comment__: Configuration for cryo cooler
data:
"@type": MyClassWithCustomMeta
fields:
stringPointWithCustomMeta:
"@type": MyCustomMetaClass
"@genType": INT32
metadataInstance: myCustomMDI
metadataInstanceVersion: 1
value: 13
13.6.3. Java search example
Listing 6‑5 shows the Java example for configuration search. The same API can also be used for CPP and Python. A different search language is used for local and remote DB:
When searching in local database, a json path expression should be used: https://github.com/json-path/JsonPath
When searching in remote database, an Elasticsearch query DSL should be used: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-syntax
Searching with both languages is shown in the example.
Listing 6‑5 Java search example
// remote search example
CiiConfigClient client = CiiConfigClient.getInstance();
// search or configuration with name 'temperatureSensor'
String configurationName = "temperatureSensor";
final String searchExpression = String.format("data.name:%s", configurationName);
try {
List<CiiTargetConfig> foundConfigurations = client
.searchConfigRepo(ApiSearch.REMOTE, searchExpression);
} catch (CiiConfigNoTcException e) {
// handle exception in case there was no such configuration
} catch (CiiConfigSearchException e) {
// handle exception in case an error occurred while searching for configuration
}
// local search example
CiiConfigClient client = CiiConfigClient.getInstance();
// search locally for configuration with name 'temperatureSensor'
String configurationName = "temperatureSensor";
final String search = String.format("$..[?(@.name == '%s')]", configurationName);
try {
List<CiiTargetConfig> foundConfigurations = client.searchConfigRepo(ApiSearch.LOCAL, search);
} catch (CiiConfigNoTcException e) {
// handle exception in case there was no such configuration
} catch (CiiConfigSearchException e) {
// handle exception in case an error occurred while searching for configuration
}
13.6.4. Large data – binary files
The MdArray and MdBinary types support storing of large data. The CII Configuration will store all binary data (MdBinary) as file references. All the arrays (MdArray) exceeding 100kb (this limit is hard coded) will be stored as binary files and only references to files will be passed to predefined metadata field “file”. Data is stored as a raw data in big-endian order.
Configuration instance YAML definition supports deployment of the raw data files. To properly deploy the file, it must be placed in the subfolder named “files”. The name of the file must be set in the file field of the config point. In Listing 6‑6 the picture1 config point will use the file MyBigData.raw as file reference. When the instance is deployed the file will be copied to local DB or remote large data storage.
Listing 6‑6 Configuration instance field with file
picture1:
"@type": MdBinary
metadataInstance: ConfBinaryStd
metadataInstanceVersion: 1
value:
file: MyBigData.raw
13.6.5. Extending metadata classes with additional logic
Cii Configuration provides built-in check functions (5.3), but has no direct support for extending the metadata classes with additional logic. The only way to add additional check functions is to modify the source code of the CII Configuration. For such an action, deep knowledge of internals is needed. As it is not intended for the end user to modify the source code, here only some short guidelines are provided.
To add additional logic in Java, the logic has to be put into one of the existing metadata classes. The logic has to be programmed in a separate private method, and then added to existing setValue method. The recommended classes for additional logic are MdNumber, MdArray, MdString. The logic method can also be put in MdBase (as protected), but it has to be called in setValue method of the MdNumber, MdArray or MdString classes.
For CPP additional logic has to be added as a private method in the MdBase.hpp file. The custom logic private method then has to be added to the existing validate method. As the method must support different types of an input value, it has to be a templated method.
As the Python bindings to CPP are used, no additional work is needed for the Python.
13.7. Config client API listing
Config client API provides methods for the CRUD operations of configuration and metadata instances. After the listing with methods description, the source code of specific programming language API is provided (A.1,A.2,A.3). Additional methods for searching the configurations are also present. The following methods are provided:
getInstance() |
|---|
This is Java only method that returns instance of Config client API |
saveTargetConfig(uri, rootNode) |
|---|
Saves the configuration class to the specified URI location (cache, local, remote) and returns a version that was assigned to the saved configuration. Used programming language instance of configuration class must be set to rootNode parameter. |
saveTargetConfigWithMetadata(uri, rootNode) |
|---|
Saves the configuration class to the specified URI location (cache, local, remote) and returns a version that was assigned to the saved configuration. The method will also save all the metadata instances that are programmatically created and used in the programming language configuration class instance. Used programming language instance of configuration class must be set to rootNode parameter. |
retrieveConfig(uri) | retrieveConfig(uri, version) |
|---|
Retrieves the chosen version of target configuration from the specified location (determined by the uri). If the version is set to -1, the last target configuration will be retrieved (configuration with the highest version that is stored at the specified location. The overload with no version will return the latest version. |
updateConfig(uri, rootNode) | updateConfig(uri, version, rootNode) |
|---|
Updates target configuration at the specified URI with the data from the Configuration class set to the rootNode parameter. Overload with no version updates the last known configuration version. |
deleteConfig(uri, version) |
|---|
Deletes exact version of target config at defined URI location. If the version is set to -1, all configurations of the given URI will be deleted. |
saveMetadata(host, metadataInstance) |
|---|
Saves new metadata instance into the specified location. Parameter host determines the location under which metadata will be saved to. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
retrieveMetadata(host, instanceName) | retrieveMetadata(host, instanceName, version) |
|---|
Retrieves specific version of the metadata from the chosen location saved under the chosen instanceName. Parameter host determines the location where metadata will be retrieved from. The overload without version will return the last saved version. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
updateMetadata(host, instanceName, version) |
|---|
Updates specific version of the metadata with instanceName, under the chosen host location. Parameter host determines the location in which metadata will be updated. In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
deleteMetadata (host, instanceName, version) |
|---|
Deletes specific version of the metadata with instanceName, under the chosen host location. Parameter host determines the location where metadata will be deleted from In Java the first part of the path that determines the location must be provided (cii.config://cache/, cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“cache”, “local”, “remote”). |
searchConfigRepo (host, query) |
|---|
Searches for target configurations that can be found using the chosen search expression. Depending on the location of the searches different search expressions should be used. For local searches a search expression should be written in JSON path format [3]. When searching in remote database a search expression should be written in elastic search query DSL format [4]. Host: In Java the first part of the path that determines the location must be provided (cii.config://local/ or cii.config://remote/). The CPP and Python API uses string representation of location (“local”, “remote”). |
getChildren (uri) |
|---|
The method returns all the children config point URIs under the certain path. The method will return list of URIs with the config points. |
setWriteEnabled (enabled) | isWriteEnabled() |
|---|
Sets or checks the write enabled state flag. If write is enabled we can save, update and delete target configurations. If write is not enabled we can only retrieve the data. |
13.7.1. Java API code
public static synchronized CiiConfigClient getInstance() throws CiiConfigInitializationError { ... }
public List<CiiConfigStatus> getConfigStatus(URI uri) throws CiiInvalidURIException { ... }
public int saveTargetConfig(URI uri, CiiConfigClass rootNode)
throws CiiConfigSaveException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public int saveTargetConfigWithMetadata(URI uri, CiiConfigClass rootNode)
throws CiiConfigWriteDisabledException, CiiInvalidURIException, CiiConfigSaveException { ... }
public void updateConfig(URI uri, CiiConfigClass rootNode)
throws CiiConfigNoTcException, CiiConfigUpdateException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public void updateConfig(URI uri, int version, CiiConfigClass rootNode)
throws CiiConfigNoTcException, CiiConfigUpdateException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public CiiTargetConfig retrieveConfig(URI uri)
throws CiiConfigNoTcException, CiiInvalidURIException { ... }
public CiiTargetConfig retrieveConfig(URI uri, int version)
throws CiiConfigNoTcException, CiiInvalidURIException { ... }
public void deleteConfig(URI uri, int version)
throws CiiConfigDeleteException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public List<CiiTargetConfig> searchConfigRepo(ApiSearch searchPath, String searchExp)
throws CiiConfigNoTcException, CiiConfigSearchException { ... }
public List<String> remoteIndexSearch(String index, String query)
throws CiiConfigSearchException { ... }
public <T extends MdBase> T retrieveMetadata(URI searchLocation,
String metadataInstanceName, int version, Class<T> clazz)
throws IOException, CiiInvalidURIException { ... }
public <T extends MdBase> int saveMetadata(URI saveLocation, T metadataInstance)
throws CiiConfigSaveException, CiiInvalidURIException, CiiConfigWriteDisabledException { ... }
public void deleteMetadata(URI deleteLocation, String metadataInstanceName, int version)
throws CiiConfigWriteDisabledException, CiiInvalidURIException, CiiConfigDeleteException { ... }
public List<URI> getChildren(URI uri) throws CiiInvalidURIException, CiiConfigSearchException { ... }
public void setWriteEnabled(boolean writeEnabled) { ... }
public boolean isWriteEnabled() { ... }
13.7.2. CPP API code
static int saveTargetConfig(const elt::mal::Uri& uri, const CiiConfigClass& root_node);
static int saveTargetConfigWithMetadata(const elt::mal::Uri& uri, CiiConfigClass& root_node);
static std::list<std::shared_ptr<CiiDataPointMetadataBase>> getTargetConfigMetadata(CiiConfigurationBase& root_node);
static int saveMetadata(const std::string& host, CiiDataPointMetadataBase& metadata);
static std::shared_ptr<CiiConfigClass> retrieveConfig(
const elt::mal::Uri& uri, int version = -1);
static std::shared_ptr<CiiDataPointMetadataBase> retrieveMetadata(
const std::string& host, const std::string& instance_name, int version = -1);
static void updateConfig (const elt::mal::Uri& uri, int version,
const CiiConfigClass& targetConfig);
static void updateMetadata(const std::string& host, const CiiDataPointMetadataBase& metadata);
static void updateMetadata(const std::string& host, int version, const CiiDataPointMetadataBase& metadata);
static void deleteConfig (const elt::mal::Uri& uri, int version = -1);
static void deleteMetadata (const std::string& host, const std::string& instance_name, int version = -1);
template <typename T>
static std::shared_ptr<T> getConfigData(const elt::mal::Uri& uri, int version = -1) {
return std::dynamic_pointer_cast<T>(retrieveConfig(uri, version));
}
template <typename T>
static std::shared_ptr<T> getMetadata(const std::string& host, const std::string& instance_name, int version = -1) {
return std::dynamic_pointer_cast<T>(retrieveMetadata(host, instance_name, version));
}
static const std::list<elt::mal::Uri> getChildren(const elt::mal::Uri& parent_uri);
static const std::list<std::shared_ptr<CiiConfigClass>> searchConfigRepo(const std::string& host, const std::string& query);
static void setWriteEnabled(bool enabled);
static bool isWriteEnabled();
13.7.3. Python bindings code
py::class_<elt::config::CiiConfigClient, std::shared_ptr<elt::config::CiiConfigClient>>(m, "CiiConfigClient",
"Configration Client API")
.def_static("save_target_config", &elt::config::CiiConfigClient::saveTargetConfig,
"Save a new version of configuration class")
.def_static("save_target_config_with_metadata", &elt::config::CiiConfigClient::saveTargetConfigWithMetadata,
"Save configuration node and the metadata object it references")
.def_static("retrieve_config", &elt::config::CiiConfigClient::retrieveConfig,
"Retrieve a configuration object previously saved via saveTargetConfig")
.def_static("retrieve_config",
[](const ::elt::mal::Uri &uri) { return elt::config::CiiConfigClient::retrieveConfig(uri, -1); },
"Retrieve a configuration object previously saved via saveTargetConfig")
.def_static("update_config", &elt::config::CiiConfigClient::updateConfig,
"Overwrite previously saved configuration or save it using a specific version")
.def_static("delete_config", &elt::config::CiiConfigClient::deleteConfig,
"Delete a configuration object")
.def_static("delete_config", [](const elt::mal::Uri &uri) {
elt::config::CiiConfigClient::deleteConfig(uri);
}, "Delete a configuration object")
.def_static("get_config_data", &elt::config::CiiConfigClient::getConfigData<elt::config::CiiConfigClass>,
"Retrieve a configuration object")
.def_static("save_metadata", &elt::config::CiiConfigClient::saveMetadata,
"Save a new version of a metadata instance")
.def_static("update_metadata", &elt::config::CiiConfigClient::updateMetadata,
"Retrieve a metadata object previously saved via saveMetadata()")
.def_static("delete_metadata", &elt::config::CiiConfigClient::deleteMetadata,
"Delete a metadata object")
.def_static("delete_metadata", [](const std::string &host, const std::string& instance_name) {
::elt::config::CiiConfigClient::deleteMetadata(host, instance_name);
}, "Delete a metadata object")
.def_static("retrieve_metadata", &elt::config::CiiConfigClient::retrieveMetadata,
"Retrieve a metadata object previously saved via saveMetadata")
.def_static("retrieve_metadata", [](const std::string &host, const std::string &instance_name) {
return elt::config::CiiConfigClient::retrieveMetadata(host, instance_name);
}, "Retrieve a metadata object previously saved via saveMetaData")
.def_static("search_config_repo", &elt::config::CiiConfigClient::searchConfigRepo,
"Search a data provider for specific configurations")
.def_static("set_write_enabled", &elt::config::CiiConfigClient::setWriteEnabled,
"Enables or disables the possibility of writing into the configuration databases")
.def_static("is_write_enabled", &elt::config::CiiConfigClient::isWriteEnabled,
"Returns True if the API allows writing into configuration databases, false otherwise")
.def_static("get_children", &elt::config::CiiConfigClient::getChildren,
"Query children nodes")
13.8. Config point value type to language types mapping
Below is the mapping table between config point types and data types of all programming languages that the configuration can use.
Table 7‑1 Config point types mapping table
ELT CII BASIC TYPE |
Java |
CPP |
Python |
|
|---|---|---|---|---|
INT32 |
Integer |
std::int32_t |
int |
|
INT64 |
Long |
std::int64_t |
int |
|
UINT8 |
Integer |
std::uint8_t |
int |
|
UINT16 |
Long |
std::uint16_t |
int |
|
UINT32 |
Long |
std::uint32_t |
int |
|
SINGLE |
Float |
float |
float |
|
DOUBLE |
Double |
double |
float |
|
BOOLEAN |
Boolean |
bool |
bool |
|
STRING |
String |
std::string |
str |
|
BINARY |
byte[] |
std::vector<std::uint8_t> |
bytearray |
|
YAML |
elt::config::datatypes:: YamlNode |
elt. config. datatypes. YamlNode |
13.9. Default metadata instance mapping
Table 7‑2 shows the default metadata instance and class type mappings. The columns “metadata instance @genType” and “class genericType” are CII Basic types. These columns are used in YAML class definition for metadata classes and metadata instances. The fields are described in Appendix E.
Table 7‑2 Default metadata mappings
Metadata class name |
Metadata instance @genType |
Class genericType |
Is Array |
Default metadata instance |
|---|---|---|---|---|
MdInt32 |
INT32 |
INT32 |
F |
ConfInt32St d |
MdInt64 |
INT64 |
INT64 |
F |
ConfInt64St d |
MdFloat |
SINGLE |
SINGLE |
F |
ConfFloatSt d |
MdDouble |
DOUBLE |
DOUBLE |
F |
ConfDoubleS td |
MdBool |
BOOLEAN |
BOOLEAN |
F |
ConfBoolStd |
MdString |
STRING |
STRING |
F |
ConfStringS td |
MdStringArr ay |
VECTOR_STRI NG |
STRING |
T |
ConfStringA rrayStd |
MdInt32Arra y |
VECTOR_INT3 2 |
INT32 |
T |
ConfInt32Ar rayStd |
MdFloatArra y |
VECTOR_SING LE |
SINGLE |
T |
ConfFloatAr rayStd |
MdDoubleArr ay |
VECTOR_DOUB LE |
DOUBLE |
T |
ConfDoubleA rrayStd |
MdInt32Matr ix |
MATRIX2D_IN T32 |
INT32 |
T |
ConfInt32Ma trixStd |
MdFloatMatr ix |
MATRIX2D_SI NGLE |
SINGLE |
T |
ConfFloatMa trixStd |
MdDoubleMat rix |
MATRIX2D_DO UBLE |
DOUBLE |
T |
ConfDoubleM atrixStd |
MdBinary |
BINARY |
BINARY |
F |
ConfBinaryS td |
MdYaml |
YAML |
YAML |
F |
ConfYamlStd |
13.10. Class definition reserved words
Table 7‑3 shows list of words that cannot be used as field names in config or metadata class definitions.
Table 7‑3 List of words not allowed for config and metadata class names
Field name |
|---|
name |
uri |
defaultValue |
value |
comment |
ciiType |
genType |
metadataChanged |
minLimit |
maxLimit |
allowedValues |
size |
field |
file |
genericType |
parent |
__comment__ |
isRef |
13.11. YAML Data point type
Inclusion of the support for YAML data point type was requested with ECII-702 which requested such support in OLDB. As OLDB relies on config, YAML data point type support had to be also added to config.
It was requested that OLDB should support data points that should accept Yaml::Node data type defined by yaml-cpp.
Directly mapping Yaml::Node datatype was not feasible as it is unknown Python, therefore intermediate data type elt::config:datatypes::YamlNode was defined in C++.
Datatype internally serialises to yaml document (string). Python binding for elt::config::datatypes::YamlNode is available as elt.config.datatypes.YamlNode.
YamlNode can be initialized in one of the following ways in C++:
#include <datatypes/yamlNode.hpp>
...
// Empty yaml document
auto node = elt::config::datatypes::YamlNode();
// Initialize with valid yaml document source. When argument cannot be
// parsed as valid yaml, elt::config::CiiValue is thrown
node = elt::config::datatypes::YamlNode("[this, is, yaml, list]");
// Initialize with YAML::Node argument
YAML::Node source = ...
node = elt::config::datatypes::YamlNode(source);
// Extract yaml document as string
std::string doc = node.AsString();
// Extract yaml document as YAML::Node
YAML::Node ydoc = node.AsYamlObject();
// Assignment of new value from string
node = "[this, is, another, yaml, list]";
// Assignment of new value from YAML::Node
node = ydoc;
Python bindings replicate and extend C++ functionality:
from elt.config.datatypes import YamlNode
# Empty yaml document
node = YamlNode()
# Initialize yaml document from string containing valid yaml. Again CiiValueError is raised in # case argument does not contain valid yaml.
node = YamlNode('[10, 20, 30, 40]')
# Initialize yaml document from python object
node = YamlNode(['this', 'is', 'python', 'list'])
# Update value with new yaml document source
node.set_source('[90, 20, 30, 40]')
# Update yaml node with new python object
node.set_value([90, 20, 30, 40])
# Extract yaml document source as string
s = node.as_string()
# Extract yaml document as python object
assert(node.as_value(), [90, 20, 30, 40])
Apart from the fact that YamlNode data type is not native and must be initialized in the one of the ways listed above, Config API supports data points that contain this type.
Vectors of YamlNode elements are not supported by Config API.
13.12. JSON/YAML Schema
Schemas are defined for classes and instances. All local and remote DB JSON files must be in line with the defined schemas. Target configuration and instances schemas are used for value bindings. Class schemas are used for class generation. For local and remote DB, YAML structures are translated into JSON structures with nested elements.
The following schema descriptions are indented in the same manner as the actual YAML files. The indentation follows the YAML files indentation structure.
All class schemas start with the starting structure, which is presented only here and not listed in the subchapters. This structure contains the following elements: “field: type, required/optional; description”. The definition is as follows:
configuration: string, required; defines the root for all configuration schemas
configClassesConfiguration or metadataClassesConfiguration string, required; defines the type of configuration schema.
13.12.1. Configuration class schema
Configuration class schema (field: type, required/optional; description):
classes: array, required; List of classes to be generated.
name: string, required; Name of the configuration class.
parent: string, required; Parent configuration class name.
- __comment__: string, optional; Class comment. This comment ischanged to programming language specific comment onbindings generation.
members: array, required; member fields of classes (config points)
name: string, required; config point name.
type: string, required; config point metadata class type.
genericType: string, optional; generic type of config point (CII data type)
isRef: boolean, optional; True if this member is a reference to another class.
13.12.2. Metadata class schema
Metadata class schema (field: type, required; description):
classes: array, required; List of metadata classes to be generated.
name: string, required; Name of the metadata class.
parent: string, required; Parent metadata class name.
- __comment__: string, optional; Metadata class comment. This comment ischanged to programming language specific comment onbindings generation.
members: array, required; member fields of metadata classes
name: string, required; metadata member field name.
type: string, required; CII data type of metadata class.
13.12.3. Configuration instance schema
Target configuration (configuration instance) schema has the following starting structure:
config: string, required; Root node.
instance: string, required; Definition of config instance.
The starting structure is followed by the schema structure for the details. The details schema is child node of instance (field: type, required/optional; description):
__comment__: string, optional; YAML/JSON comment for the target configuration.
data: Object, required; Data object.
@type: string, required; Configuration class type.
fields: string, required; List of all members that have the same member name and type as defined in the corresponding class definition. Member name and value pairs are added to YAML (“field”: “value”):
@type: string, required; Metadata class type.
metadataInstance: string, required; Metadata instance name.
metadataInstanceVersion: string, optional; Version of metadata instance. Last existing version is used if the version is not specified.
value: string | number | boolean | array, required; the actual value of config point.
@genType: string, optional; generic type used in the instance.
To address the fields of referenced classes, special name rule is used. The name is a concatenation of reference instance name followed by the reference field name, separated by a dot “.” character is used. The same logic is used when one reference class references another class.
Examples:
Configuration class contains referenced adFlowClass. The “adFlow1.sensorsSize” will address the sensorsSize field in the adFlow1 referenced class instance.
Configuration class contains referenced adFlowClass, which references currentFlowClass. The “adFlow1.currFlow1.running” will address the running field in the currFlow1 referenced class instance, which is referenced from adFlow1.
13.12.4. Metadata instances schema
Metadata instances schema has the following starting structure:
metadata: string, required; Root node.
instances: string, required; Definition of metadata instance.
The starting structure is followed by the schema structure for the details. The details are child nodes of instances node: (field: type, required; description):
@name: string, required; Name of metadata instance.
@type: string, required; Metadata class of metadata instance.
@genType: string, required; Generic type used by the instance. This type must be the same as the defined metadata class type.
List of all member names that are in line with metadata class. Member name and value pair are added to YAML (“field”: “value”):
field: string, required; metadata class field name.
value: string | number | boolean | array, required; the actual value of metadata field.
13.13. Configuration client settings
Configuration client API needs some internal setting to work properly.
They can be provided through a config file in ini format, which is pointed to by the CONFIG_CLIENT_INI environment variable.
# Example of config client ini file
# Location of config storage (host:port)
# Translates to:
# - ::sw::redis::ConnectionOptions.host
# - ::sw::redis::ConnectionOptions.port
redisAddress localhost:7000
# Timeout before we successfully send request to or receive response from redis.
# 0ms means no timeout, and block until we send or receive successfuly.
# Translates to ::sw::redis::ConnectionOptions.socket_timeout
# connectionTimeout: 15000
# Max number of connections, including both in-use and idle ones.
# Translates to: ::sw::redis::ConnectionPoolOptions.size
# redisConnectionPoolSize: 15
# Max time to wait for a connection. 0ms means client waits forever.
# Translates to: ::sw::redis::ConnectionPoolOptions.wait_timeout
# redisConnectionPoolWaitTimeoutMs: 10000
# Max lifetime of a connection. 0ms means we never expire the connection.
# Translates to: ::sw::redis::ConnectionPoolOptions.connection_lifetime
# redisConnectionPoolConnectionLifetimeMs: 30000
For values that you don’t define, default values will be used.
13.14. Code produced by the generators
Listing 7‑1 Java generated code by the config tool
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an
* Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
package elt.config.classes;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectWriter;
import com.fasterxml.jackson.annotation.JsonIgnore;
import com.fasterxml.jackson.core.JsonProcessingException;
import java.util.List;
import elt.error.CiiInvalidTypeException;
import elt.config.exceptions.CiiConfigException;
import elt.config.CiiConfigClass;
import elt.config.classes.meta.MdInt32;
import elt.config.classes.meta.MdDouble;
import elt.config.classes.meta.MdString;
//Personal config test class
public class MySimpleClass extends CiiConfigClass {
private MdInt32 testIntValue;
private MdDouble testDoubleValue;
private MdString testStringValue;
//default constructor is needed by the ser/deser engine.
public MySimpleClass(){
super();
this.testIntValue = MdInt32.getDefaultMDI();
this.testDoubleValue = MdDouble.getDefaultMDI();
this.testStringValue = MdString.getDefaultMDI();
}
public MySimpleClass(String name,
MdInt32 testIntValue,
MdDouble testDoubleValue,
MdString testStringValue) {
super(name);
this.testIntValue = testIntValue;
this.testDoubleValue = testDoubleValue;
this.testStringValue = testStringValue;
}
public MySimpleClass(String name,
int testIntValue,
double testDoubleValue,
String testStringValue) throws CiiInvalidTypeException {
super(name);
this.testIntValue = new MdInt32(testIntValue);
this.testDoubleValue = new MdDouble(testDoubleValue);
this.testStringValue = new MdString(testStringValue);
}
public MdInt32 getTestIntValue() {
return testIntValue;
}
public void setTestIntValue(MdInt32 testIntValue) {
this.testIntValue = testIntValue;
}
@JsonIgnore
public int getTestIntValueValue() {
return testIntValue.getValue();
}
@JsonIgnore
public void setTestIntValueValue(int testIntValue) throws CiiConfigException {
this.testIntValue.setValue(testIntValue);
}
public MdDouble getTestDoubleValue() {
return testDoubleValue;
}
public void setTestDoubleValue(MdDouble testDoubleValue) {
this.testDoubleValue = testDoubleValue;
}
@JsonIgnore
public double getTestDoubleValueValue() {
return testDoubleValue.getValue();
}
@JsonIgnore
public void setTestDoubleValueValue(double testDoubleValue) throws CiiConfigException {
this.testDoubleValue.setValue(testDoubleValue);
}
public MdString getTestStringValue() {
return testStringValue;
}
public void setTestStringValue(MdString testStringValue) {
this.testStringValue = testStringValue;
}
@JsonIgnore
public String getTestStringValueValue() {
return testStringValue.getValue();
}
@JsonIgnore
public void setTestStringValueValue(String testStringValue) throws CiiConfigException {
this.testStringValue.setValue(testStringValue);
}
public String toString() {
try {
ObjectWriter writer = new ObjectMapper().writer().withDefaultPrettyPrinter();
return writer.writeValueAsString(this);
} catch (JsonProcessingException e) {
e.printStackTrace();
return "";
}
}
}
Listing 7‑2 HPP generated code from by config tool
#ifndef CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
#define CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
#include <ciiNodeContainer.hpp>
#include <ciiNodeFactory.hpp>
#include <string>
#include <vector>
using namespace elt::config;
namespace elt{
namespace config{
namespace classes{
class MySimpleClass : public ::elt::config::CiiConfigClass {
public:
DECLARE_NODE_CLASS();
DECLARE_NODE_FIELD_METADATA(MdInt32, testIntValue);
DECLARE_NODE_FIELD_METADATA(MdDouble, testDoubleValue);
DECLARE_NODE_FIELD_METADATA(MdString, testStringValue);
};
// class MySimpleClass
} //namespace elt
} //namespace config
} //namespace classes
#endif // CII_SVCS_CONFIG_TEST_MYSIMPLECLASS_HPP_
Listing 7‑3 CPP generated code from the config tool
/*************************************************************
* @copyright (c) Copyright ESO 2019 All Rights Reserved ESO (eso.org) is an Intergovernmental
* Organisation, and therefore special legal conditions apply.
*************************************************************/
#include <ciiNodeFactory.hpp>
#include "mySimpleClass.hpp"
namespace elt{
namespace config{
namespace classes{
IMPLEMENT_NODE_CLASS(MySimpleClass, MySimpleClass)
} //namespace elt
} //namespace config
} //namespace classes
Listing 7‑4 Python generated code from the config tool
py::class_<MySimpleClass, std::shared_ptr<MySimpleClass>,
::elt::config::CiiConfigClass>(m,"MySimpleClass")
.def_static("getNewNodeInstance", []() {
return ::elt::config::CiiNodeFactory::getNewNodeInstance<MySimpleClass>();
})
.def("get_testIntValue", &MySimpleClass::get_testIntValue)
.def("set_testIntValue", (void (MySimpleClass::*)(const CiiCPIntegerDataClass::data_type_t&))
&MySimpleClass::set_testIntValue)
.def("set_testIntValue", (void (MySimpleClass::*)(const CiiCPIntegerDataClass::data_type_t&,
const std::shared_ptr<CiiCPIntegerDataClass>&))
&MySimpleClass::set_testIntValue)
.def("get_testDoubleValue", &MySimpleClass::get_testDoubleValue)
.def("set_testDoubleValue", (void (MySimpleClass::*)(const CiiCPDoubleDataClass::data_type_t&))
&MySimpleClass::set_testDoubleValue)
.def("set_testDoubleValue", (void (MySimpleClass::*)(const CiiCPDoubleDataClass::data_type_t&,
const std::shared_ptr<CiiCPDoubleDataClass>&))
&MySimpleClass::set_testDoubleValue)
.def("get_testStringValue", &MySimpleClass::get_testStringValue)
.def("set_testStringValue", (void (MySimpleClass::*)(const MdString::data_type_t&))
&MySimpleClass::set_testStringValue)
.def("set_testStringValue", (void (MySimpleClass::*)(const MdString::data_type_t&,
const std::shared_ptr<MdString>&))
&MySimpleClass::set_testStringValue)
;
}
Listing 7‑5 Deployed JSON file from the config-tool
{
"__comment__" : "Configuration for cryo cooler",
"data" : {
"@type" : "MySimpleClass",
"testStringValue" : {
"@type" : "MdString",
"metadataInstance" : "ConfStringStd",
"metadataInstanceVersion" : 1,
"value" : "My string defined value",
"@genType" : "STRING"
},
"testIntValue" : {
"@type" : "MdNumber",
"metadataInstance" : "ConfInt32Std",
"metadataInstanceVersion" : 1,
"@genType" : "INT32",
"value" : 155
},
"testDoubleValue" : {
"@type" : "MdNumber",
"metadataInstance" : "ConfFloatStd",
"metadataInstanceVersion" : 1,
"@genType" : "SINGLE",
"value" : 155.33
}
}
}