6. Configuration

Document ID:
Revision:	1.1
Last modification:	March 18, 2024
Status:	Released
Repository:	https://gitlab.eso.org/cii/info/cii-docs
File:	config-ng.rst
Project:	ELT CII
Owner:	Marcus Schilling

Document history

Revision	Date	Changed/ Reviewed	Section(s)	Modification
0.8	15.03.2022	Jure Repinc Marcus Schilling	All	Document created / reviewed
0.9	30.10.2023	Jure Repinc	6.4.2	Document cach ing
1.0	06.12.2023	Jure Repinc Marcus Schilling	6.4.2 6.8 6.11	root file provider, list entries
1.1	18.03.2024	Marcus Schilling	0	Public doc

Confidentiality

This document is classified as Public.

Scope

This document is a manual for the Configuration system of the ELT Core Integration Infrastructure software.

Audience

This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.

Glossary of Terms

API	Application Programming Interface
CII	Core Integration Infrastructure
MDI	Metadata instance
YAML	YAML Ain’t Markup Language

6.1. Overview

This is the user manual for the CII Configuration API (Config-ng API). All examples in this manual are also present in the cii-demo repository.

The Config-ng API is provided for C++ and Python. It provides support for parsing YAML documents (with support for additional information or directives via tags) into configuration documents. A Configuration document contains set of named instances with optional data type and metadata information.

A Configuration document allows access to its instances and supports additional operations like saving, merging with another configuration document and updating with programmatically prepared instance map.

6.2. Includes and imports

6.2.1. C++

To use the config-ng API from C++, the following directives are needed:

wscript (project)

cnf.check_wdep(wdep_name='config-ng.cpp.config-ng', uselib_store='config-ng')
# Transitive Dependencies:
cnf.check_cfg(package='log4cplus', uselib_store='log4cplus', args='--cflags --libs')

wscript (module)

use=[...,
 'config-ng'
 # Transitive Dependencies:
 'log4cplus'

source

#include <config-ng/ciiConfigApi.hpp>

6.2.2. Python

To use the config-ng API from Python, the following directives are needed:

wscript (project)

cnf.check_wdep(wdep_name='config-ng.python.config-ng', uselib_store='config-ng')

wscript (module)

use=[..., 'config-ng' ]

source

import elt.configng

6.3. API Components

API provides the following basic components:

• Client

• Document

• Node

• Data type

• Metadata

Client is the entry point to the API. It provides basic operations like loading document from provided filename/URI, retrieval or modification of config-ng search path.

Document represents a configuration object that was either loaded from file/stream or created in memory. Operations like loading produce a new document instance. Document itself offers additional operations (like merging, check, update and save).

Node represents an item (also known as instance) within a configuration object. Generally, it can have a name and can represent a scalar (single value), a list of items or a map (dictionary) of items. Access to nodes is available through the document instance interface desribed later.

Data type describes a data type assigned to the configuration instance (Node) if any. Data type of instance is optional. There is a set of built-in data types. Defining custom data types is supported by using a predefined yaml syntax. A data type definition can contain its own metadata attributes. Some predefined attributes are understood by the config-ng (when running the Check() operation on a document):

default

Provides default value of the configuration item when its value is not explicitly provided.

min

Allowed minimum value of the scalar configuration item.

max

Allowed maximum value of the scalar configuration item.

allowed_values

Provides list of allowed scalar values for scalar configuration item.

Metadata is associated with Node and can be seen as a map of attributes assigned to item. These attributes have a name and can be scalars, lists, or maps. There is no data type assigned to the attributes of the item. Some attributes are automatically generated by the config-ng at the time of the parsing of the initial yaml document. There are length for lists and rows and columns for matrices.

6.4. Creation of Documents

Documents can be created by parsing YAML source either from input/output stream or provided URI. Additionally documents can also be constructed programatically.

6.4.1. Parsing YAML source from input/output stream

Note that parsing document from YAML can cause exceptions either within config-ng or within the underlying YAML parser, therefore proper exception handling is needed (refer to config-ng examples).

6.4.1.1. C++

In C++, YAML source can be provided through a std::istream object as demonstrated below:

#include <iostream>
#include <sstream>
#include <config-ng/ciiConfigApi.hpp>

int main() {
  const char *document_source = R"DOC(
   a: 10
   b: this is a string
  )DOC";
  std::stringstream document_source_stream;
  document_source_stream << document_source;
  ::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load(document_source_stream);
  std::cout << document.Instances()["a"].As<std::int>() << ", "
            << document.Instances()["b"].As<std::string>()
            << std::endl;

  return 0;
}

6.4.1.2. Python

Loading from Python’s standard io objects is supported:

import io
import elt.configng
document_source = """
 a: 10
 b: this is a string
"""
document = elt.configng.CiiConfigClient.load_stream(io.StringIO(document_source))
print(document.instances.a.as_value())
print(document.instances.b.as_value())

6.4.2. Loading YAML source from file/URI

Documents can be loaded from a local file by specifying a URI. The supported URI syntax is cii.config://<authority>/<path> where:

<authority>

can be either local, or root (since CII v4).

<path>

is the path to the file.

The “<authority>” component determines the file search strategy, ie. how the “<path>” component gets interpreted. The available file search strategies are implemented by so-called file providers. They are described next.

6.4.2.1. File-provider: local

The “local” file provider will interpret the path as relative to the search path currently set in the client.

The search path consists of a list of directories to be searched for a specific file. Directories are searched in the order they appear in the list. First match of the filename wins. From a user point of view, the search path is a string containing zero or more directories delimited by : (i.e. "/path/to/dir1:/path/to/dir2:path/to/dir3".

The search path can be set to the empty string. In this case, search for files is limited to only relative to current working directory of the process.

Client initialises the search path from “CFGPATH” environment variable, if defined. To obtain the current search path, it provides method GetSearchPath() in C++ and get_search_path() in Python. When the search path was initialized from “CFGPATH”, the current working directory is prepended to the search path defined in “CFGPATH”.

To modify the search path SetSearchPath(new_path) is provided in C++ and set_search_path(new_path) in Python. The following examples demonstrate manipulation of the search path.

C++ example

// save current search path
std::string saved_search_path = ::elt::configng::CiiConfigClient::GetSearchPath();
std::string my_search_path = "/usr/local/conf:/local/conf";
// establish own search path
::elt::configng::CiiConfigClient::SetSearchPath(my_search_path);
// do something
...
// restore search path
::elt::configng::CiiConfigClient::SetSearchPath(saved_search_path);

Python example

# save current search path
saved_search_path = elt.configng.CiiConfigClient.get_search_path()
# establish own search path
my_search_path = '/usr/local/conf:/local/conf'
elt.configng.CiiConfigClient.set_search_path(my_search_path)
# do something
...
# restore search path
elt.configng.CiiConfigClient.set_search_path(saved_search_path)

6.4.2.2. File-provider: root

The “root” file provider accesses all files relative to its “document_root” option. Default value of “document_root” option is path ‘/’ (i.e. root of file system). The URI “cii.config://root/etc/passwd” therefore accesses the file “/etc/password”. The “root” file provider does not use the search path (CFGPATH env. variable), its mapping of path to file is 1 to 1, there is no additional steps to determine the actual filename.

To set the “document_root” for the “root” file provider:

// C++
::elt::configng::CiiConfigClient::GetFileProviderByName("cii.config://root")->SetOption("document_root", "/absolute/path/to/new/document_root");

# Python
elt.configng.CiiConfigClient.get_file_provider_by_name('cii.config://root').set_option('document_root', '/absolute/path/to/new/document_root')

6.4.2.3. Filenames and relative URIs

As a short-hand alternative to a full URI, it is possible to specify just a path, e.g. “my/path/filename.yaml”.

To convert the path into a URI, the default file provider is used, e.g. “my/path/filename.yaml” –> “cii.config://local/my/path/filename.yaml”.

6.4.2.4. Default file provider

The default file provider is automatically used whenever only a path part of a URI is passed to one of the ::elt::configng::CiiConfigClient::Load or ::elt::configng::CiiConfigDocument::Save methods.

The default setting for the default file provider is local.

To change the default file provider:

// C++
::elt::configng::CiiConfigClient::SetDefaultFileProviderName("cii.config://root");

# Python
elt.configng.CiiConfigClient.set_default_file_provider_name('cii.config://root')

The analogous getter methods are not shown here.

6.4.2.5. Cache setup in C++

Config-ng can cache loaded documents, so that big document reloads can be avoided. The cache is disabled by default and must be enabled explicitly trough CiiConfigClient.

#include <iostream>
#include <config-ng/ciiConfigApi.hpp>
int main() {
  // Obtain cache state
  bool cache_state = ::elt::configng::CiiConfigClient::GetCacheState();
  if (cache_state == false) {
    // Enable cache
    ::elt::configng::CiiConfigClient::SetCacheState(true);
  }
  ...
}

6.4.2.6. Cache setup in Python

import elt.configng

# Obtain cache state
cache_state = elt.configng.CiiConfigClient.get_cache_state()
if not cache_state:
    # Enable cache
    elt.configng.CiiConfigClient.set_cache_state(True)

6.4.3. Actual document loading

C++ Document is loaded by calling method ::elt::configng::CiiConfigClient::Load() as demonstrated below. Note that parsing YAML and loading document can produce exceptions therefore exception handling is needed (refer to examples for more information):

// Load path/file.yaml using search path
::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load("cii.config://local/path/file.yaml")

// Load path/file.yaml using document root
::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load("cii.config://root/path/file.yaml")

In the same manner as C++, Python version provides method elt.configng.CiiConfigClient.load():

# Load path/file.yaml usign current search path
::elt::configng::CiiConfigDocument document = elt.configng.CiiConfigClient.load('cii.config://local/path/file.yaml')

# Load path/file.yaml usign document root
::elt::configng::CiiConfigDocument document = elt.configng.CiiConfigClient.load('cii.config://root/path/file.yaml')

6.4.4. Building document programmatically

To construct a document programatically, a special helper class CiiConfigMapBuilder must be used to construct the ‘map’ of the instances to be added to document. CiiConfigMapBuilder provides a simple interface to add elements to the map. The general workflow is like this:

• Create empty document instance

• Use map builder to prepare an instance map of instances to be added to the document

• Update document with prepared map. With C++, use method `SetInstancesFromMap() on the document instance. With Python, use method set_instances_from_map() on the document instance.

Note that on the API level there are slight differences between Python and C++ API (due of language differences), but the general workflow is the same.

C++ Example

// See programmatic example for more comprehensive information
#include <vector>
#include <string>
#include <iostream>
#include <config-ng/ciiConfigApi.hpp>
// NOTE: additional includes needed
#include <config-ng/ciiConfigMapBuilder.hpp>
#include <config-ng/ciiConfigValueConverter.hpp>

int main() {
   // prepare empty document
   ::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigDocument();
   // Create CiiConfigMapBuilder
   ::elt::configng::CiiConfigMapBuilder builder;
   // Builder provides operators to simplify construction of node hirearchy
   // Note that values cannot directly be assigned to the builder items,
   // factory method ::elt::config::CiiConfigInstanceNode::From() must be used
   // to turn C++ value into correct node to be later assigned to document

   // Create untyped instance a and assign value 3 to it
   builder["a"] = ::elt::configng::CiiConfigInstanceNode::From(3);
   // Create untyped instance b and assign list 1,2,3 to it
   builder["b"] = ::elt::configng::CiiConfigInstanceNode::From(std::vector<int>{1,2,3});
   // Create type instance c, using builtin type vector_int32
   builder["c"] = ::elt::configng::CiiConfigInstanceNode::From(std::vector<int>{1,2,3},
     "!cfg.type:vector_int32");
   // Create map with untyped items a and b
   builder["map"]["a"] = ::elt::configng::CiiConfigInstanceNode::From(3);
   builder["map"]["b"] = ::elt::configng::CiiConfigInstanceNode::From("a string");

   // Once map is constructed, document can be updated
   document.SetInstancesFromMap(builder);
   // Display instance a
   std::cout << document.Instances()["a"].as<int>();
   return 0;
}

Python Example

# See programmatic example for more comprehensive information
import elt.configng
# Prepare empty document
document = elt.configng.CiiConfigDocument()
# Create CiiConfigMapBuilder
builder = elt.configng.CiiConfigMapBuilder()
# Create untyped instance a and assign value 3 to it
builder['a'] = 3
# Create untyped instance b and assign list [1,2,3] to it
builder['b'] = [1,2,3]
# Create typed instance c and assign list [1,2,3] to it.
# Whenever typed instance is required, use of elt.confing.CiiConfigNodeFacory.make_from
# method must be used, as it takes additinal parameters optional tag denoting data type
# to be used when creating instance and optional converter method for custom
# value conversion when required
builder['c'] = elt.configng.CiiConfigNodeFactory.make_from([1,2,3], '!cfg.type:vector_int32')
# Crreate map with untyped items a and b
builder['map']['a'] = 3
builder['map']['b'] = 'a string'
# Once map is constructed, document can be updated
document.set_instances_from_map(builder)
# Display instance a
print(document.instances.a.as_value())

6.5. Access to YAML source tree of a document

Once a document is loaded from YAML stream or file, the original YAML node tree is available to the user trough CiiConfigDocument::GetYamlSourceRootNode() in C++ and CiiConfigDocument.get_yaml_source_root_node() in Python.

To operate on the returned YAML root node in C++, yaml-cpp API must be used.

To operate on the returned YAML root node in Python, PyYAML API must be used.

Note that subsequent changes on the returned YAML node tree do not reflect on the config document, and vice versa, subsequent changes on the config document do not reflect on the YAML node tree. It is recommended that when dealing directly with the YAML node tree, the best course is to save the modified tree as YAML into file or stream, and then load it into another config document. The sample, analogously, applies to changes on the config document: save it to YAML stream/file, and then load it directly with a YAML parser.

6.6. Document instance interface

Once the document is created or loaded, access to its instances is available via the document instance interface. The document instance interface allows access to specific configuration instances. Beside the value, instances contain additional information:

name

Name of the instance (only valid with top level and Map member instances)

type

Three user visible types are predefined: Scalar, Sequence, Map.

metadata

Metadata atrributes of the instance.

data type

Optional config ng data type associated with the instance

origin

Source file location (line, column) of the YAML element from which the instance has been constructed.

Sequence and Map instances provide non recursive (by default) and recursive iterators and methods to access the instance value.

6.6.1. Accessing document instance interface

6.6.1.1. C++

The document instance interface is available through method ::elt::configng::CiiConfigDocument::Instances() method. i.e.:

::elt::configng::CiiConfigDocument document = ::elt::configng.CiiConfigClient::Load("cii.config://local/file.yaml");
// For reading only
const ::elt::configng::CiiConfigInstanceNamespace &instances = document.Instances();

Access to a specific instance is done through operator[]:

// Target type is known, instance value will be automatically
// converted to target type (note exception is thrown if conversion not possible)
double d = instances["theDouble"];
// Target type is not known, conversion method As() must be used
std::cout << instances["theDouble"].As<double>() << std::endl;
// Assign new value to the instance
instances["theDouble"].Assign(6.7);

Obtaining additional information:

::elt::configng::CiiConfigInstanceNamespace &theDouble = document.Instances()["theDouble"];
// Check the type
if (theDouble.IsScalar()) {
   ...
} else if (theDouble.IsSequence()) {
  // Tterate non recursively over instance
  for (const auto &item: instance) {
    ...
  }
  ...
} else if (theDouble.IsMap()) {
  ...
}
// Obtain the origin
::elt::configng::CiiConfigItemOrigin origin = theDouble.GetOrigin();
// Display the origin
std::cout << "Filename: " << origin.source << " line: " << origin.line << " column: "
          << origin.column << std::endl;
// obtain config-ng data type of the instance
::elt::configng::CiiConfigDataType data_type = theDouble.GetDataType();
// access metadata of the instance
::elt::configng::CiiConfigMetadata &metadata = theDouble.GetMetadata();
// and metadata properties
if (metadata.Has("max")) {
   std::cout << "theDouble has max property: " << metadata["max"].As<double>()
             << std::endl;
}

6.6.1.2. Python

The document instance interface is available through method elt.confing.CiiConfigDocument.get_instances(), or through the attribute accessor instances on the document object, i.e.:

document = elt.confing.CiiConfigClient.load('cii.config://local/file.yaml')
# Use attribute accessor instances to access instance 'theDouble'
theDouble = document.instances.theDouble
# Or the classic way
instances = document.get_instances()
theDouble = instances['theDouble']
# Print value of the instance
print(theDouble.get_value())
# Alternative to obtain the value
# is to use as_value() method which takes optional converter argument
# that is actually a function to call when instance value needs to be
# converted to python value
print(theDouble.as_value(lambda x: x.get_value() + 1))
# Assign new value to the instance
theDouble.assign(6.7)

Obtaining additional information:

theDouble = document.instances.theDouble
# Check the type
if theDouble.is_scalar():
   ...
elif theDouble.is_sequence():
  # Iterate non recursively over instance
  for item in theDouble:
      print(item.get_value())
elif theDouble.is_map():
  ...
# Obtain origin of the instance
origin = theDouble.get_origin()
# Display the origin, get_info() method returns tuple (filename, line, column)
# indicating origin of the instance
print('Origin filename %s, line: %s, column: %s' % origin.get_info())
# Obtain config-ng data type of the instance
data_type = theDouble.get_data_type()
# When data type is not assigned to the instance, None is returned
if data_type is None:
   print('Instance theDouble is untyped')
else:
   print('Data type of the theDouble instance is ', data_type.get_name())
# Access metadata of the instance
metadata = theDouble.get_metadata()
# And metadata properties ...
if metadata.has('max'):
   print('theDouble has max property: ', metadata['max'].get_value())

6.6.2. Iteration over document instances

The object returned by the document instance interface offers a recursive iterator over all document instances by default.

6.6.2.1. C++

const ::elt::configng::CiiConfigInstanceNamespace &instances = document.Instances();
for (const auto &[name, node]: instances) {
    std::cout << "Name: " << name << ", Origin: " << node.GetOrigin() << std::cerr;
    // Obtain associated metadata
    ::elt::configng::CiiConfigMetadata &metadata = node.GetMetadata();
    // Obtain data type of the instance node from metadata
    ::elt::configng::CiiConfigDataType &data_type = metadata.GetDataType();
    std::cout << " ... data type: " << data_type.GetName()
              << " originating: " << data_type.GetOrigin() << std::endl;
    std::cout << " ... basic data type: "
              << elt::configng::util::ToString(data_type.GetBasicDataType()) << std::endl;
   if (data_type.IsVector() || data_type.IsMatrix()) {
      std::cout <<  ".... element data type: " << data_type.GetElementDataType().GetName();
   }
 }

6.6.2.2. Python

document = elt.configng.CiiConfigLoad('filename.yaml')
for name, item in document.instances:
    # Data type can be None, indicating that config-ng data type is not assigned
    # to the instance
    data_type = item.get_data_type()
    print(name, ' Data type: ', data_type)
    print(name, ' Originating: ', item.get_origin()

    if data_type is not None:
        print('Basic data type: ', data_type.get_basic_data_type())
        if data_type.is_matrix() or item.data_type.is_sequence():
            print('..... element data type: ', data_type.get_element_data_type())

6.7. Document Validation

Metadata that is produced for each instance node is initially not validated, therefore it might not reflect the actual state of document.

To check the document validity against the schema (defined by elements with CII custom tags) and validity of the metadata, one must invoke check operation on the document. This validates the schema and all values of the document instances.

The Check operation returns an object describing the issues detected during document validation. In case this object does not contain any issues with error severity, the document is valid. This implies that the document is valid when only issues with warning severity were detected.

It is recommended to perform document validation after document loading or any operation that updates document like merge with another document or update with a programmaticaly built map of instances.

6.7.1. C++

The document validation operation is invoked by calling ::elt::configng::CiiConfigDocument::Check() on the document instance.

// Load document
::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load("cii.config://local/file.yaml");
// Perform document validation
::elt::configng::CiiConfigDocumentIssues issues = document.Check();
// The following test returns true in case there was at least one issue with error severity
// was detected
if (issues) {
  std::cerr << "Document is not valid! The following issues were detected: " << std::endl;
  // One can iterate over issues. Each issue is instance of::elt::configng::CiiConfigDocumentIssue
  // class.
  for (auto &issue: issues) {
     // Print out the issue
     std::cerr << " " << issue << std::endl;
     // Or examine it
     if (issue.IsError()) {
        std::cerr << "This is issue with code " << static_cast<int>(issue.GetCode())
                  << " and message: " << issue.GetMessage() << std::endl;
     }

} else {
  // Document is valid at this point, but issues with warning severity can still be present
  std::cout << "Document is valid." << std::endl;
  if (issues.HasWarnings()) {
      std::cout << "The following warnings were issued during validation: " << std::endl;
      for (auto &issue: issues) {
        std::cout << " " << issue << std::endl;
   }
}

6.7.2. Python

Calling method check() on an elt.configng.CiiConfigDocument instance invokes the document validation operation.

# Load document
document = elt.configng.CiiConfigClient.load('cii.config://local/file.yaml')
# Perform document validation
issues = document.check()
if issues.has_errors():
    print('Document is not valid! The following issues were detected:')
    for issue in issues:
        # Print the issue
        print(' ', issue)
        # Or examine it
        if issue.is_error():
        print('This is issue with code ', issue.get_code(), ' with message: ',
            issue.get_message())
else:
    # Document is valid, but issues with warning severity can still be present
    print('Document is valid.')
    if issues.has_warnings():
        print('The following warnings were issued during validation:')
        for issue in issues:
            print(' ', issue)

6.8. Saving documents to YAML

A document can be emitted in YAML format to local files or streams. It is recommended that the documennt is validated before emitting it in YAML format. The Save operation supports additional options that can prevent overwriting existing file or inlining includes (meaning not generating !cfg.include directives in the emitted YAML document).

Tip: Saving to an output stream also provides an easy way of printing documents to the terminal, e.g. document.save (sys.stdout) will dump the full document to the console including formatting.

As of CII v4, ::elt::configng::CiiConfigDocument::Save (elt.configng.CiiConfigDocument.save) method also supports a URI as an argument. If a filename is specified, the default file provider is used to save the file (RootFileProvider in path relative to its “document_root”, LocalFileProvider relative to current working directory (absolute paths are forbidden with LocalFileProvider)).

6.8.1. C++

// Load document
::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load('cii.config://local/file.yaml');
// Validate document
::elt::configng::CiiConfigDocumentIssues issues = document.Check();
// Only save valid document
if (!issues) {
    // Save to file. Existing files are overwriten by default. Includes are not inlined
    // Note that argument here is acutally file name not URI.
    document.Save('file1.yaml');
    // Save to file but prevent overwriting existing file.
    // In case file already exists, ::elt::configng::CiiConfigExistError is thrown
    try {
        document.Save('file1.yaml,
            {::elt::configng::CiiConfigDocument::SaveOptions::NO_OVERWRITE});
    } catch (const ::elt::configng::CiiConfigExistsError &e) {
        std::cerr << "Could not overwrite the file (" << e.what() << ")" << std::endl;
    }
    // Save to stream, inline includes
    std::ostringstream stream;
    document.Save(stream,
        {::elt::configng::CiiConfigDocument::SaveOptions::INLINE_INCLUDES});
    // Save to stream, generate !cfg.include directives for included files
    std::ostringstream another_stream;
    document.Save(another_stream);
}

6.8.2. Python

# Load document
document = elt.configng.CiiConfigClient.load('cii.config://local/file.yaml');
# Validate document
issues = document.check()
# Only save valid document
if not issues.has_errors():
    # Save to file. Existing files are overwritten by default. Includes are not inlined.
    # Note that argument here is actually file name not URI.
    document.save('file1.yaml')
    # Save to file but prevent overwriting existing file.
    # In case file already exists, elt.configng.CiiConfigExistsError is raised.
    try:
        document.save('file1.yaml',
                      (elt.configng.CiiConfigDocument.SaveOptions.NO_OVERWRITE,))
    except elt.configng.CiiConfigEistsError as e:
       print('Could not overwrite the file %s' % e)
    # Save to stream, inline includes
    stream = io.StringIO()
    document.save(stream,
                  (elt.configng.CiiConfigDocument.SaveOptions.INLINE_INCLUDES,))
    # Save to stream, generate !cfg.include directives for included files
    another_stream = io.StringIO()
    document.save(another_stream)

6.9. Merging documents

One can request a merge of two documents via the merge operation. This operation modifies the target document in place.

A merge operation is by definition not commutative, i.e. replacing the merge target with the merge source will result in possibly a different outcome, unless, both the target and the source have no common parts or unless all of the common parts are of the same type and value.

In short, data types/values that are present in the source but not present in the target will be added to the target. Values of the same type that are present in the source as well as in the target will be overwritten in the target, assuming values from the source. Values of incompatible types present in the source as well as in the target will lead to merge issues.

The Merge operation does not modify target document in case it detects any issue that would prevent the merge from fully succeeding, unless explicitly permitted by the user.

It is recommended to validate both documents before attempting to execute merge operation. For clarity, validations are not performed in the following examples.

6.9.1. C++

// Load target document.
::elt::configng::CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load("cii.config://local/document.yaml");
// Load document to merge
::elt::configng::CiiConfigDocument document_to_merge = ::elt::configng::CiiConfigClient::Load(
  "cii.config://local/document_to_merge.yaml");
// Try merge operation. List of issues that prevented merge is returned. In case returned
// list is empty, merge succedded.
std::vector<::elt::configng::CiiConfigDocument::MergeIssue> merge_issues = document.Merge(document_to_merge);
if (merge_issues.empty()) {
   std::cout << "Merge succeeded" << std::endl;
} else {
   std::cerr << "Merge failed, the following issues were detected:" << std::endl;
   for (const auto &issue: merge_issues) {
      std::cerr << " " << issue << std::endl;
   }
   // Target document was not updated.
   // Perform partial merge, only update those instances that can be safely merged.
   // Partial merge is requested by supplying additinal boolean argument with true value.
   // Merge issues for instances that could not be merged are still returned
   merge_issues = document.Merge(document_to_merge, true)
}

6.9.2. Python

# Load target document
document = elt.configng.CiiConfigClient.load('cii.config://local/document.yaml')
# Load document to merge
document_to_merge = elt.configng.CiiConfigClient.load(
    'cii.config://local/document_to_merge.yaml')
# Try merge operation. List of issues that prevented merge is returned. In case returned
# list is empty, merge succeeded.
merge_issues = document.merge(document_to_merge)
if not merge_issues:
    print('Merge succeeded')
else:
    print('Merge failed, the following issues were detected:')
    for issue in merge_issues:
        print(' ', issue)
    # Target document was not updated.
    # Perform partial merge, only update thos unstances that can be safely merged.
    # Partial merge is requested by supplying additional bool argument with True value.
    # Merge issues for instances that could not be merged are still returned
    merge_issues = document.merge(document_to_merge, True)

6.10. Cloning documents

As of CII v4, Config document provides a method to clone (deep copy) an existing document.

6.10.1. C++

CiiConfigDocument document = ::elt::configng::CiiConfigClient::Load(document_source_stream);
CiiConfigDocument cloned_document = document.Clone();

6.10.2. Python

document = elt.configng.CiiConfigClient.load_stream(io.StringIO(document_source))
cloned_document = document.clone()
# or:
import copy
cloned_document = copy.deepcopy(document)

6.11. Listing documents

As of CII v4, Config client provides a method to list existing documents.

Note: this method is only supported with RootFileProvider as there is no meaningful implementation for LocalFileProvider.

6.11.1. C++

::elt::configng::CiiConfigClient::ListDirectoryEntries(const std::string &uri)

6.11.2. Python

elt.configng.CiiConfigClient.list_directory_entries(uri: str)

The method returns list of pairs (python: tuple) each containing URI of the entry and boolean indicator whether that entry is a directory (true/false, python: True/False).

6.12. Exceptions

Config-ng API calls can produce a number of exceptions. For details please refer to the API docs.

All exceptions generated by the config-ng API have common basic class (::elt::configng::CiiConfigError in C++ and elt.configng.CiiConfigError in Python).

List of config-ng exceptions that are defined in Python and C++:

CiiConfigError

Base clas of config-ng exceptions.

CiiConfigBuildError

Generated during process of building configuration document when major inconsistency is detected.

CiiConfigTypeError

Indicates that argument with wrong data type was used.

CiiConfigValueError

Indicates that argument with wrong value was used.

CiiConfigNotFoundError

Item was not found

CiiConfigExistsError

Item or file already exists.

CiiConfigNotImplementedError

Feature was not implemented yet.

CiiConfigIterationError

Inconsistency detected during iteration.

CiiConfigIllegalUriError

Illegal URI was used.

An additional exception is defined in C++ implementation:

CiiConfigDocumentNotFromFileError

Attempted to save document that was not loaded from file without providing file name.

6.13. YAML tags recognized by config-ng

Config-ng uses several specific tags in different contest. All tags recognized by config-ng have prefix cfg.

6.13.1. cfg.include

Request inclusion of another YAML file.

Syntax: !cfg.include <URI>: [NOERROR]

Where <URI> is the URI or the path of the file to include in the document. If <path> is used, the file is looked up according to the rules of the default file provider.

Optional NOERROR flag indicates that in case the file is not found, the operation should continue. Without this flag, an exception is thrown when the requested file could not be found.

Example:

# Include basic definitions
!cfg.include cii.config://local/basic_definitions.yaml:
# Include additional extensions, but do not fail if the file was not found
!cfg.include cii.config://local/extensions: NOERROR

6.13.2. cfg.type

Reference data type.

Syntax: !cfg.type:<TYPENAME>

<TYPENAME> is the name of a built-in data type or user-defined data type.

Example:

# Untyped instance
a: 10
# Typed instance, in this case b is of built in type int32
b: !cfg.type:int32 10

6.13.3. cfg.typedef

Define type alias or custom data type.

Syntax: !cfg.typedef <TYPENAME>[(<BASE>)]

<TYPENAME> is the name of the new data type. <BASE> is optional and must be the name of an existing built-in or user-defined data type. When specified, config-ng derives a new data type from the provided base data type.

Example:

# Type alias
!cfg.typedef myint(int32):

# Type alias with additional metadata
!cfg.typedef extint(int32):
    min: 10
    max: 100
    default: 25

# User defined data type (like struct)
!cfg.typedef Point:
    x: !cfg.type:double
    y: !cfg.type:double


# Derived, inherits the members of the basic data type,
# in this case x and y. Adds new member z.
!cfg.typedef Point3D(Point):
    z: !cfg.type:double

6.13.4. cfg.optional, cfg.required

Modifiers that can be used within user-defined data type member definition.

cfg.optional indicates that the value of this member needs not be present when initializing an instance of that data type.

cfg.required indicates that the value of this member must always be present when initializing an instance of this data type.

When neither of these modifiers are used, the value of the member is initialized from the default value, if not explicitly specified.

Example:

!cfg.typedef Point:
    # x must be always specified
    !cfg.required x: !cfg.type:double
    # y is initialized from default not specified
    y: !cfg.type:double
    # z is optional
    !cfg.optional z: !cfg.type:double

valid_point_1: !cfg.type:Point { x: 10 }
valid_point_2: !cfg.type:Point { x: 10, y: 55 }
valid_point_3: !cfg.type:Point { x: 10, z: 33 }
# The following instance is invalid, x is missing
invalid_point: !cfg.type:Point { y: 3, z: 3 }

6.14. List of built-in data types

List of built-in data types recognized by config-ng:

• int8

• uint8

• int16

• uint16

• int32

• uint32

• int64

• uint64

• single

• double

• boolean

• string

• binary

• vector_uint8

• vector_int8

• vector_uint16

• vector_int16

• vector_uint32

• vector_int32

• vector_uint64

• vector_int64

• vector_single

• vector_double

• vector_boolean

• vector_string

• matrix2d_uint8

• matrix2d_int8

• matrix2d_uint16

• matrix2d_int16

• matrix2d_uint32

• matrix2d_int32

• matrix2d_uint64

• matrix2d_int64

• matrix2d_single

• matrix2d_double