7. Online Database

Document ID:
Revision:	1.14
Last modification:	March 18, 2024
Status:	Released
Repository:	https://gitlab.eso.org/cii/info/cii-docs
File:	oldb.rst
Project:	ELT CII
Owner:	Marcus Schilling

Document History

Revision	Date	Changed/ reviewed	Section(s)	Modification
0.1	05.07.2019	jpribosek, manovak	5	Document creation. GUI Section.
0.2	19.07.2019	jpribosek, jstrnisa	1-4 Appendix A Appendix B	Added client API sections.
0.3	05.08.2019	mvitorovic bterpinc	All	Document review and fixes.
1.0	24.09.2019	jpribosek	All	Final updates, release.
1.1	20.1.2020	jstrnisa	All	Major update of all sections after ESO topical review
1.2	13.7.2020	bterpinc	3.1 4.1 5.6	Statement about pub/sub queue adjustment added. Note about the synchronous methods added. Additional External Redis serves explanation added.
1.3	16.7.2020	bterpinc	7	CLI tools update
1.4	04.4.2022	mschilli	7	Updated cpp py examples
1.5	09.5.2022	dkumar	All	HDFS to Remo te FS, disab le formula c heck when wr iting
1.6	05.07.2023	mschilli	7.6.2.2	Details on timestamp resolution
1.7	21.08.2023	mschilli	7.3 - 7.6	Updates for CII v4 OLDB (LFS,CE)
1.8	24.08.2023	jrepinc	7	Added chapter describing YAML data point type.
1.9	25.08.2023	jrepinc	7.4.1.6	Described Python subscription caveat with GIL (ECII-77 9)
1.10	28.08.2023	mschilli	7.6.2.3.2	Syntax for references in formulas has changed
1.11	25.10.2023	jrepinc	7.5.5 7.5.8 7.6.2.3.4	Data point configuratio n, data poin t aging, isSubscripti onNotificati only,creatin g data point s with isSub scriptionNot ificationOnl flag
1.12	17.11.2023	dkumar		Subscribe method NOTE added
1.13	28.02.2024	mschilli	4, 7	datapoint aliases ; oldb-cli: short uris, create, delete
1.14	18.03.2024	mschilli	0	Public doc

Confidentiality

This document is classified as Public.

Scope

This document is manual for the Online Database system of the ELT Core Integration Infrastructure software.

Audience

This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.

Glossary of Terms

API	Application Programming Interface
CII	Core Integration Infrastructure
CLI	Command Line Interface
DP	Data Point
GUI	Graphical User Interface
ES	ElasticSearch
JSON	Javascript object notation
OLDB	Online Database
SVN	Subversion
YAML	YAML Ain’t Markup Language

References

ELT Software Development Homepage https://www.eso.org/projects/elt/develop
CII User Manuals - Service Management: https://www.eso.org/~eltmgr/CII/latest/manuals/html/docs/services.html
CII User Manuals - Internal Configuration System: https://www.eso.org/~eltmgr/CII/latest/manuals/html/docs/config.html
Redis main page: https://redis.io/
Redis INFO command https://redis.io/commands/info

7.1. Overview

This document is a user manual for usage of CII OLDB system. It explains how to use the OLDB Client API through Client API library and GUI Application to interact with the CII Online Database.

All examples in this manual will be also presented in SVN code (project oldb-examples).

7.2. Introduction

The Core Integration Infrastructure (CII) Online Database (OLDB) provides distributed data publishing and access to actual or live data for user interface and control applications that do not have low-latency or real-time performance requirements. The term “online” refers to the fact that the database provides current and live values for the data points of the control system.

Figure 2‑1: OLDB API interactions

A data point represents a single value in the OLDB and consists of a value, timestamp, quality, and metadata. Data points are identified by a unique URI (section 6.2.1).

Data point value is the actual data the data point is holding. It can be any of the predefined types (Described in Appendix A). The type of the data point can be a primitive type (e.g. integer, double, string) or a more complex type (vector or 2D matrix).

The timestamp is the timestamp of the last data point value modification.

The quality represents the state of the data point. It can be OK, SUSPECT and BAD. Only OK data points can be read and written to (except when correcting the quality to OK). The meaning of each quality state is as follows:

OK: the value of the Property is valid - this is the default quality for any new data point that is created with a provided value.

BAD: the value is not OK and should not be used - this is the default quality for any new data point that is created without a value (i.e. when a default value is used).

SUSPECT: there is a value, but it is not clear if it is reliable. For example, the readout of a sensor is inconsistent with other readings and the sensor might be faulty.

Metadata is the meta information about the data point and consists of information like DP data type, formula, quality expression and other fields specific to DP data type (for a detailed description of metadata content see section 6.2.3). Metadata is maintained by CII Configuration system [3], but can also be manipulated through the OLDB Client.

Data points can define a calculation formula (included in metadata) that determines the value of the data point. The formula can include URIs of other data points and so the value of the data point can depend on values of other data points. A data point with formula is called calculated data point and cannot be written to by users, since its value is determined by the formula.

OLDB Client API provides a means to manipulate data points (create, read, write and delete) through the OLDB client. Users can access and manipulate OLDB through the Client API library (srv-oldb) or OLDB GUI and CLI applications. The srv-oldb library is written in C++ and Python. For examples, see Chapter 4 where complete examples are listed in both languages.

OLDB uses Redis key-value store [4] for storing the data. Small data points are stored in Redis as keys, while large DP that exceed the limit defined in the OLDB configuration (section 3.3.1) are split automatically before being saved. The Redis storage is further divided into two parts:

main storage: This is the default location for data points. All small sized data points are stored here.

extra storage: This is a special storage where only predetermined data points (defined in data point configuration) are stored. It is intended for larger values that would affect the performance of the main storage.

Both default and in-memory-only storages are horizontally scalable and can consist of any number of Redis servers either single instance or clusters.

The lifetime of the data points is not bounded by the lifetime of the OLDB client instance because their values and configurations are saved in the OLDB storage and Configuration service respectably. If the Redis storage is configured to be persistent, the data points also persist after Redis server shutdown.

7.3. Prerequisites

This section describes the prerequisites for using the OLDB library and applications. Note that the preparation of the environment (i.e. installation of required modules, configuration) is not in the scope of this document. For information on how to run the services, refer to [2].

7.3.1. WAF Modules

For using the OLDB library and CLI application the following modules must be built and installed with WAF in the specified order:

elt-mal
client-api
srv-config
oldb-client
srv-oldb

For using the OLDB GUI application in addition to the above two more modules must be built and installed:

elt-qt-widgets
oldb-gui

7.3.2. Services

7.3.2.1. Redis

OLDB uses Redis [4] key-value store as its main storage of data point values. At a minimum the OLDB needs at least one main storage server, at least one extra storage (External Redis Server - 5.6) storage and exactly one pub/sub server. One Redis instance can serve as all three servers so minimally one Redis instance must be running for OLDB to work.

The number of default and in-memory-only storage servers is unbounded. Only one Redis server can be designated to be a pub/sub server and can be either a single instance or cluster.

7.3.2.2. Calculation Engine

For the OLDB framework to work as intended, the calculation service (OLDB service) must be running. The calculation service is responsible for calculating the value and quality of calculated data points. The calculation service consists of any number of calculation node processes and a scheduler process. Each calculation node is responsible for the calculation of a subset of all existing data points and the scheduler process is responsible for distributing the calculated data points among the calculation nodes.

All the calculation nodes must be running before the scheduler process. The scheduler must be configured with addresses of existing calculation nodes so it can connect to them.

To add a calculation node, the scheduler configuration must be updated with the new node’s address and the scheduler process must be restarted. The old nodes will retain their calculated data points and continue to recalculate their values. Note that during scheduler downtime newly created calculated data points will not be distributed and calculated. They will be distributed however when the scheduler process restarts.

If a calculation node shuts down the subset of calculated data points assigned to it will not be reassigned to another calculation node and will not be calculated. For this to happen the scheduler process must be restarted.

The proper way to shut down the calculation service is to first stop the scheduler process and then the calculation nodes.

7.3.2.3. Configuration Service

OLDB uses the internal CII configuration storage [3] to store its configuration. The configuration must be accessible for the OLDB Client and Calculation Service to successfully initialize.

7.3.3. Configuration

Both the OLDB Client and the calculation service must be configured to work properly. Both OLDB and Calculation Service configurations are stored in a versioned way.

7.3.3.1. OLDB Client Configuration

The OLDB client configuration is stored in the CiiOldbConfigClass object. The configuration is fetched from the CII Configuration Service by the OLDB client at start-up.

The configuration class contains the following fields:

redisServers: a MdStringArray class containing a list of strings that contains parameters for default storage servers. Each string corresponds to one server and is formatted as <sequential-number>:<hostname>:<port>:<server-type> where server-type is either SINGLE or CLUSTER and describes the type of the server (either single instance or cluster). The <sequential-number> should be the sequential number of servers in the list starting at 0.
redisServersExternal: a MdStringArray class containing a list of strings that contains parameters for in-memory-only storage servers. Each string corresponds to one server and is formatted as <alias>:<hostname>:<port>:<server-type> where server-type is either SINGLE or CLUSTER and describes the type of the server (either single instance or cluster).
pubSubServerInfo: a MdString class containing a string that contains parameters for the OLDB pub/sub server. The string is formatted in the same manner as strings in the redisServers field, except that the <sequential-number> has no significance since there is always one server.
pubSubHandlerProccesingInterval: a MdInt32 containing an integer
that represents the interval in milliseconds at which a pub/sub message handler should handle messages. To prevent the incorrect ordering of command execution, the incoming pub/sub messages are ordered by timestamp in unbounded queues and processed every pubSubHandlerProccesingInterval milliseconds. The performance of the pub-sub system will degrade with larger pubSubHandlerProccesingInterval value, be but the correct order of message execution will be more guaranteed. The recommended value for this field is 10. The size of the pub/sub event queue is unbounded, and thou it is not adjustable. Note: This reason for this setting is the proper timestamp ordering of the incoming messages.
valueSizeLimitRedisExt: a MdInt32 containing an integer that
represents the value size limit in bytes for in-memory-only servers. A data point value written to these servers is sliced and distributed among several key-value pairs so that each chunk does not exceed the valueSizeLimitRedisExt. If this value is too small write operations on large data points will be slower. It should not be greater than 512 Mb (512 * 1024 * 1024 bytes) since this is the limit of one key-value pair on Redis. The recommended value for this field is 128 Mb (128 * 1024 * 1024 bytes). The value must be set in bytes (e.g., 134217728).
extRedisKeyExpireTime: a MdInt32 containing an integer that
represents the expire time in milliseconds of the keys on the in-memory-only servers. Whenever a data point is written to in-memory-only storage, the old value keys are not rewritten (some client could at that time be reading them) but only set to expire (are deleted) after the extRedisKeyExpireTime. This value should always be bigger than a time of a write operation on any data point, else a read operation could return an error or malformed value. The recommended value is 5000.

Note that the OLDB client configuration cannot be changed dynamically. If the configuration has been changed, the OLDB client must be restarted for the changes to take effect.

The OLDB configuration is stored in CII config storage under the URI:

cii.config://remote/oldb/configurations/oldbclientconfig

For testing purposes, a default remote configuration can be deployed on the Configuration Service using the oldb-initRedis script. This script will initialize testing OLDB configuration data. Every time the script is run, it will create a new version. The OLDB client always reads the last version of the configuration at start up.

oldb-initRedis

7.3.3.2. Calculation Service Configuration

The scheduler process must be configured before it can be run. The configuration is done through the CiiOldbSchedulerConfig in the CII configuration storage [3]. The only field this configuration needs is a list of string URIs of calculation nodes that the scheduler should handle.

The calculation service configuration is stored in CII config storage under the URI:

cii.config://remote/oldb/calculationnodes

7.4. OLDB Library Usage

This section explains through examples of how to use the OLDB API library in an application. The OLDB API library provides the user the functionality to create, read, write, query, subscribe to and delete data points in the CII OLDB. The API is distributed among two classes. The CiiOldb and CiiOldbDataPoint.

7.4.1. Includes/Imports

For basic usage of the OLDB library, the user needs the CiiOldb client class, the CiiOldbDataPoint and the CiiOldbDpValue class. The first two classes contain the API for manipulating data points and the last one is a container for the data point value triplet (i.e. value, quality, timestamp). Next the language specific URI class is needed. If data points are created with provided metadata, respective metadata classes need to be imported. See section 6.2.3 for the list of metadata classes.

7.4.1.1. C++

#include <ciiOldbDpValue.hpp>
#include <ciiOldbFactory.hpp>
//Metadata classes
#include <meta/MdOldbNumber.hpp>

7.4.1.2. Python

import elt.oldb
import elt.config.Uri

For subscribing to data point changes, the CiiOldbSubscription interface or a class implementing this interface must be imported.

7.4.1.3. C++

#include <ciiOldbSubscription.hpp>

7.4.1.4. Python

Included in elt.oldb module.

Since OLDB API also throws exceptions, these need to be imported to. For C++ these exceptions can be found in the <ciiOldbExceptions.hpp> header. For Python these exceptions are included in elt.oldb module.

Linked library includes can be seen in code examples of 4.1.2, 4.1.3 and 4.1.4

7.4.2. Basic Example

The examples in this section can be found in the oldb-examples project (sample-app.cpp, cii-oldb-examples-sample-app-py.py).

In basic example (Listing 4‑1, Listing 4‑2) four constant (without formulas) data points of type DOUBLE, STRING, MATRIX2D_DOUBLE and VECTOR_INT32 are created. Multiple different creation methods are used to create these data points (see appendix C.1 on creation methods). These data points are then read to confirm their initial values. New values are then written to the data points.

Then the data points are queried using a glob expression and again with additional filter arguments (query for the specific type and value range).

Then the children of specific directory parent in the URI hierarchy are queried.

At last, the data points are deleted and OLDB client is closed.

Note: All OLDB client methods are synchronous towards the Redis server.

Listing 4‑1: C++ Basic Example

/**
 * @copyright (c) Copyright ESO 2019 All Rights Reserved
 * ESO (eso.org) is an Intergovernmental Organisation, and therefore special legal conditions apply.
 * @ingroup oldb-examples
 */
#include <mal/utility/Uri.hpp>

#include <ciiOldbFactory.hpp>
#include <ciiOldbDpValue.hpp>
#include <ciiOldbExceptions.hpp>


int main(int ac, char *av[]) {
  try {
    // Initialize OLDB

    std::shared_ptr<elt::oldb::CiiOldb> oldb_client = ::elt::oldb::CiiOldbFactory::GetInstance();
    ::elt::oldb::CiiOldbGlobal::SetWriteEnabled(true);
    std::string RND_PREFIX = ::elt::oldb::CiiOldbUtil::NewUUID();

    // Define URIs

    ::elt::mal::Uri double_dp_uri{"cii.oldb:///" +RND_PREFIX+ "/sampleroot/child/device/doubledp"};
    ::elt::mal::Uri matrix_dp_uri{"cii.oldb:///" +RND_PREFIX+ "/sampleroot/child/matrixdp"};
    ::elt::mal::Uri string_dp_uri{"cii.oldb:///" +RND_PREFIX+ "/sampleroot/child/stringdp"};
    ::elt::mal::Uri vector_dp_uri{"cii.oldb:///" +RND_PREFIX+ "/sampleroot/child/vectordp"};

    // CREATE DATA POINTS

    // Create DOUBLE data point with provided metadata instance.
    // Metadata must first be created and saved to Config Service.

    const std::string double_dp_meta_instance_name = RND_PREFIX + "customDoubleDpMeta";
    std::shared_ptr<elt::config::CiiDataPointMetadataBase> double_dp_meta = CiiDataPointMetadataFactory::getNewMetadataInstance<
      ::elt::config::classes::meta::MdOldb<double>>(double_dp_meta_instance_name);

    double_dp_meta->setComment("metadata of a constant DP");
    double_dp_meta->set_default_value(0.0);
    double_dp_meta->set_min_limit(0.0);
    double_dp_meta->set_max_limit(10.0);

    // meta data needs to be created before calling CreateDataPoint methods

    ::elt::oldb::CiiOldbUtil::SaveOrUpdateMetadata(*double_dp_meta);

    std::shared_ptr<::elt::oldb::CiiOldbDataPoint<double>> double_dp = oldb_client->CreateDataPoint<double>(
      double_dp_uri, double_dp_meta_instance_name);

    // Create MATRIX2D_DOUBLE data point with default matrix metadata OldbDoubleMatrixStd
    // and provided initial value. The default metadata must exist on Config Service.
    // Run oldb-initRedis script to generate default metadata instances.

    std::vector<double> matrix_double =
      {1.0, 20.0, 30.0, 40.0, 50.0, 4.4, 45.3, 34.4, 445.3, 301.3,
       2.0, 33.3, 34.3, 33.3, 33.3, 5.5, 32.2, 33.4, 222.3, 203.1};

    std::shared_ptr<::elt::config::CiiConfigClient::CiiDataPointMetadataBase> mdi_matrix_double =
       ::elt::config::CiiConfigClient::RetrieveMetadata(
         ::elt::oldb::CiiOldbGlobal::GetTargetConfigStorage(),
         ::elt::oldb::CiiOldbGlobal::MDI_MATRIX_DOUBLE,
         ::elt::oldb::CiiOldbGlobal::CONFIG_VERSION);

    bool is_matrix = true;
    std::shared_ptr<::elt:oldb::CiiOldbDataPoint<std::vector<double>>> matrix_dp =
       oldb_client->CreateDataPointByValue(matrix_dp_uri, matrix_double, is_matrix);

    // Create STRING data point with default string metadata OldbStringStd and
    // provided initial value.

    std::shared_ptr<::elt::config::CiiConfigClient::CiiDataPointMetadataBase> mdi_string =
       ::elt::config::CiiConfigClient::RetrieveMetadata(
         ::elt::oldb::CiiOldbGlobal::GetTargetConfigStorage(),
         ::elt::oldb::CiiOldbGlobal::MDI_STRING,
         ::elt::oldb::CiiOldbGlobal::CONFIG_VERSION);

    std::shared_ptr<::elt:oldb::CiiOldbDataPoint<std::string>> string_dp = oldb_client->CreateDataPointByValue(
      string_dp_uri, std::string("ABCDEF"));

    // Create VECTOR_INT32 data point with provided metadata instance name.
    // Metadata with this instance name must exist in Config Service.

    std::shared_ptr<::elt::config::CiiConfigClient::CiiDataPointMetadataBase> mdi_vector_int32 =
       ::elt::config::CiiConfigClient::RetrieveMetadata(
         ::elt::oldb::CiiOldbGlobal::GetTargetConfigStorage(),
         ::elt::oldb::CiiOldbGlobal::MDI_VECTOR_INT32,
         ::elt::oldb::CiiOldbGlobal::CONFIG_VERSION);

    std::cout << "Creating datapoints\n";

    std::shared_ptr<::elt::oldb::CiiOldbTypedDataBase> vector_dp_base =
       oldb_client->CreateDataPoint(vector_dp_uri, ::elt::oldb::CiiOldbGlobal::MDI_VECTOR_INT32);
    ::elt::oldb::CiiOldbDataPoint<std::vector<std::int32_t>> vector_dp =
      std::dynamic_pointer_cast<::elt::oldb::CiiOldbDataPoint<std::vector<std::int32_t>>>(
        vector_dp_base);
    if (!vector_dp) {
      throw ::elt::oldb::CiiOldbException("Casting to data point of vector int32 type failed");
    }

    // Read data point values.
    // Unline matrix and string data points, the double_dp was created wihout a value.
    // That means a default value of BAD quality was used and since we are calling ReadValue before
    // WriteValue, we need to request that the quality check should be skipped.

    std::shared_ptr<::elt::oldb::CiiOldbDpValue<double>> const_dp_value = double_dp->ReadValue(false);
    std::shared_ptr<::elt::oldb::CiiOldbDpValue<std::vector<double>>> matrix_dp_value = matrix_dp->ReadValue();
    std::shared_ptr<::elt::oldb::CiiOldbDpValue<std::string>> string_dp_value = string_dp->ReadValue();

    // vectors do not have default value when created, needs a write before read
    vector_dp->WriteValue(std::vector<std::int32_t>{2, 4, 6});
    std::shared_ptr<::elt::oldb::CiiOldbDpValue<std::vector<std::int32_t>>> vector_dp_value = vector_dp->ReadValue();

    // Write to data points

    std::cout << "Writing datapoints\n";

    std::vector<double> new_matrix_value(2500);
    for (std::int32_t i = 0; i < 2500; ++i) {
      new_matrix_value.push_back(static_cast<double>(i));
    }

    double_dp->WriteValue(2.0);
    matrix_dp->WriteValue(new_matrix_value);
    string_dp->WriteValue("12345");
    vector_dp->WriteValue(std::vector<std::int32_t>{2, 4, 6});

    // Read the data point values.

    std::cout << "Reading datapoints\n";

    const_dp_value = double_dp->ReadValue();
    matrix_dp_value = matrix_dp->ReadValue();
    string_dp_value = string_dp->ReadValue();
    vector_dp_value = vector_dp->ReadValue();

    // Get multiple data points satisfying an URI glob expression

    std::cout << "Multi-retrieval of datapoints\n";

    std::vector<::elt::mal::Uri> uris = {::elt::mal::Uri("cii.oldb:///sampleroot/**")};
    std::vector<std::shared_ptr<::elt::oldb::CiiOldbTypedDataBase>> search_result = oldb_client->GetDataPoints(uris);

    // Get multiple DOUBLE data points with value between 1.0 and 3.0


    std::cout << "Filtered Search\n";

    std::vector<std::shared_ptr<CiiOldbDataPoint<double>>> filtered_search_result = oldb_client->GetDataPoints<double>(
      uris, ::elt::common::CiiBasicDataType::DOUBLE, 1.0, 3.0);

    // Get the children of directory URI cii.oldb:///sampleroot. Returns a map specifying whether
    // a child is a data point or just a directory

    std::map<std::string, bool> children = oldb_client->GetChildren(::elt::mal::Uri("cii.oldb:///sampleroot"));

    // Delete data points

    std::cout << "Deleting datapoints\n";

    oldb_client->DeleteDataPoint(double_dp_uri);
    oldb_client->DeleteDataPoint(matrix_dp_uri);
    oldb_client->DeleteDataPoint(string_dp_uri);
    oldb_client->DeleteDataPoint(vector_dp_uri);

    return 0;
  } catch (const ::elt::oldb::CiiOldbException& ex) {
    std::cerr << "CiiOldbException occured while executing sample code. What: "
      << ex.what() << '\n';
  }
  return -1;
}

Listing 4‑2: Python Basic Example

#!/usr/bin/env python
"""
@copyright (c) Copyright ESO 2019 All Rights Reserved
ESO (eso.org) is an Intergovernmental organisation,
and therefore special legal conditions apply.
@ingroup client-apis-python
@author Cosylab
"""
#pylint: disable=E1101,C0103,R0914,C0330,W0612
#This script shows basic usage of OLDB API.

import sys
import traceback
import uuid
import elt.config
import elt.oldb

# Define URIs

RND_PREFIX = str(uuid.uuid4())

double_dp_uri = elt.config.Uri("cii.oldb:///%s/sampleroot/child/device/doubledp" % RND_PREFIX)
string_dp_uri = elt.config.Uri("cii.oldb:///%s/sampleroot/child/device/stringdp" % RND_PREFIX)
vector_dp_uri = elt.config.Uri("cii.oldb:///%s/sampleroot/child/device/vectordp" % RND_PREFIX)
matrix_dp_uri = elt.config.Uri("cii.oldb:///%s/sampleroot/child/device/matrixdp" % RND_PREFIX)

# Initialize OLDB client
oldb_client = elt.oldb.CiiOldbFactory.get_instance()

def _main():
    """main method implementation"""
    # enable writing
    elt.oldb.CiiOldbGlobal.set_write_enabled(True)

    # create DOUBLE data point with provided metadata instance
    # Metadata must first be created and saved to Config Service.

    double_dp_meta = \
        elt.oldb.typesupport.DOUBLE.\
        get_new_number_metadata_instance("%scustomDoubleDpMeta" % RND_PREFIX)
    double_dp_meta.set_comment("metadata of a constant DP")
    double_dp_meta.set_min_limit(0.0)
    double_dp_meta.set_max_limit(10.0)
    double_dp_meta.set_default_value(0.0)

    # Create & Save Metadata
    elt.oldb.CiiOldbUtil.save_or_update_metadata(double_dp_meta)

    # Create DOUBLE data point
    double_dp = oldb_client.create_data_point(double_dp_uri, double_dp_meta.get_instance_name())

    # Create MATRIX2D_DOUBLE data point with default matrix metadata OldbDoubleMatrixStd
    # and provided initial value. The default metadata must exist on Config Service.

    matrix_double = elt.oldb.VectorDOUBLE([
      1.0, 20.0, 30.0, 40.0, 50.0, 4.4, 45.3, 34.4, 445.3, 301.3,
      2.0, 33.3, 34.3, 33.3, 33.3, 5.5, 32.2, 33.4, 222.3, 203.1])

    mdi_matrix_double = elt.config.CiiConfigClient.retrieve_metadata(
        elt.oldb.CiiOldbGlobal.get_target_config_storage(),
        elt.oldb.CiiOldbGlobal.MDI_MATRIX_DOUBLE,
        elt.oldb.CiiOldbGlobal.CONFIG_VERSION)

    is_matrix = True

    matrix_dp = oldb_client.create_data_point_by_value(matrix_dp_uri, matrix_double, is_matrix)

    # Create STRING data point with default string metadata OldbStringStd anf provided initial value

    mdi_string = elt.config.CiiConfigClient.retrieve_metadata(
        elt.oldb.CiiOldbGlobal.get_target_config_storage(),
        elt.oldb.CiiOldbGlobal.MDI_STRING,
        elt.oldb.CiiOldbGlobal.CONFIG_VERSION)

    string_dp = oldb_client.create_data_point_by_value(string_dp_uri, "ABCDEF")

    # Create VECTOR_INT32 data point with default vector metadata OldbDoubleVectorStd

    mdi_vector32 = elt.config.CiiConfigClient.retrieve_metadata(
        elt.oldb.CiiOldbGlobal.get_target_config_storage(),
        elt.oldb.CiiOldbGlobal.MDI_VECTOR_INT32,
        elt.oldb.CiiOldbGlobal.CONFIG_VERSION)

    vector_dp = oldb_client.create_data_point(vector_dp_uri,
                                              elt.oldb.CiiOldbGlobal.MDI_VECTOR_INT32)

    # Read data point values.
    # Unline matrix and string data points, the double_dp was created wihout a value.
    # That means a default value of BAD quality was used and since we are calling read_value
    # before write_value, we need to request that the quality check should be skipped.

    # double_dp was created wihout a value which means a default value
    # of BAD quality was used, therefore when reading we need to request
    # that the quality check should be skipped

    const_dp_value = double_dp.read_value(False)
    matrix_dp_value = matrix_dp.read_value()
    string_dp_value = string_dp.read_value()

    # vectors do not have default value when created, needs a write before read

    vector_dp.write_value([2, 4, 6])
    vector_dp_value = vector_dp.read_value()

    print('Initial Value of double_dp: ', const_dp_value.get_value())
    print('Initial Value of string_dp: ', string_dp_value.get_value())
    print('First Value of vector_dp: ', vector_dp_value.get_value())

    # Write new values to the Data points

    new_matrix_value = elt.oldb.VectorDOUBLE()
    for x in range(250):
        new_matrix_value.append(x)

    double_dp.write_value(2.7)
    matrix_dp.write_value(new_matrix_value)
    string_dp.write_value('New string value')
    vector_dp.write_value([7, 8, 9, 10])

    # Read values
    const_dp_value = double_dp.read_value()
    matrix_dp_value = matrix_dp.read_value()
    string_dp_value = string_dp.read_value()
    vector_dp_value = vector_dp.read_value()

    print('New Value of double_dp: ', const_dp_value.get_value())
    print('New value of matrix_dp: ', matrix_dp_value.get_value())
    print('New Value of string_dp: ', string_dp_value.get_value())
    print('New Value of vector_dp: ', vector_dp_value.get_value())

    # Get Multiple data points satisfying URI glob expression

    uris = [elt.config.Uri('cii.oldb:///%s/sampleroot/**' % RND_PREFIX)]

    search_result = oldb_client.get_data_points(uris)
    print('Search Result: ', search_result)

    # Get Multiple DOUBLE data points with value between 1.0 and 3.0

    filtered_search_result = oldb_client.get_data_points_double(
                        uris, elt.config.CiiBasicDataType.DOUBLE, 1.0, 3.0)
    print('Filtered search result: ', filtered_search_result)

    # Get the children of directory URI cii.oldb:///sampleroot. Returns a map specifying whether
    # a child is a data point or just a directory

    children = oldb_client.get_children(elt.config.Uri('cii.oldb:///sampleroot'))

    # Delete data points
    oldb_client.delete_data_point(double_dp_uri)
    oldb_client.delete_data_point(matrix_dp_uri)
    oldb_client.delete_data_point(string_dp_uri)
    oldb_client.delete_data_point(vector_dp_uri)

    # Delete meta data
    elt.config.CiiConfigClient.delete_metadata(
            elt.oldb.CiiOldbGlobal.get_target_config_storage(),
            double_dp_meta.get_instance_name())
    return 0

def main():
    """main method wrapper"""
    result = 0
    try:
        result = _main()
    #pylint: disable=W0703
    except Exception as e:
        print(e)
        traceback.print_tb(sys.exc_info()[2])
        result = 5
    return result

if __name__ == '__main__':
    sys.exit(main())

7.4.3. Subscribing Example

The examples in this section can be found in the oldb-examples project (subscription-sample-app.cpp, cii-oldb-examples-subscription-sample-app-py.py).

In the subscribing example (Listing 4‑3, Listing 4‑4) a data point of type DOUBLE is created and a subscription is added to it to detect value changes and deletion.

Listing 4‑3: C++ Subscribing Example

/**
 * @copyright (c) Copyright ESO 2019 All Rights Reserved
 * ESO (eso.org) is an Intergovernmental Organisation, and therefore special legal conditions apply.
 * @ingroup oldb-examples
 */
#include <mal/utility/Uri.hpp>

#include <ciiOldbFactory.hpp>
#include <ciiOldbSubscription.hpp>
#include <ciiOldbDpValue.hpp>
#include <ciiOldbExceptions.hpp>


namespace elt {
namespace oldb {
namespace app {

class AppOldbDpSubscription : public CiiOldbDpSubscription<double> {
 public:
  AppOldbDpSubscription():
    CiiOldbDpSubscription<double>(::elt::common::CiiBasicDataType::DOUBLE) {}

  void DpRemoved(::elt::mal::Uri uri) override  {
    std::cout << "dpRemoved:" << uri.string() << '\n';
  }

  void NewValue(std::shared_ptr<CiiOldbDpValue<double>> value, ::elt::mal::Uri uri) override  {
    std::cout << "newValue:" << uri.string() << " " << value->GetValue() << '\n';
  }
};

}  // namespace app
}  // namespace oldb
}  // namespace elt


int main(int ac, char *av[]) {
  try {
    // Initialize OLDB
    std::shared_ptr<elt::oldb::CiiOldb> oldb_client = ::elt::oldb::CiiOldbFactory::GetInstance();
    ::elt::oldb::CiiOldbGlobal::SetWriteEnabled(true);

    // Define URIs
    std::string RND_PREFIX = ::elt::oldb::CiiOldbUtil::NewUUID();
    ::elt::mal::Uri double_dp_uri{"cii.oldb:///" +RND_PREFIX+ "sampleroot/child/device/doubledp"};

    // CREATE DATA POINTS

    // meta data needs to be created before calling CreateDataPoint methods

    std::shared_ptr<::elt::config::CiiConfigClient::CiiDataPointMetadataBase> mdi_double =
       ::elt::config::CiiConfigClient::RetrieveMetadata(
         ::elt::oldb::CiiOldbGlobal::GetTargetConfigStorage(),
         ::elt::oldb::CiiOldbGlobal::MDI_DOUBLE,
         ::elt::oldb::CiiOldbGlobal::CONFIG_VERSION);

    std::shared_ptr<::elt:oldb::CiiOldbDataPoint<double>> double_dp = oldb_client->CreateDataPointByValue(double_dp_uri, 1.0);

    // Subscribe to a data point value changes.

    std::shared_ptr<::elt::oldb::app::AppOldbDpSubscription> subscription = std::make_shared<::elt::oldb::app::AppOldbDpSubscription>();

    double_dp->Subscribe(subscription);

    // Subscription takes affect after some period after Subscribe

    std::this_thread::sleep_for(std::chrono::seconds(2));

    // Write to data points

    double_dp->WriteValue(2.0);

    // Takes time for subscription listener to be called

    std::this_thread::sleep_for(std::chrono::seconds(2));

    // Delete data points

    oldb_client->DeleteDataPoint(double_dp_uri);

    std::this_thread::sleep_for(std::chrono::seconds(3));

    return 0;
  } catch (const ::elt::oldb::CiiOldbException& ex) {
    std::cerr << "CiiOldbException occured while executing sample code. What: "
              << ex.what() << '\n';
  }
  return -1;
}

Listing 4‑4: Python Subscribing Example

#!/usr/bin/env python
"""
@copyright (c) Copyright ESO 2019 All Rights Reserved
ESO (eso.org) is an Intergovernmental organisation,
and therefore special legal conditions apply.
@author Cosylab
"""
#pylint: disable=E1101,C0103,R0914,C0330,W0612

#This script presents DP subscription example.

import sys
import traceback
import time
import threading
import uuid
import elt.config
import elt.oldb

RND_PREFIX = str(uuid.uuid4())

# Initialize OLDB client
oldb_client = elt.oldb.CiiOldbFactory.get_instance()

# Define URIs
double_dp_uri = elt.config.Uri("cii.oldb:///%s/sampleroot/child/device/double_dp" % RND_PREFIX)

class AppOldbDpSubscription:
    """Subscription listener implementation, must implement
       new_value and dp_removed methods
    """
    #pylint: disable=R0201
    def new_value(self, value, uri):
        """Handle DP value change event"""
        print('new_value, value=%s, uri=%s' % (value.get_value(), uri.string()))

    def dp_removed(self, uri):
        """Handle DP remove event"""
        print('dp_removed, uri=%s' % (uri.string(),))

class WorkThread(threading.Thread):
    """Work thread implementation"""

    def run(self):
        """Modify DP then delete it"""
        dp = oldb_client.get_data_point(double_dp_uri)
        for x in range(1, 3):
            dp.write_value(x*.5)
            time.sleep(0.2)
        del dp
        oldb_client.delete_data_point(double_dp_uri)
        # Give subscription system time to pass the message around
        time.sleep(1.0)

def _main():
    """main method implementation"""
    # enable writing
    elt.oldb.CiiOldbGlobal.set_write_enabled(True)

    # create data points
    # metadata needs to be created before calling cereate_data_point method

    mdi_double = elt.config.CiiConfigClient.retrieve_metadata(
        elt.oldb.CiiOldbGlobal.get_target_config_storage(),
        elt.oldb.CiiOldbGlobal.MDI_DOUBLE,
        elt.oldb.CiiOldbGlobal.CONFIG_VERSION)

    double_dp = oldb_client.create_data_point_by_value(double_dp_uri, 1.0)

    # Subscribe to data point. Pass instance of SubscriptionListener() to the subscription.
    listener = elt.oldb.typesupport.DOUBLE.get_new_subscription_instance(AppOldbDpSubscription())
    double_dp.subscribe(listener)

    # Release GIL for sufficient time to allow other threads to execute
    # See notes below this listing
    time.sleep(1.0)

    # Do not write to the data points from the same thread. Deadlock because of GIL.
    # launch work thread that manipulates data
    workThread = WorkThread()
    workThread.start()
    workThread.join()
    return 0

def main():
    """main method wrapper"""
    result = 0
    try:
        result = _main()
    #pylint: disable=W0703
    except Exception as e:
        print(e)
        traceback.print_tb(sys.exc_info()[2])
        result = 5
    return result

if __name__ == '__main__':
    sys.exit(main())

There is important caveat when using subscriptions and writing data point values in the same script. Python interpreter is GIL based meaning that only one thread of execution is active at one time.

When the script creates subscription and then immediatelly begins to write new values to the data point, subscription listener will not get new_value notifications.

This happens because main thread did not release the GIL to allow the whole mechanism to set up (subscriptions are handled in separate thread).

There is no clean solution to this. The suggested workaround is to release GIL in the main thread briefly after the subscription was created. Most handy method is by calling time.sleep() function. For more information please consult ticket ECII-779.

7.4.4. Advanced Example

The source code of the example is located in the CII Demo Repository (https://gitlab.eso.org/cii/info/cii-demo):

oldb-examples/cpp/advanced-sample-app/src/advanced-sample-app.cpp
oldb-examples/python/app/advanced-sample-app/src/cii-oldb-examples-advanced-sample-app-py.py

The advanced example demonstrates:

Creation and use of a calculated datapoint
Creation and use of a quality expression
Usage of the OLDB write protection mechanism
as well as other advanced aspects of OLDB usage.

Note that the calculation service must be running for this example to produce expected results.

7.4.5. Alias Example

The source code of the example is located in the CII Demo Repository (https://gitlab.eso.org/cii/info/cii-demo):

oldb-examples/cpp/aliases-sample-app/src/aliases-sample-app.cpp

The example demonstrates:

Basic Operations: Creation, Read, Update, Delete operations on aliases

// Data point needs not exist at the time of alias creation.
// Aliases are type agnostic, but one needs to know the type
// of the data point at the time of retrieval.

// create
client->CreateDataPointAlias (dp1_alias_uri, dp1_uri);

// read
std::shared_ptr<::elt::oldb::CiiOldbDataPoint<std::string>>
   alias_data_point = client->GetDataPoint<std::string> (dp1_alias_uri);

dp1_uri_again = alias_data_point->GetTargetUri();

// update
alias_data_point->WriteValue ("Text Value");

// delete
client->DeleteDataPoint (dp1_alias_uri);

Deleting underlying datapoints
Subscriptions on aliases and their underlying datapoints

7.5. Advanced Topics

This section describes in detail the advanced functionalities of OLDB API library.

7.5.1. OLDB Statistics

Basic traffic statistics can be obtained for the OLDB Client and Calculation service.

7.5.1.1. OLDB Client Statistics

For OLDB client the basic operation statistics can be obtained directly from Redis by using the redis-cli command line client application. Note that Redis must be installed on the machine so that the redis-cli application is available. Redis client app can be run in the terminal. It can be given options -h (for hostname) and -p (for the port) to connect to specific Redis server. By default it will connect to localhost:6379.

redis-cli -h 10.71.0.247
10.71.0.247:6379>

Do get basic statistics about the read and write operations the INFO command with commandstats option should be run (Listing 5‑1).

Listing 5‑1: INFO commandstats result

10.71.0.247:6379> INFO commandstats
# Commandstats
cmdstat_expire:calls=54,usec=754,usec_per_call=13.96
cmdstat_subscribe:calls=690,usec=9138,usec_per_call=13.24
cmdstat_exists:calls=296,usec=2683,usec_per_call=9.06
cmdstat_get:calls=948,usec=8600,usec_per_call=9.07
cmdstat_mget:calls=11,usec=88,usec_per_call=8.00
cmdstat_mset:calls=59,usec=662,usec_per_call=11.22
cmdstat_del:calls=190,usec=1708,usec_per_call=8.99
cmdstat_command:calls=3,usec=3138,usec_per_call=1046.00
cmdstat_set:calls=29601,usec=411882,usec_per_call=13.91
cmdstat_publish:calls=29992,usec=318727,usec_per_call=10.63
cmdstat_info:calls=3,usec=273,usec_per_call=91.00

The most important fields in the (Listing 5‑1) are:

cmdstat_get:calls – number of read operations.

cmdstat_set:calls – number of write operations.

cmdstat_del:calls – number of delete operations.

See [5] for full explanation of the INFO command and how to get other information about the Redis server (e.g. master – slave statistics for cluster servers).

7.5.1.2. Calculation Service Statistics

As of CII v4, the calculation service does not provide statistics. This may be re-added in a future version.

The logs are located at /var/log/elt/cii-oldb-calc-XYZ.

7.5.2. Disabling Writes to Data Points

Write operations on data points can be disabled or enabled by calling the setWriteEnabled method on the OLDB client. If disabled, all calls of writeValue on data points will throw a CiiOldbWriteDisabledException. Note that the write disabling only effects the same process environment in which it was done. Two OLDB clients on different machines (or in different processes on the same machine) can have different settings regarding the permission of write operations. Write operations are by default enabled.

 //Disable write operations on data points
oldbClient.setWriteEnabled(false);
  try {
    inputDubleDp.writeValue(0.2);
  } catch (CiiOldbWriteDisabledException e) {
    System.out.println("Write on DP not permited");
  }
oldbClient.setWriteEnabled(true);

When writes to the data points are enabled, a user can also enable an additional check for a formula existance when writing. If the data point formula is not empty and the write is not done by a calculation engine, write will fail with an exception CiiOldbIllegalOperationException. User can enable this check by calling the method SetCheckFormulaIfWriteEnabled in the OLDB client API.

7.5.3. Custom functions in Data Point Formulas

As of CII v4, the Calculation engine does not support custom functions. This may be re-added in a future version.

7.5.4. Manipulating Data Point Metadata

Metadata of the data point can be updated dynamically. Certain metadata attributes (e.g. limits or formula) can be changed or the data point’s metadata instance can be replaced altogether (change of the metadata instance name). This can be done by calling the setMetadata method on the data point which accepts a string metadata instance name. The data point metadata can be changed in two ways:

The attributes of the metadata of this data point have been changed (the provided metadata instance name is the same as the one the data point already has). These changes must be saved to Configuration Service before calling this method for it to have any effect.
The metadata instance of the data point has been replaced (the provided metadata instance name is different than the one the data point already has). The new metadata must already exist and must be of the same type as the old one. This change will be propagated to the data point configuration (Section 5.5).

The change will be propagated to the calculation service so that if the change is relevant to it (e.g. a data point has become a calculated data point), it can take proper actions (e.g. add it to a calculation node).

//Update a constant data point metadata to change it into calculated data point
//(add a formula to metadata)
squareMd.setFormula(String.format("DP_VALUE('%s')^2", inputDoubleUri.toString()));


configClient.updateMetadata(
CiiOldbUriUtils.createMetaDataConfigUri(squareMd.getMetadataInstanceName()),
squareMd.getMetadataInstanceName(), -1, squareMd);

squareDoubleDp.setMetadata(squareMd);

Note that the type of data point cannot be changed. Trying to update metadata with a different type of metadata will result in CiiOldbInvalidTypeException. That is, the class of the metadata cannot change once it has been defined in the data point at creation. If there is a need for data point type change the data point must be deleted and created again with the updated metadata.

The change will however not be propagated to other instances of the data point. For the change to take effect on these data points must be fetched again from the OLDB.

7.5.5. Data Point Configuration

Each data point has a configuration in Configuration Service which is created automatically at data point creation. It is primarily used for internal management so that the user needs not to concern themselves with it except in a case where the storage location needs to be changed.

The data point configuration is stored in the CiiOldbDataPointInfo object. This configuration class contains the following fields.

uri: the MdString class containing the URI of the data point as the string
metadataInstanceName: the MdString class containing the data point’s metadata instance name as a string.
metadataClassName: the MdString class containing the data point’s metadata class name as a string (See Table 6‑1 for a list of OLDB metadata classes).
serverAlias: The MdString class containing the alias for external Redis server (in-memory-only data storage) on which this data point value is stored. All newly created data points have this field null which means that their value is stored in main storage (default Redis server).
dataTypeGroup: unused MdString class.
isSubscriptionNotificationOnly: the MdBoolean class defining behaviour of data point subscriptions on data point value changes. Default is false, which causes subscriptions to receive new data point value via NewValue() subscription callback. When set to true, ValueChanged() subscription callback is invoked providing only data point uri without new value. This is useful for large data points (i.e vectors, matrices) where transferring large quantities of data can hinder application performance.

Fields relevant to the user is serverAlias and isSubscriptionNotificationOnly. Field serverAlias can be used to change the storage location of the data point (Section 5.7). Field isSubscriptionNotificationOnly modifies behaviour of data point subscriptions as described above.

7.5.6. External Redis Servers

To provide the ability for low latency high throughput, the OLDB client has an additional setting that enables the usage of multiple independent Redis servers to run in parallel.

By default, all newly created data points are stored in the default storage location (default Redis servers). This location is defined in the data point configuration. For low latency high throughput configuration, it is possible to also define in-memory-only storage locations (external Redis servers) and transfer data point values there.

External Redis servers are used for data points with larger values that would negatively affect the responsiveness of the main storage. They are regular Redis servers (can be single instances or clusters), but data point values sliced and saved under multiple key-value pairs. The number of slices depends on the data size limit of each slice which is defined with the valueSizeLimitRedisExt field in the OLDB configuration (Section 3.3.1). The list of the external Redis servers that OLDB client will connect to is written in the OLDB configuration. This setting is read at OLDB client start up (See redisServersExternal filed in Section 3.3.1). There each external server is formatted as <alias>:<hostname>:<port>. This alias, which is defined by the administrator can be used to assign a data point to the external server. A single specific data point can be assigned to specific Redis External Server. The data point setting serverAlias is used to control this behavior (5.5).

If extra storage servers are intended to be used for high throughput of large data, properly tuning Redis will be needed. The setting can be set to single or cluster Redis, but when low latency is required a cluster can cause additional delay. By design, Redis is single-threaded, which means that every write or read to the server, will block it. In case a cluster is used, the time to copy the data between the cluster parts need to be considered. If large blocks of data are used, these times can be very long.

The data points that have value written to the external Redis server have also a key-value field on the default Redis storage. This key-value doesn’t represent the actual data point value, but it is only the address of the external server (alias of the external Redis).

7.5.7. Defining the Storage Location of a Data Point

By default datapoints are hosted on the main storage. To create a datapoint on an extra storage server (external redis), users set the creation mode before creating the data point.

// datapoint will live on main redis
oldb_client->CreateDataPointByValue (data_point_uri1, someValueX);

// reconfigure creation mode
elt::oldb::CiiOldbDpCreateContext custom_context;
std::string extredis = "testExtRedisServer";
custom_context.server_alias = extredis;
oldb_client->SetDataPointCreateContext(custom_context);

// datapoints will live on extredis
oldb_client->CreateDataPointByValue (data_point_uri2, largeValueX);
oldb_client->CreateDataPointByValue (data_point_uri3, largeValueY);

// reset creation mode
::elt::oldb::CiiOldbDpCreateContext default_context;
oldb_client->SetDataPointCreateContext(default_context);

// datapoint will live on the main redis
oldb_client->CreateDataPointByValue (data_point_uri4, someValueY);

Moving an already existing datapoint to a different storage is a highly involved operation, which we discourage. The needed steps include: retrieving and updating the data point configuration using the internal config API, and changing the serverAlias field. If this field is set to null the data point value will be moved to the main storage servers, otherwise it will be moved to external Redis server with this alias. For the configuration change to take effect in the OLDB storages, the data point must be fetched again from the OLDB and a value needs to be written to it. Also note that clients currently using the data point are not notified of the change, so they may be need to be restarted, in order to re-read the oldb.

7.5.8. Enabling notification-only subscriptions on datapoints

To allow oldb client to create data points with notification only subscriptions, instance of structure ::elt::oldb::CiiOldbDpCreateContext must be initialized with is_subscription_notification_only member set to true and then applied to the client with SetDataPointCreateContext() method.

To restore initial client behaviour when creating data points, call client’s SetDataPointCreateContext() method with unmodified instance of ::elt::oldb::CiiOldbDpCreateContext structure.

Example:

// C++
::elt::oldb::CiiOldbDpCreateContext context;
// All data points created with this context will have their configuration setting
// isSubscriptionNotificationOnly set to true, causing their subscriptions to
// receive notifications of the value change trough ValueChanged() callback.
context.is_subscription_notification_only = true;
auto client = ::elt::oldb::CiiOldbFactory.GetInstance();
client->SetDataPointCreateContext(context);
// Create data points ...
client->CreateDataPoint(...)

// Once we are done
::elt::oldb::CiiOldbCreateContext default_context;
client->SetDataPointCreateContext(default_context);
// Data points created after this point, will have their subscriptions receiving
// notifications of the value change through NewValue() callback.

# Python
import elt.oldb
context = elt.oldb.CiiOldbDpCreateContext()
# All data points created with this context will have their configuration setting
# isSubscriptionNotificationOnly set to true, causing their subscriptions to
# receive notifications of the value change trough ValueChanged() callback.
context.is_subscription_notification_only = True
client = elt.oldb.CiiOldbFactory.get_instance()
client.set_data_point_create_context(context)
# Create data points ...
client.create_data_point(...)

# Once we are done
default_context = elt.oldb.CiiOldbDpCreateContext()
client.set_data_point_create_context(default_context)
# Data points created after this point, will have their subscriptions receiving
# notifications of the value change through NewValue() callback.

7.6. OLDB API LIBRARY

This section explains the details of the srv-oldb library. The OLDB API is split between two classes. The CiiOldb and CiiOldbDataPoint.

7.6.1. CiiOldb

The core OLDB Client API is the singleton CiiOldb class. The CiiOldb class contains methods for creating retrieving and deleting data points. See Appendix C for a description of CiiOldb API.

7.6.2. CiiOldbDataPoint

The other part of the OLDB Client API is the CiiOldbDataPoint class (Figure 3). It represents the data point in the OLDB and allows reading from and writing to it. The CiiOldbDataPoint class includes three main fields:

URI: the unique data point identifier.

CiiOldbDpValue: the value class containing the data point data value, timestamp and quality. It is parameterized with the type of the data value.

CiiOldbMetaData: the metadata class that contains data point type, formula, quality expression and other data type specific information.

The CiiOldbDataPoint is a generic class parameterized with the programming language type of the data value (for C++).

Figure 3: CiiOldbDataPoint class diagram

7.6.2.1. Data Point URI

Data point URI is the unique hierarchy identifier of the data point and contains OLDB schema cii.oldb:/// and the path to the data point (e.g. cii.oldb:///root/child/dp1).

Note: To make OLDB hierarchy working as expected, the data point URI should not contain any triple underscore (“___”) character. Because internally all “/” characters are converted to triple underscores, a triple underscores in a level name would result in an additional level.

Note: For the same reason, do not use path components that end with an underscore. For example cii.oldb:///a/b_/c will translate to a sequence of 4 underscores which would wrongly translate back to cii.oldb:///a/b/_c.

Note: Due to a limitation of the config service, the path component of the data point URI should not contain a sole number, i.e. cii.oldb:///a/12/dp1 is not allowed. Creating a data point with a number in the URI path will throw an exception clearly stating that a number in URI path is not allowed.

URI uri = point.getUri();

Note: The OLDB is case insensitive so two URIs differing only in case are considered the same. All URIs are internally stored in lower case.

Note: It is valid that a data point URI path is a sub-path of another data point URI path since the Configuration Service does not hold the URIs in a hierarchy structure. So for example, if there a data point with URI cii.oldb:///some/path/a, a data point with cii.oldb:///some/path/a/b can be created.

7.6.2.2. Data Point Value

The CiiOldbDpValue is a generic class parameterized with the programming language type of the data value (for C++, but not Python because it is not strongly typed).

CiiOldbDpValue<T> value = datapoint->ReadValue(false);

The CiiOldbDpValue contains three pieces of information that dynamically change during the data point lifetime:

7.6.2.2.1. Value

The actual data value of the data point. Data types of values are listed in Appendix A.

7.6.2.2.2. Quality

The CiiOldbDpQuality enumeration that represents the quality state of the data point. Possible values are OK, SUSPECT or BAD.

The meanings of the quality state are as follows:

OK: the value of the Property is valid

BAD: the value is not OK and should not be used

SUSPECT: there is a value, but it is not clear if it is reliable. For example, the readout of a sensor is inconsistent with other readings and the sensor might be faulty.

STALE: the value as not been updated recently (based on maxAge, see below) and should not be used.

If the data point quality is not OK, data point value cannot be read and written to. In order to read the datapoint with a BAD/SUSPECT quality is necessary to call ‘ReadValue’ method with a ‘False’ boolean value. Note that the difference between BAD and SUSPECT is justsemantic. The OLDB library threats these two states the same. If anything goes wrong with the data point internally (e.g. error in the data point formula) the quality is set to BAD. It is up to the user to set the quality to SUSPECT if appropriate.

7.6.2.2.3. Timestamp

The moment in time when the data point was last modified.

Timestamps are in UTC, with nanosecond resolution, since the epoch.

Example: This code would produce a valid timestamp in C++ (ignoring differences between Unix Time and UTC time):

auto now = std::chrono::system_clock::now();
auto nanos = std::chrono::time_point_cast<std::chrono::nanoseconds>(now);
auto since = nanos.time_since_epoch();
int64_t timestamp = since.count();

7.6.2.3. Metadata

The MdOldb is the class containing all the metadata information about the data point. It is not expected that this information would change often but it is possible to manipulate it through the OLDB API (Section 5.4). It is a subtype of MdBase from the Cii Config Service [3] so it contains all the fields contained in the MdBase. The OLDB relevant fields of the MdBase are genType (the type of the data point value) and defaultValue (the default value of the data point). The OLDB specific fields contained in the base class MdOldb are data point formula and qualityExpression.

For other fields in MdBase class consult the [3] document.

The CiiOldbDataPoint class contains a subtype of MdOldb specific to the data point type (Table 6‑1). The user of the CiiOldbDataPoint must know the type of the data point and the metadata class corresponding to it so they can retrieve the metadata object, as seen in the following example:

MdOldbString metadata = point.getMetadata(MdOldbString.class);

If at runtime the type of the data point is not known, the user can determine it by retrieving the metadata as the MdOldb base type, retrieving the generic type from it and casting the metadata in the right class.

MdOldb metadata = point.getMetadata(MdOldb.class);
AttributeType type = metadata.getGenType();
if (type == AttributeType.STRING) {
  metadata = (MOldbString) metdata;
}

All the built-in OLDB metadata classes are listed in Table 6‑1. All classes derive from MdOldb (which derives from config MdBase) class which defines basic OLDB fields (quality and formula). The metadata class associated with the data type can be read in Appendix B.

Note that it is not meant to extend the existing metadata classes and define new “types” of data points since the data types defined by Core Integration Infrastructure are covered with existing metadata. It is in theory possible to extend OLDB metadata classes, but it means extending the OLDB design and coding of new extended classes.

Table 6‑1: Built-in metadata classes

Class	Description	Specific Fields
MdOldbNumber	Metadata for all scalar number types. Parameterized with the language mapping of the type	minLimit (data point value type): the lower limit of the data point value. maxLimit (data point value type): the upper limit of the data point value. units (string): the units of the data point.
MdOldbString	Metadata for string data types. Not parameterized.	allowedValues (list): all the values the data point value is allowed to have. If the list is empty or null, all values are allowed. size (integer): the length of the string value. This is not a limit just informational filed.
MdOldbArray	Metadata for all vector data points. Parameterized with the type of the elements.
MdOldbMatrix	Metadata for all matrix data points. Parameterized with the type of the elements. This is a subtype of MdOldbArray*	width (integer): The width of the matrix (the number of elements in each matrix row).
MdOldbBinary	Metadata for byte array data points.

*Note: The matrix type data point actually holds the matrix flattened in to a vector. The metadata then holds the width of the matrix (number of elements in each row) so the user can reconstruct the matrix (e.g. if data point value is a vector of length 25 and the width property of the metadata is 5, the data point represents a 5x5 matrix). The length of the vector value must be divisible by the width otherwise the CiiConfigDataLimitsException is thrown.

7.6.2.3.1. Data Point Type

The Core Integration Infrastructure defines the types of all data values. All data point values are one of these predefined types. The predefined type effectively defines the type of the data point. Programmatically the data point type is defined in the AttributeType enumeration class (elt-common). See Appendix A for table of types relevant for data point values and their mappings to specific programming language types. The data point type is stored in the data point metadata in the genType field. Data point type cannot change during the lifetime of a data point. Data point with specific URI must be deleted and created again if the type needs to be changed.

AttributeType type = metadata.getGenType();

7.6.2.3.2. Data Point Formula

Data point formula is a mathematical expression by which the value of the data point can be calculated. If data point defines a formula, it is a calculated data point. Calculated data points cannot be directly written to (trying to do so will result in the CiiOldbIllegalOperationException). They are handled by the Calculation Service. The data point formula is defined in the data point metadata.

String formula = metadata.getFormula();

The Calculation Service uses the python language, and the data point formula must be a valid python expression and must return a value of the data point type.

Data point formula can contain references to other data points. This makes the data point dependent on other data points. The dependent data point value will be recalculated when one of its dependencies change. A valid reference to another data point is via a call to one of the DP_xyz functions with the data point URI given as a string argument. The engine expects the full URI of the data point.

Note: The URI string argument MUST be enclosed in SINGLE-QUOTES. See the example below. Using double-quotes, or triple quotes, or other variants, will not work, due to shortcomings in the current implementation.

Example:

DP_VALUE('cii.oldb:///root/dp1') + 3 * sqrt(DP_VALUE('cii.oldb:///root/dp2'))

Data point formula must not establish cyclic dependencies. Doing that will result in a CiiOldbCyclicDependencyException at the data point creation or metadata update.

The following references can be used to access a datapoint:

DP_VALUE(uri)
DP_QUALITY(uri)
DP_TIMESTAMP(uri)

and to access earlier data (from the previous computation):

DP_PREV_VALUE(uri)
DP_PREV_QUALITY(uri)
DP_PREV_TIMESTAMP(uri)

Any syntax errors in the data point formula will result in data point quality to be set to BAD by the Calculation Service.

7.6.2.3.3. Data Point Quality Expression

The quality expression is an expression which defines the data point quality. The data point quality represents the quality state of the data point and can have values OK, SUSPECT or BAD.

Data point quality evaluation is done by the Calculation Service (Section 3.2.3) and must be a valid python expression. The data point quality expression must return one of the strings “OK”, “SUSPECT” or “BAD”.

Example:

if_else (DP_VALUE('cii.oldb:///root/dp1') > 1.0,  "OK",  "BAD")

Any syntax error in the quality expression will result in data point quality to be set to BAD by the Calculation Service.

A quality expression can only be set on a calculated datapoint (i.e. one that also has a value formula).

A quality expression may reference the value of its own datapoint (in a value formula, this would be an illegal self-reference).

7.6.2.3.4. Data point value aging

Metadata attribute maxAge enables aging of data point values.

If defines the number of seconds without an update, after which the data point value goes “stale”. Once a data point value is stale, data point quality is changed to STALE. The callback method ValueStale() of any data point subscriptions is also invoked.

If the data point is subsequently updated (e.g. a new value and new last update time), the stale quality is removed, the original quality of the data point is restored. The callback method NewValue() of any data point subscriptions is also invoked.

The STALE quality is a “bad” quality (like BAD and SUSPECT) in the sense that a data point read will throw an exception reading a stale value unless the call is explcitly make to ignore quality.

Default value for maxAge attribute is 0, meaning the data point values never to become stale.

To turn on data point value aging, metadata attribute maxAge must be set to a non-zero, positive value and updated metadata applied to the data point.

Example:

// C++
auto metadata = datapoint->GetMetadata();
metadata.setComment("Enabled aging, values stale after 2 seconds");
metadata.set_maxAge(2);
::elt::oldb::CiiOldbUtil::SaveOrUpdateMetadata(metadata);
// Make sure data point is aware that metadata has changed
datapoint->SetMetadata(metadata->getInstanceName());

# Python
metadata = datapoint.get_metadata()
metadata.set_maxAge(2)
elt.oldb.CiiOldbUtil.save_or_update_metadata(metadata)
# Make sure data point is aware that metadata has changed
datapoint.set_metadata(metadata.get_instance_name())

The maxAge metdata is implemented in the client, it is not managed by the OLDB server. That is, stale values in Redis are not marked as stale.

7.6.2.3.5. Data point heart beats

A data point which contains a maxAge meta data attribute, may also have a heartbeatPoint meta data attribute. The heartbeatPoint is a (full) path to another data point, which is used solely for the purposes of determining if the data point is stale or not. That is, if data point, A, contains a maxAge of 10 (seconds) and has a heartbeatPoint of B, the determination of whether A is stale or fresh, is made against the last update time of B. The last update time of A is not used this computation.

Continuing with the example above, a subscription to A, will (internally) create a subscription to B. If B is updated with a new value, the new update time of B will be examined, and A subscriptions will have one of two call-backs triggered: ValueFresh(), if A is now fresh, based on B’s update time, or ValueStale(), if it is already older than “now”+maxAge.

This mechanism allows data points containing live data to point to a single, shared heartbeat point, and be updated on-change only, without going stale.

IMPORTANT: A heartbeatPoint must be of type double. The value or quality of the heartbeat data point does not matter, only the last update time is considered.

7.7. OLDB CLI

This section describes how to use OLDB command line application. The oldb-cli tool can be used for read, write, subscribe, create, delete operations and with data points that have textual outputs. The output of the tool can be parsed and used in other scripts.

New in CII v4: oldb-cli allows to omit the oldb-scheme prefix from a <data point uri>. If you omit the scheme, “cii.oldb://” will be assumed.

Example:

# with scheme
$ oldb-cli read cii.oldb:///glob/root/child/testintdp

# without scheme
$ oldb-cli read /glob/root/child/testintdp

7.7.1. Read operations

The tool reads the current value of the data point at the given data point URI and displays it on the standard output. Depending on the options used it can also display data point’s quality expression and formula as well. Arrays and matrices are displayed in the standard JSON format. It is also possible to write the value to a file (Section 7.1.3).

The syntax for read operations is:

oldb-cli read <data point URI> [-f <output file>|-e|-vf|-time|-quality|-value]

The -h argument prints help description to standard output.

7.7.1.1. Read value

To read a value of a data point, use the client together with an URI that points to the desired data point. If the data type of value is BINARY or its string representation exceeds 500 characters, the value won’t be displayed on the standard output and you will have to use the output file option to retrieve it (Section 7.1.3), otherwise an error will be thrown. To output just a single part of the (timestamp/quality/value) group, the options can be used (-time, -quality, -value).

Example:

$ oldb-cli read cii.oldb:///glob/root/child/testintdp
Timestamp: 2020-01-13T14:49:54.702Z
Quality:  OK
Value:  13

7.7.1.2. Read Value Formula and Quality Expression

To also retrieve value formula or quality expression of a data point use the options -vf (for value formula) and/or -e (for quality expression).

Example:

$ oldb-cli read cii.oldb:///glob/root/child/testintdp -vf -e
Quality expression: if (CURR_VAL() > 0) SETQUAL(OK) else SETQUAL(BAD) endif
Value formula: DP_VALUE('/root/child/device/DP1') + 3 * DP_VALUE('/root/child/device/DP2')
Timestamp: 2020-01-13T14:49:54.702Z
Quality:  OK
Value:  13

7.7.1.3. Save value to File

To save the data point’s value to a file use the -f option, followed by a path to file to write to. Unless the datatype of value is BINARY or its string representation exceeds 500 characters, the value will still get displayed on standard output as well.

Example:

Saving to file:

$ oldb-cli read cii.oldb:///glob/root/child/teststringdp -f testStringDpValue
Timestamp: 2020-01-07T19:09:13.117Z
Quality:  OK
Value:  ABCDEF

Reading the value from file:

$ cat testStringDpValue
ABCDEF

7.7.2. Write operations

The tool gets the data point at the given URI and writes a new value to it. Data point’s quality, timestamp, formula and quality expression can all be changed through the tool with the right options. Arrays and matrices must be given in the standard JSON format. Values can be given through a file too.

The syntax for writing is:

oldb-cli write <data point uri> <value>  [-f] [-h] [-q <quality>] [-e <quality expression>] [-t <timestamp>] [-vf]

The -h argument prints help description to standard output.

7.7.2.1. Write value

To write a value to data point, use the client together with an URI that points to the desired data point and value to be written. If the datatype of the value to be written is BINARY, the value must be given through the -f option (Section 7.2.5).

Example:

$ oldb-cli write cii.oldb:///glob/root/child/testintdp 1234
Timestamp: 2020-01-14T15:34:52.392Z
Quality:  OK
Value:  1234

7.7.2.2. Write Quality and Timestamp

When setting a data point’s value, the quality and timestamp can also be set.

To set the quality use the -q option followed by the desired quality (OK/SUSPECT/BAD).

Example:

$ oldb-cli write cii.oldb:///glob/root/child/testintdp 1234 -q BAD

Timestamp: 2020-01-14T15:34:52.392Z

Quality:  OK
Value:  1234

To set a timestamp to the data point use the -t option, followed by the timestamp in ISO 8601 extended date-time format.

Example:

$ oldb-cli write cii.oldb:///glob/root/child/testintdp 1234 -t 2020-01-07T19:09:13.117Z
Timestamp: 2020-01-07T19:09:13.117Z
Quality:  OK
Value:  1234

Note: all three data point properties (value, quality, timestamp) can be set in one operation using all the above options.

7.7.2.3. Write Formula

If you want to set a formula to a data point, replace the value argument with the value formula, and use the -vf option. The functionality will fist read the metadata instance name that belongs to the given data point and then writes the formula to the metadata instance.

Note: Formula is a part of the data point metadata instance. As one metadata instance can belong to many datapoints, this change can affect multiple data points.

Example:

oldb-cli write cii.oldb:///glob/root/child/testintdp3 'DP_VALUE(\'cii.oldb:///glob/root/child/testintdp2\') + 3 * sqrt(DP_VALUE(\'cii.oldb:///glob/root/child/testintdp\'))' -vf

7.7.2.4. Write Quality Expression

If instead of setting a fixed quality, you want to set a quality expression, you can do so by using the -e option followed by the desired quality expression. In this case, the value argument can be omitted.

Example:

oldb-cli write cii.oldb:///glob/root/child/testintdp2 -e 'if_else (DP_VALUE('cii.oldb:///glob/root/child/testintdp2') > 0,  "OK", "BAD")'

7.7.2.5. Write Value from File

To write a value from a file, replace the value argument with a path to the file containing the value and use the -f option.

Example:

Writing value from a file:

$ oldb-cli write cii.oldb:///glob/root/child/testintdp /home/esodev/values/testIntDpValue -f
Timestamp: 2020-01-14T15:40:34.783Z
Quality:  OK
Value:  25

7.7.2.6. Write Value directly from Formula

Value to specific data point can be written directly from the expression formula. To use the formula, the value must start with the “=” character.

Example:

Writing values from a formula:

$ oldb-cli write cii.oldb:///glob/root/child/testdoubledp '=e(3)'
Timestamp: 2020-07-17T08:48:22.399Z
Quality:  OK
Value:  20.085536923187668


$ oldb-cli write cii.oldb:///glob/root/child/testintdp '=DP_VALUE(\'cii.oldb:///glob/root/child/testintdp2\') + DP_VALUE(\'cii.oldb:///glob/root/child/testintdp3\')'

Timestamp: 2020-07-17T08:50:00.445Z
Quality:  OK
Value:  4435

7.7.4. Create operation

The create operation creates a datapoint of a specified type. For further configuration of the new data point, use the write operations. The datapoint will have its own dedicated metadata instance, therefore setting e.g. a formula on the datapoint will not affect other datapoints.

The syntax for the create operations is:

oldb-cli create <datapoint_uri>  <type>

# The -h argument prints the operation-specific help
oldb-cli.py create -h

Example:

$ oldb-cli create cii.oldb:///glob/root/child/teststring  STRING

$ oldb-cli write  cii.oldb:///glob/root/child/teststring  "my string"

7.7.5. Delete operation

The delete operation deletes a datapoint.

The syntax for the delete operations is:

oldb-cli delete <datapoint_uri>

# The -h argument prints the operation-specific help
oldb-cli.py delete -h

Example:

$ oldb-cli delete cii.oldb:///glob/root/child/teststring

7.8. GUI

This section describes the components of OLDB GUI. OLDB GUI has functionalities to read, write, create, delete and update data points. The GUI can also be used to read metadata of data points, but changing of metadata is not supported.

7.8.1. General

OLDB GUI is a graphical application that can be used for managing the data points and displaying their values. The data points are retrieved from the remote OLDB service.

oldbGui [path_to_application_config_file] [path_to_log_config_file]

The oldb gui executable takes 2 optional arguments:

path_to_application_config_file: a path to the file that contains the settings for the OLDB GUI application. An example value for the attribute: ../config/app-config.ini. A detailed description of the application configuration file can be found in section 8.2.
path_to_log_config_file: a path to the file that contains the settings for the log4cplus logging library. If it is not provided, the basic log4cplus configuration is used. Usually, the configuration file for the log4cplus library is named log4cplus.properties. Example value for the attribute: ../log4cplus.properties.

7.8.2. Configuration File

The configuration file for the OLDB GUI application contains the information needed to connect to the remote configuration service (i.e. the data source). The configuration file contains:

DataSource: this section contains all the data needed to connect to the primary configuration service. This service is used for retrieving the metadata about every selected data point. The hostname attribute provides the address of the service (in a form of an IP address or a host name) and the port attribute provides a port on which the service listens for incoming connections.
MatrixEdit: this section determines the max number of rows (maxHeight) and columns (maxWidth) that a given vector or matrix can have in order for its data to be displayed and edited directly in the OLDB GUI (for vectors, the maxWidth defines the size limit). Vectors and matrices that are larger than the MatrixEdit values can only be downloaded to the local disk and uploaded from the local disk.

If “path_to_application_config_file” is not provided to the application, default values will be used.

7.8.3. Functionality

7.8.3.1. Main Window

The main window is displayed when we execute the OLDB GUI. The application screenshot is shown in Figure 8‑1. As we can see from the figure, the functionality of the OLDB GUI is related to the management of data points.

Figure 8‑1: Main window

The GUI is divided into 2 parts. The left part displays the list of folders and data points retrieved from the remote service, search bar and the buttons for adding and deleting data points. The user can select the folders and data points by clicking them.

The right part is used for displaying the data about the selected data point: its value and metadata.

7.8.3.2. Menu

The OLDB GUI application only contains two menu entries, the Subscriptions (Figure 8‑2) and Options. The Subscription menu entry contains the 3 items for managing data point subscriptions (i.e. the windows for continuously displaying the latest data point values):

Load Subscriptions: loads the list of data point subscriptions from a file on disk (through file explorer) and opens them in windows in saved positions. All subscriptions are started (the value updates will be visible). If a subscription window is already displayed for a data point with a given URI, a new subscription window won’t be displayed once more.
Save Subscriptions: saves the list of currently displayed data point subscription windows (and their position on the screen) into a CSV file on disk. A file explorer window is opened and the user can enter the file name and location.
Close All Subscriptions: closes all the data point subscription windows that are currently displayed (unsubscribes from the data points).

Figure 8‑2: Application menu

The Options menu contains a checkbox item “write enabled”. This checkbox must be checked to enable write operations on data points. If this checkbox is not checked, writing to data points will not be possible. It is not checked by default.

7.8.4. Hierarchy view

The hierarchical view displays the hierarchical structure of folders and data points. It enables the user to do the following actions:

Browse and view the data points that are stored in the remote service.
Add new data points.
Delete existing data points.
Trigger the loading of data point values.
1. Selecting Folder

The user selects a folder by clicking the inner node in the data point hierarchy tree (Figure 8‑3). If the leaf tree node is clicked, the data point is selected. The folders are created automatically, based on the data points’ URIs.

When a folder is selected, all the data in the data point frame is cleared. The folders don’t have any values or metadata, they are only used for organizing data points. When a folder is selected the user can create a new data point by clicking on the Add button below the hierarchy tree.

Figure 8‑3: Selecting folder

Selecting Data Point

The user selects a data point by clicking the leaf node in the data point hierarchy tree (Figure 8‑4). The data points are organized in the tree structure based on their URIs.

For the easier searching of data points, a search filter bar is located at the top of the hierarchy tree. The user can input a part of the URI and when he/she hits the Enter key, all the data points whose name doesn’t contain the string input are filtered out (the string comparison is case insensitive). In order to display all the data points again, the user needs to clear the filtering text and hit the Enter key again.

When a configuration is selected, the following data is loaded in the data point frame of the GUI:

data point value, quality and timestamp: the most recent value, quality and timestamp of the selected data point.
the list of data point’s metadata attributes and their values.

Figure 8‑4: Data point selected

Note that it is not possible to change data point metadata through OLDB GUI application.

The data point URI is at the moment not showed anywhere in the data point frame. The user can reconstruct the data point URI from the tree path.

Adding a New Data Point

The adding of new data point is initiated by clicking the Add button below the hierarchy tree. The button is only enabled if a folder is selected.

When the Add button is clicked, a dialog for inserting a data point is displayed (Figure 8‑5). The dialog contains the following fields:

URI prefix: a read-only field that contains the URI prefix
URI suffix: the input field where the user can input the URI suffix or the name of the data point. The final URI of the configuration is formed by concatenating the URI prefix and suffix.
Metadata instance: here, the user selects the metadata instance for the newly created data point.

After the user clicks the OK button, the dialog is closed and a new data point is created on the remote server.

Figure 8‑5: Adding new data point

Deleting an Existing Data Point

The deleting of existing data point is initiated by clicking the Delete button below the hierarchy tree. The button is only enabled if a data point is selected.

When the Delete button is clicked, a confirmation dialog is displayed where the user can confirm or cancel the deletion. On user confirmation, the data point is deleted on the remote server.

Figure 8‑6: Data point deletion confirmation dialog

Data Point Value and Metadata Frame

The data point value and metadata frame occupy the right side of the application window. It displays the value and all the metadata values for the selected data point. It enables the user to do the following actions:

Refresh data point values.
Update data point values.
Subscribe to data point value changes.
1. Manually Refreshing Data Point Value

The manual refreshing of the value for the selected data point is triggered by clicking the Read button just below the area for displaying the data point value. After doing this, a request is sent to a remote server, the most recent data point value is retrieved and displayed in the GUI.

Setting the Data Point Value

There are 3 groups of data point values:

primitive values: the setting of the primitive value for the data point is performed using the input data dialog (Figure 8‑7) that is displayed when clicking the modify button next to the text field for displaying the data point value. The dialog contains a simple input field where the user can input the data value. After the value is entered and the dialog is closed (by clicking OK), the Write button must be clicked for value to be written to the OLDB storage.
vector & matrix values: they are displayed in a tabular form (Figure 8‑8). The table values are set by clicking different table cells. After doing this, the same dialog displays as with the primitive values. Here, the user inputs the value for the table cell. In order for the modified values to be sent to the remote service, the Write button needs to be clicked.
binary data values: the binary data values don’t get displayed in the OLDB GUI. It must be stored to or loaded from the local disk.

Whenever the user modifies the data point value in the GUI the timestamp field updates to “now”, so that when Write button is clicked the timestamp written to the OLDB is of the last change. The user can also modify the timestamp field in the same manner than the value.

Figure 8‑7: Data input dialog

Figure 8‑8: Vector & matrix data point value

The vector & matrix values as well as binary data values can also be loaded from a local disk and be stored to a local disk:

the loading from a local disk can be started by clicking the Upload button. In the dialog that gets shown, the user selects the file on local disk, from which the data point values will be loaded. After the file has been loaded it can be sent to the remote service by clicking the Write button.
the storing to the local disk can be started by clicking the Download button. In the dialog that gets shown, the user selects the file on local disk, to which the data point values will be stored.

For the vector & matrix data the user can also change the number of rows and columns. The row-and-column-setting dialog (Figure 8‑10) is displayed by right-clicking the table and selecting the Resize table item from the context menu (Figure 8‑9). If the matrix size is made smaller, any values already entered will be deleted for the removed columns or rows. If the matrix is made larger, the additional cells are empty.

Figure 8‑9: “Resize table” context menu

Figure 8‑10: Dialog for setting the number of rows and columns.

Subscribing to Data Point Value Changes

Subscribing to the value changes of the selected data point is performed by clicking the Subscribe button that is located under the widget for displaying data point value. At this point the new window dialog is displayed that displays the most recent value of the given data point. Whenever the data point value changes on the server it is automatically updated in the subscription dialog. Multiple subscription dialogs can be displayed at the same time, meaning that the user can monitor the values of multiple data points. For a given data point, however, only a single subscription window can be displayed, having duplicate subscription windows for the same data point is not possible.

The user can organize the subscription windows in any way they like, they can even store the position of the subscription windows into a file. Later, the user can use the file to restore the windows. This enables the user to have a pre-defined sets of data points that they can monitor (see section 8.3.2 on how this is done).

Figure 8‑11: Arrangement of multiple subscription windows on the screen

7.9. Data Point Value Type to Language Types Mapping

Below is the mapping table between data point types and data types of all programming languages that the OLDB can use.

Table 8‑1: Data point types mapping table

ELT CII BASIC TYPE	Java	C++	Python
INT8	Byte	std::int8_t	int
INT16	Short	std::int16_t	int
INT32	Integer	std::int32_t	int
INT64	Long	std::int64_t	int
UINT8	Byte	std::uint8_t	int
UINT16	Short	std::uint16_t	int
UINT32	Integer	std::uint32_t	int
UINT64	Long	std::uint64_t	int
SINGLE	Float	float	float
DOUBLE	Double	double	float
BOOLEAN	Boolean	bool	bool
STRING	String	std::string	str
BINARY	java.nio.ByteBuffer	std::vector<std::uint8_t>	bytes
VECTOR_INT32	List<Integer>	std::vector<std::int32_t>	list
VECTOR_SINGLE	List<Float>	std::vector<float>	list
VECTOR_DOUBLE	List<Double>	std::vector<double>	list
VECTOR_STRING	List<String>	std::vector<std::string>	List
MATRIX2D_INT32	List<Integer>	std::vector<std::int32_t>	list
MATRIX2D_SINGLE	List<Float>	std::vector<float>	list
MATRIX2D_DOUBLE	List<Double>	std::vector<double>	List
YAML		elt::oldb::datatypes: YamlNode	elt. oldb. dataty pes.Ya mlNode

7.10. Default Metadata Instance Names

ELT CII BASIC TYPE	Metadata Instance Name	Metadata Class name
INT8	OldbInt8Std	MdOldbNumber
INT16	OldbInt16Std	MdOldbNumber
INT32	OldbInt32Std	MdOldbNumber
INT64	OldbInt64Std	MdOldbNumber
UINT8	OldbUInt8Std	MdOldbNumber
UINT16	OldbUInt16Std	MdOldbNumber
UINT32	OldbUInt32Std	MdOldbNumber
UINT64	OldbUInt64Std	MdOldbNumber
SINGLE	OldbSingleStd	MdOldbNumber
DOUBLE	OldbDoubleStd	MdOldbNumber
STRING	OldbStringStd	MdOldbString
BINARY	OldbBinaryStd	MdOldbBinary
VECTOR_INT32	OldbInt32ArrayStd	MdOldbArray
VECTOR_SINGLE	OldbFloatArrayStd	MdOldbArray
VECTOR_DOUBLE	OldbDoubleArrayStd	MdOldbArray
VECTOR_STRING	OldbStringArrayStd	MdOldbArray
MATRIX2D_INT32	OldbInt32MatrixStd	MdOldbMatrix
MATRIX2D_SINGLE	OldbFloatMatrixStd	MdOldbMatrix
MATRIX2D_DOUBLE	OldbDoubleMatrixStd	MdOldbMatrix
YAML	OldbYamlStd	MdOldbYaml

7.11. Metadata attribute constraints

METADATA ATTRIB.	CONSTRAINT	CONSTRAINT VALUE
Comment	Max size in characters	80 chars

7.12. YAML Data point type

Inclusion of the support for YAML data point type was requested with ECII-702.

It was requested that OLDB should support data points that should accept Yaml::Node data type defined by yaml-cpp.

Directly mapping Yaml::Node datatype was not feasible as it is unknown to Python, therefore intermediate data type elt::oldb::datatypes::YamlNode (this is actually an alias to elt::config::datatypes::YamlNode) was defined in C++.

Datatype internally serialises to yaml document (string). Python binding for elt::oldb::datatypes::YamlNode is available as elt.oldb.datatypes.YamlNode.

YamlNode can be initialized in one of the following ways in C++:

#include <datatypes/yamlNode.hpp>
...
// Empty yaml document
auto node = elt::oldb::datatypes::YamlNode();
// Initialize with valid yaml document source. When argument cannot be
// parsed as valid yaml, elt::config::CiiValue is thrown
node = elt::oldb::datatypes::YamlNode("[this, is, yaml, list]");
// Initialize with YAML::Node argument
YAML::Node source = ...
node = elt::oldb::datatypes::YamlNode(source);
// Extract yaml document as string
std::string doc = node.AsString();
// Extract yaml document as YAML::Node
YAML::Node ydoc = node.AsYamlObject();
// Assignment of new value from string
node = "[this, is, another, yaml, list]";
// Assignment of new value from YAML::Node
node = ydoc;

Python bindings replicate and extend C++ functionality:

from elt.oldb.datatypes import YamlNode
# Empty yaml document
node = YamlNode()
# Initialize yaml document from string containing valid yaml. Again CiiValueError is raised in # case argument does not contain valid yaml.
node = YamlNode('[10, 20, 30, 40]')
# Initialize yaml document from python object
node = YamlNode(['this', 'is', 'python', 'list'])
# Update value with new yaml document source
node.set_source('[90, 20, 30, 40]')
# Update yaml node with new python object
node.set_value([90, 20, 30, 40])
# Extract yaml document source as string
s = node.as_string()
# Extract yaml document as python object
assert(node.as_value(), [90, 20, 30, 40])

Apart from the fact that YamlNode data type is not native and must be initialized in the one of the ways listed above, OLDB API supports data points that contain this type.

Vectors of YamlNode elements are not supported by OLDB.

7.13. OLDB API LISTING

This a listing of OLDB API methods with short description for both CiiOldb and CiiOldbDataPoint classes. The method signatures are written in Java. The signatures are equal (with some exceptions in some of createDataPoint methods) in all three languages except for the native data types of each language (e.g. for string, double, integer). For data type mapping between languages see Table 8‑1. For URI variables java uses java.net.URI, C++ uses <mal/utility/Uri.hpp> and Python uses elt.config.Uri.

Note that all OLDB API methods are synchronous (blocking) methods.

For detail explanation of exceptions thrown by these methods, consult the oldb-client source code.

7.13.1. CiiOldb

static CiiOldb getInstance()
The method returns the singleton instance implementation of the OLDB client CiiOldb.

CiiOldbDataPoint<?> createDataPoint(URI uri, String metadataInstanceName, String metadataClassName)

The method creates a data point at the defined URI with defined metadata. The data point data type and other restrictions are defined by its metadata which must already exist. Once the data point is created its data type cannot be changed.

Note: This method signature is different in C++ and Python where the String metadataClassName parameter is omitted. This is because of Java language specifics.

CiiOldbDataPoint<String> createDataPointByValue(URI uri, String value)
Creates The method creates a data point of type STRING at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbStringStd”. The metadata must already exist.

CiiOldbDataPoint<Byte> createDataPointByValue(URI uri, Byte value)
Creates The method creates a data point of type INT8 at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbInt32Std”. The metadata must already exist.

CiiOldbDataPoint<Short> createDataPointByValue(URI uri, Short value)
Creates The method creates a data point of type INT16 at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbInt16Std”. The metadata must already exist.

CiiOldbDataPoint<Integer> createDataPointByValue(URI uri, Integer value)
Creates The method creates a data point of type INT32 at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbInt32Std”. The metadata must already exist.

CiiOldbDataPoint<Float> createDataPointByValue(URI uri, Float value)
Creates The method creates a data point of type SINGLE at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbSingleStd”. The metadata must already exist.

CiiOldbDataPoint<Double> createDataPointByValue(URI uri, Short value)
Creates The method creates a data point of type DOUBLE at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbDoubleStd”. The metadata must already exist.

CiiOldbDataPoint<ByteBuffer> createDataPointByValue(URI uri, ByteBuffer value)
Creates The method creates a data point of type BINARY at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “OldbBinaryStd”. The metadata must already exist.

CiiOldbDataPoint<List<T>> createDataPointByValue(URI uri, List<T> value, Class<T> clazz, boolean isMatrix)

Creates The method creates a data point of matrix or vector type at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “Oldb<element-type>ArrayStd”. The metadata must already exist. The <element-type> is “Int32”, “Single”, “Double” or “String” (vector only). The isMatrix parameter defines whether a matrix or vector data point is created. The clazz parameter defines the class of the elements of matrix/vector.

Note: The Class<T> clazz parameter is only needed in Java version of the method because of language specifics.

CiiOldbDataPoint<List<T>> createDataPointByValue(URI uri, List<T> value, Class<T> clazz)

Creates The method creates a data point of vector type at the defined URI with defined value. It uses the default OLDB string metadata with metadata instance name “MdO<element-type>ArrayStd”. The metadata must already exist. The <element-type> is “Int32”, “Single”, “Double” or “String”. The clazz parameter defines the class of the elements of vector.

Note: The Class<T> clazz parameter is only needed in Java version of the method because of language specifics.

void deleteDataPoint(URI uri)
The method deletes a data point referenced by the URI.

Map<URI, Boolean> getChildren(URI uri)

Finds all direct children of the given URI. Returns the map of all found URI with boolean value indicating whether the URI represents a data point (true if data point, false if only folder). Since OLDB allows to create a data point with an URI that is a sub-path of another URI, there can be both folder and data point URIs present. In that case the boolean value is true.

Note: Because OLDB is case insensitive all URIs returned will be in lower case.

Example

Consider the following OLDB URIs:

cii.oldb:///root/child/device/dp1
cii.oldb:///root/child/dp2
cii.oldb:///root/child/dp2/dp3

A call to method getChildren with URI = cii.oldb:///root/child will return the map in Table 8‑2.

Table 8‑2: Example of getChildren call result

Key	Value
cii.oldb:///root/child/device	FALSE
cii.oldb:///root/child/dp2	TRUE

Note that path cii.oldb:///root/child/dp2 hierarchically represents both data point and a folder. In this case the getChildren returns TRUE for this path. This means that a TRUE for an URI does not mean that the URI doesn’t also represent a folder.

CiiOldbDataPoint<?> getDataPoint(URI uri)

Returns a data point at a specified URI.

Note that the caller of the getDataPoint method must know the data point type to cast the received object to the right type (only required for the strongly typed languages like C++). The type of the data point can be obtained from metadata (Section 6.2.3).

Note: this method does not provide transactional safety. There is no guarantee that the data point is not deleted the moment after it was retrieved (data point object of a deleted data point is not usable).

List<CiiOldbDataPoint<?>> getDataPoints(List<URI> uris)

The method returns a list of data points matching the specified URIs.

The URIs in the list can be written as a simple glob pattern using three special characters.

* for any number of characters restricted to the URI hierarchy level

** for any number of characters spanning multiple levels

^ for exactly one character

Example:

If we have a simple OLDB that contains data points with the following URIs:

cii.oldb:///root/child/device/dp1

cii.oldb:///root/child/device/dp21

cii.oldb:///root/child/device/dp23

cii.oldb:///root/child1/motor/dp2

Then calls to getDataPoints with these glob patterns would result in following data points:

cii.oldb:///root/child/device/dp2* -> dp21, dp23

cii.oldb:///root/child/** -> dp1, dp21, dp23

cii.oldb:///root/child/device/dp^ -> dp1

Note: this method does not provide transactional safety. There is no guarantee that a data point is not deleted the moment after it was retrieved (data point object of a deleted data point is not usable).

List<CiiOldbDataPoint<T>> getDataPoints(List<URI> uris, AttributeType dataType, T minValue, T maxValue)

The method returns a list of data points matching the specified URIs, filtered by type and min and max limits. As with the getDataPoints above, this method supports the simple glob patterns.

Note: this method does not provide transactional safety. There is no guarantee that a data point is not deleted the moment after it was retrieved (data point object of a deleted data point is not usable).

void setWriteEnabled(boolean writeEnabled)
Enables or disables all writes to data points.

void close();
Closes connections and cleans up resources.

Below are the C++ CiiOldb API listings. The Python client uses Python bindings and has therefore the same API as the C++ client.

Listing 8‑2: C++ CiiOldb API

std::shared_ptr<CiiOldbTypedDataBase> CreateDataPoint(const ::elt::mal::Uri& uri, const std::string& meta_data_instance_name);

template<typename T>
 std::shared_ptr<CiiOldbDataPoint<T>> CreateDataPointByValue(
const ::elt::mal::Uri& uri, const T& value);

template<typename T>
std::shared_ptr<CiiOldbDataPoint<std::vector<T>>> CreateDataPointByValue(
const ::elt::mal::Uri& uri, const std::vector<T>& value, bool is_matrix);

std::shared_ptr<CiiOldbTypedDataBase> GetDataPoint(const ::elt::mal::Uri& uri) const;

template<typename T> std::shared_ptr<CiiOldbDataPoint<T>> GetDataPoint(
const ::elt::mal::Uri& uri, ::elt::common::CiiBasicDataType data_type) const;

std::vector<std::shared_ptr<CiiOldbTypedDataBase>> GetDataPoints(
const std::vector<::elt::mal::Uri>& uris) const;

template<typename T> std::vector<std::shared_ptr<CiiOldbDataPoint<T>>> GetDataPoints(const std::vector<::elt::mal::Uri>& uris,
::elt::common::CiiBasicDataType data_type, T min_value, T max_value) const;

std::map<::elt::mal::Uri, bool> GetChildren(const ::elt::mal::Uri& uri) const;

static void SetWriteEnabled(bool enabled) noexcept;

static bool IsWriteEnabled() noexcept;

void DeleteDataPoint(const ::elt::mal::Uri& uri);

7.13.2. CiiOldbDataPoint

CiiOldbDpValue<T> ReadValue(bool check_bad_quality = true)

Returns a copy of this Data Point value.

Every call to ReadValue retrieves an up to date value from the OLDB, but the CiiOldbDpValue object is a snapshot at the time of retrieval and does not update automatically. The user should call ReadValue to get the updated value. To retrieve a datapoint with a not OK quality, ‘check_bad_quality’ argument should set to false.

void WriteValue(T value, int timestamp = Now(), bool is_disable_publishing = false)
Writes the value to the data point. Timestamp is UTC in nanosecs since epoch,The quality is unchanged and timestamp is by default now.

void WriteValue(T value, int timestamp, CiiOldbDpQuality quality, bool is_disable_publishing = false, string overridden_server_alias = nullptr)
Writes all three fields (value, time and quality) to the data point. Timestamp is UTC in nanosecs since epoch. ‘is_disable_publishing’ is used in case the user only need a notification and not a the new value. ‘overridden_server_alias’ allows changing the data point server configuration and transfering the value from default data provider to external or vice versa

void WriteValue(T value, int size, int timestamp = Now())
Writes the value with a specific istream size and timestamp to the data point. The quality is unchanged. Timestamp is UTC in nanosecs since epoch.

void Subscribe(CiiOldbDpSubscription listener,: bool answer_immediately = false)

Subscribes an object listener to the data point to listen to value changes.

NOTE: when answer_immediately = true, meaning that the user provided listener will be called immediately with the data point value, the Subscribe call will be slower for the bigger data points since the whole value needs to be read, otherwise only a “header” for the data point is read.

void Unsubscribe(CiiOldbDpSubscription listener)
Unsubscribes an object listener from the data point

void setQuality(CiiOldbDpQuality quality)
Sets the data point quality.

<U extends MdOldb<T>> U getMetadata(Class<U> clazz)
Returns the metadata of the data point. Caller must provide the metadata class (extends MdOldb).

<U extends MdOldb> void setMetadata(String metadataInstanceName)
Updates the data point metadata with the metadata which instance name is provided. The changes made in the metadata must be saved to config service otherwise this call will have no effect. If the instance name is different from the one this data point already has (new metadata instance name has been assigned to the data point), the new metadata must respect the data point type.

Below are the C++ CiiOldbDataPoint API listings. The Python client uses Python bindings and has therefore the same API as the C++ client.

Listing 8‑4: C++ CiiOldbDataPoint API

std::shared_ptr<CiiOldbDpValue<T>> ReadValue(bool check_bad_quality = true);

void WriteValue(const T& value, int64_t timestamp = CiiOldbUtil::Now());

void WriteValue(const T& value, int64_t timestamp, CiiOldbDpQuality quality,
const std::string *overridden_server_alias = nullptr,
const std::size_t *overriden_value_size_limit_hdfs = nullptr);

void WriteValue(std::istream& value, std::uint64_t size, int64_t timestamp = CiiOldbUtil::Now());

void WriteValue(std::istream& value, std::uint64_t size, int64_t timestamp,
CiiOldbDpQuality quality);

void Subscribe(const std::shared_ptr<CiiOldbDpSubscription<T>>& listener);

void Unsubscribe(const std::shared_ptr<CiiOldbDpSubscription<T>>& listener);

std::shared_ptr<::elt::config::classes::meta::MdOldb<T>> GetMetadata() const;

void SetMetadata(const std::string& meta_data_instance_name);

void SetQuality(CiiOldbDpQuality quality) override;

elt::common::CiiBasicDataType GetType() const override;

7. Online Database

7.1. Overview

7.2. Introduction

7.3. Prerequisites

7.3.1. WAF Modules

7.3.2. Services

7.3.2.1. Redis

7.3.2.2. Calculation Engine

7.3.2.3. Configuration Service

7.3.3. Configuration

7.3.3.1. OLDB Client Configuration

7.3.3.2. Calculation Service Configuration

7.4. OLDB Library Usage

7.4.1. Includes/Imports

7.4.1.1. C++

7.4.1.2. Python

7.4.1.3. C++

7.4.1.4. Python

7.4.2. Basic Example

7.4.3. Subscribing Example

7.4.4. Advanced Example

7.4.5. Alias Example

7.5. Advanced Topics

7.5.1. OLDB Statistics

7.5.1.1. OLDB Client Statistics

7.5.1.2. Calculation Service Statistics

7.5.2. Disabling Writes to Data Points

7.5.3. Custom functions in Data Point Formulas

7.5.4. Manipulating Data Point Metadata

7.5.5. Data Point Configuration

7.5.6. External Redis Servers

7.5.7. Defining the Storage Location of a Data Point

7.5.8. Enabling notification-only subscriptions on datapoints

7.6. OLDB API LIBRARY

7.6.1. CiiOldb

7.6.2. CiiOldbDataPoint

7.6.2.1. Data Point URI

7.6.2.2. Data Point Value

7.6.2.2.1. Value

7.6.2.2.2. Quality

7.6.2.2.3. Timestamp

7.6.2.3. Metadata

7.6.2.3.1. Data Point Type

7.6.2.3.2. Data Point Formula

7.6.2.3.3. Data Point Quality Expression

7.6.2.3.4. Data point value aging

7.6.2.3.5. Data point heart beats

7.7. OLDB CLI

7.7.1. Read operations

7.7.1.1. Read value

7.7.1.2. Read Value Formula and Quality Expression

7.7.1.3. Save value to File

7.7.2. Write operations

7.7.2.1. Write value

7.7.2.2. Write Quality and Timestamp

7.7.2.3. Write Formula

7.7.2.4. Write Quality Expression

7.7.2.5. Write Value from File

7.7.2.6. Write Value directly from Formula

7.7.3. Subscribe operations

7.7.3.1. Subscribe to datapoint

7.7.3.2. Subscribe and output to Files

7.7.4. Create operation

7.7.5. Delete operation

7.8. GUI

7.8.1. General

7.8.2. Configuration File

7.8.3. Functionality

7.8.3.1. Main Window

7.8.3.2. Menu

7.8.4. Hierarchy view

7.9. Data Point Value Type to Language Types Mapping

7.10. Default Metadata Instance Names

7.11. Metadata attribute constraints

7.12. YAML Data point type

7.13. OLDB API LISTING

7.13.1. CiiOldb

7.13.2. CiiOldbDataPoint