Persistent Configuration Repository¶

All configuration for SRTC components should be stored in the Persistent Configuration Repository. The purpose is to store all possible configurations for each Deployment Set centrally and in a persistent manner. The user normally prepares all necessary configuration in the Persistent Configuration Repository rather than interacting with the Runtime Configuration Repository. The configuration values for the active Deployment Set are then copied from the Persistent Configuration Repository to the Runtime Configuration Repository automatically by the RTC Supervisor.

Modifying the Configuration¶

The user should typically use rtctkConfigTool to interact with the configuration in the Persistent Configuration Repository. See section Configuration Tool for a general description of how to use the tool. Alternatively, if a file based or a local CII configuration is being used, i.e. file: or cii.config://local URI schemes, the underlying files stored on disk can be accessed directly to prepare the configuration. In both cases the YAML file format was chosen to make direct manipulation easier. However, the CII configuration service uses a different YAML schema than the file based adapter. In addition, for the file based adapter, external FITS files can be used for large vectors and matrices. A URI to the external FITS file can then be embedded into the YAML file for a given datapoint. This URI embedding support is lacking for the time being in the CII configuration service.

The following sub-sections will give brief examples that are more specific to manipulating the Persistent Configuration Repository with rtctkConfigTool.

Datapoint Paths¶

Configuration parameters are stored in a tree hierarchy as leaf nodes in the Persistent Configuration Repository. This can also be thought of as a folder like structure, similar to a file system. The nodes from the root of the tree to a particular leaf node form the components of a path. By adding the ‘/’ character as the path separator between each component, this forms a datapoint path string.

The current canonical structure for the path is one of the following:

/<deployment>/defaults/<component>/{static,dynamic}/<parameter>
/<deployment>/modes/<name>/<option>/<component>/{static,dynamic}/<parameter>

Where <deployment> is the name of a Deployment Set;
<name> is the name of a mode;
<option> is the name of one of the possible mode options;
<component> will typically be the SRTC component instance name; and
<parameter> can represent a hierarchical sub-path with multiple sub-folders if a deeper hierarchy of configuration parameters is desired for a particular SRTC component, besides the basic grouping into static and dynamic parameters.

Path components must contain only lowercase alpha numeric characters or the underscore, i.e. characters from the set [a-z0-9_].

Datapoint Reading¶

The following is an example of reading a parameter called flatimg, which is a matrix, into a local file called flatimg.fits for the component datatask1:

$ rtctkConfigTool --persistent-repo-endpoint file:/home/eltdev/persistent_repo \
         read persistent /depl1/defaults/datatask1/static/flatimg flatimg.fits

In this case, the location of the repository in the local file system is /home/eltdev/persistent_repo, i.e. this is the base directory.

Datapoint Writing¶

The following is an example of writing the contents of a local FITS file called flatimg.fits, which contains a matrix of floating-point doubles, into a parameter called flatimg for the component datatask1:

$ rtctkConfigTool --persistent-repo-endpoint file:/home/eltdev/persistent_repo \
         write persistent /depl1/defaults/datatask1/static/flatimg flatimg.fits

It is possible to subsequently manipulate the data directly in the local file system, in this case, under the location /home/eltdev/persistent_repo.

If the datapoint does not exist it is created automatically. Otherwise it is updated with the new value.

Datapoint Querying¶

Use the following command to list all datapoints or sub-paths under a certain path, in this case /datatask1/static:

$ rtctkConfigTool list persistent /depl1/defaults/datatask1/static

The data type for a particular datapoint can be queried with the following command:

$ rtctkConfigTool get persistent /depl1/defaults/datatask1/static/flatimg --type

Note

Since the --persistent-repo-endpoint option is not used in the above examples, this would cause the command to use the URI as given in the configuration for rtctkConfigTool itself or from any environment variable overrides.

Configuration Layout¶

The Persistent Configuration Repository is structured into one or more Deployment Sets. Each Deployment Set is placed into its own top level sub-folder. This is further divided into two folders:

defaults: contains a hierarchy of datapoints with default values.
modes: contains datapoint values for one or more modes that each have one or more options that override the default values.

Therefore the path structure for the default values follows:

/<deployment>/defaults/<component>/{static,dynamic}/<parameter>

The hierarchy under the defaults sub-folder is copied as is to the root of the Runtime Configuration Repository. Therefore a datapoint /<deployment>/defaults/<component>/{static,dynamic}/<parameter> will land up in /<component>/{static,dynamic}/<parameter> when the Runtime Configuration Repository is populated, assuming <deployment> is the current active Deployment Set.

The path structure for modes follows:

/<deployment>/modes/<name>/<option>/<component>/{static,dynamic}/<parameter>

Where each sub-folder under /<deployment>/modes is a separate mode. The mode name is given by the sub-folder name, i.e. <name>. Each mode should then have one or more options indicated by further sub-folders. Again, the name of the option corresponds to the sub-folder name, i.e. <option>. The hierarchy under the /<deployment>/modes/<name>/<option>/ path will the copied as is to the Runtime Configuration Repository if that mode option is selected, replacing any initial default values loaded from /<deployment>/defaults/.

For each mode, the default option that should be loaded initially by RTC Supervisor is indicated by the following datapoints:

/<deployment>/modes/<name>/default_option

The value for each of these datapoints should be a string indicating one of the <option> sub-folders corresponding to the mode, i.e. one of /<deployment>/modes/<name>/<option>. If the default option is not set for the mode, then the mode is not activated when initially populating the Runtime Configuration Repository, i.e. no default values are replaced by that mode.

The currently selected active Deployment Set that the RTC Supervisor should load is indicated by the datapoint with the following path:

/active_deployment

The value of /active_deployment must be a string indicating the name of one of the top level Deployment Set folders, i.e. one of the <deployment> folder names.

Two additional top level datapoint boolean flags are available. The following will disable automatic loading of the active Deployment Set by the RTC Supervisor:

/disable_populate_runtime_repo

While the following flag will disable handling of Deployment Sets by RTC Supervisor and instead perform a 1-to-1 copy of the Persistent Configuration Repository datapoint hierarchy to the Runtime Configuration Repository:

/simple_populate_runtime_repo

When /simple_populate_runtime_repo is set to true, the Deployment Set path structure indicated previously becomes irrelevant. In this case, the datapoints in the Persistent Configuration Repository should follow the following path structure instead (i.e. same structure as for the Runtime Configuration Repository):

/<component>/{static,dynamic}/<parameter>

Service Discovery Endpoint¶

The currently supported URI schemes for the service discovery parameter persistent_repo_endpoint are:

cii.config: This will select the repository adapter that interfaces with the CII configuration service. Only local is supported for the hostname part of the URI. For example cii.config://local/rtc/config.
file: This will select the simple file based repository adapter, suitable for early development and testing. For example file:/home/eltdev/persistent_repo.

Data Access Using the API¶

Note

SRTC components should not access the Persistent Configuration Repository directly. Only a limited subset of components and tools should use the API to modify the Persistent Configuration Repository. See the section Modifying the Configuration for details about the typical manner to update the Persistent Configuration Repository.

The API for accessing datapoints is identical to the Runtime Configuration Repository. The only differences are that the Persistent Configuration Repository does not support the datapoint subscription API, and the interface class to use is the PersistentRepoIf, which is fetched from the Service Discovery as follows:

auto oldb = m_services.Get<PersistentRepoIf>();

Refer to the Data Access sub-section of Runtime Configuration Repository for details about the methods for creating, reading, writing, deleting and querying datapoints.

Supported Data Types¶

The same data types are supported as for the Runtime Configuration Repository. Refer to the Supported Data Types sub-section of Runtime Configuration Repository for further details.

File Format¶

The current file based Persistent Configuration Repository implementation, i.e. using the file: scheme, uses the same internal file format as the file based Runtime Configuration Repository. Refer to the File Format sub-section of Runtime Configuration Repository for further details.

The following relates to the YAML file format relevant when using the Persistent Configuration Repository adapter for the CII configuration service and for the cii.config://local URI prefix (i.e. local configuration). The detailed description of the syntax and options available can be found in the CII Configuration manual. Here we only highlight important information that is specific to the RTC Toolkit.

Layout with Deployment Sets¶

The Persistent Configuration Repository is mapped to a number of individual YAML documents in the CII configuration service. By default these file are encoded as Deployment Sets and placed into separate sub-directories. Specifically, the files are kept in the following directory structure on the filesystem:

<repodir>/<deployment>/defaults/<component>.yaml - The initial default datapoints for a Deployment Set are placed in a YAML file per component under a defaults directory.
<repodir>/<deployment>/modes/<name>/<option>/<component>.yaml - The datapoints related to a particular mode are placed in a YAML file per component under the indicated directory structure.
<repodir>/globals.yaml - Any datapoints that do not map to the above document locations are placed in this file.

Where <repodir> is simply the top level filesystem directory where the repository is located. This would normally be configured through the CFGPATH environment variable. See the CII configuration service documentation for more details.

Within each YAML file a hierarchical structure of YAML maps (dictionaries) is used for the remaining logical path components of a datapoint’s path. The leaf nodes then store the datapoint’s value and type information. For example, the datapoint path /depl1/defaults/datatask1/static/param1 maps to the file <repodir>/depl1/defaults/datatask1.yaml with the following contents:

static:
    param1: !cfg.type:int64 123

As can be seen, the last path components of a datapoint’s logical path are exactly the dictionary key names within the YAML file.

For a more complete example of the mapping between datapoint paths and the values in the YAML files, consider the following datapoints:

/depl1/defaults/datatask1/static/param1
/depl1/defaults/datatask1/static/param2
/depl1/defaults/datatask1/dynamic/matrix
/depl1/modes/mode1/opt1/datatask1/static/param1
/depl1/modes/mode1/opt2/datatask1/static/param1
/depl1/modes/mode1/default_option
/depl1/defaults/datatask2/static/settings/param_a
/depl1/defaults/datatask2/static/settings/param_b
/active_deployment
/simple_populate_runtime_repo

These would correspond to the following files and directories:

<repodir>/depl1/defaults/datatask1.yaml
<repodir>/depl1/defaults/datatask2.yaml
<repodir>/depl1/modes/mode1/opt1/datatask1.yaml
<repodir>/depl1/modes/mode1/opt2/datatask1.yaml
<repodir>/globals.yaml

The contents of these files would be as follows:

Listing 5 <repodir>/depl1/defaults/datatask1.yaml¶

 static:
     param1: !cfg.type:int64 123
     param2: !cfg.type:string "abc"
 dynamic:
     matrix: !cfg.type:matrix2d_single [[1.0, 2.0], [3.0, 4.0]]

Listing 6 <repodir>/depl1/defaults/datatask2.yaml¶

 static:
     settings:
         param_a: !cfg.type:int32 80
         param_b: !cfg.type:double 2.125

Listing 7 <repodir>/depl1/modes/mode1/opt1/datatask1.yaml¶

 static:
     param1: !cfg.type:int64 1024

Listing 8 <repodir>/depl1/modes/mode1/opt1/datatask1.yaml¶

 static:
     param1: !cfg.type:int64 2048

Listing 9 <repodir>/globals.yaml¶

 depl1:
     modes:
         mode1:
             default_option: !cfg.type:string "opt1"
 active_deployment: !cfg.type:string "depl1"
 simple_populate_runtime_repo: !cfg.type:boolean False

Simple Layout without Deployment Sets¶

When the /simple_populate_runtime_repo flag is set to true, a simplified directory structure is used for the Persistent Configuration Repository. No Deployment Sets are encoded and the Persistent Configuration Repository is assumed to represent a 1-to-1 copy of the Runtime Configuration Repository. Therefore, the structure is a flat list of YAML files, one per component, e.g. <repodir>/<component>.yaml. Within each YAML file a nested dictionary structure is used as usual. Therefore, a file <repodir>/<component>.yaml in the simple layout can in principle be identical to <repodir>/<deployment>/defaults/<component>.yaml from the case where Deployment Sets are used.

External URIs¶

It is possible to encode vectors or matrices as FITS files and refer to them via an external URI in the YAML, instead of embedding them into the YAML file directly. This can be particularly useful for large vectors or matrices, which can be handled more conveniently as FITS files.

Note

Only datapoints for numerical vectors and matrices are supported using external URIs.

To change a datapoint from being encoded directly in the YAML, simply replace the type tag with the special type !cfg.type:uri and value with the external URI string. The uri type must be declared by adding the definition !cfg.typedef uri(string): once to the top of the file. The following is an example of the needed changes:

!cfg.typedef uri(string):

dynamic:
    matrix: !cfg.type:uri "file:matrix.fits"

The URI must use the file: scheme and must refer to a FITS file in the local filesystem.

The path can be a relative or absolute filesystem path. If a relative path is used then it is relative to the location of the YAML file in which the external URI is encoded. For example, if the YAML file’s path is /home/eltdev/repo/depl1/defaults/datatask1.yaml and the encoded external URI in the YAML is file:matrix.fits, then the path of the FITS file must be /home/eltdev/repo/depl1/defaults/matrix.fits. Alternatively, if the encoded external URI is file:subdir/matrix.fits, then the path of the FITS file must be /home/eltdev/repo/depl1/defaults/subdir/matrix.fits.

The format of the FITS file must be a 1-D or 2-D image in the primary HDU. Appropriate encoding of the NAXIS, BITPIX, BZERO and BSCALE keywords will determine the datapoint type according to the following table:

C++ Type	NAXIS	BITPIX	BZERO	BSCALE
`std::vector<int8_t>`	1	8	-128	1
`std::vector<uint8_t>`	1	8	(none)	(none)
`std::vector<int16_t>`	1	16	(none)	(none)
`std::vector<uint16_t>`	1	16	32768	1
`std::vector<int32_t>`	1	32	(none)	(none)
`std::vector<uint32_t>`	1	32	2147483648	1
`std::vector<int64_t>`	1	64	(none)	(none)
`std::vector<uint64_t>`	1	64	9223372036854775808	1
`std::vector<float>`	1	-32	(none)	(none)
`std::vector<double>`	1	-64	(none)	(none)
`MatrixBuffer<int8_t>`	2	8	-128	1
`MatrixBuffer<uint8_t>`	2	8	(none)	(none)
`MatrixBuffer<int16_t>`	2	16	(none)	(none)
`MatrixBuffer<uint16_t>`	2	16	32768	1
`MatrixBuffer<int32_t>`	2	32	(none)	(none)
`MatrixBuffer<uint32_t>`	2	32	2147483648	1
`MatrixBuffer<int64_t>`	2	64	(none)	(none)
`MatrixBuffer<uint64_t>`	2	64	9223372036854775808	1
`MatrixBuffer<float>`	2	-32	(none)	(none)
`MatrixBuffer<double>`	2	-64	(none)	(none)

Entries indicated as (none) in the above table mean that the keyword does not need to be present in the FITS file header.

Limitations and Known Issues¶

If using the file based Persistent Configuration Repository adapter, i.e. using the file: URI scheme, then the same limitations currently apply as the file based Runtime Configuration Repository. Refer to the Limitations and Known Issues sub-section of Runtime Configuration Repository for further details.

When using the Persistent Configuration Repository backed by the CII configuration service, i.e. using the cii.config://local URI scheme, then the following limitations apply:

The data type std::string (i.e. RtcString) and any std::vector<T> type (i.e. RtcVector<T>) currently are only supported in read-only mode. When attempting to write to the datapoint via the API the call will fail.
The data type MatrixBuffer<std::string> (i.e. RtcMatrixString) is not currently supported.
Due to a bug in the CII configuration service API, the !cfg.include inclusion mechanism will currently cause unexpected errors if used.
Embedded URIs to external FITS files are currently only supported in read-only mode. This means that a user can modify the YAML by hand to add the URIs, but it is not possible to use the PersistentRepoAdapter to create, write or delete such datapoints. It is therefore also only possible to read a vector or matrix datapoint with rtctkConfigTool that uses the external URI mechanism.
The adapter for the CII configuration service still needs optimisation and may not currently be very performant.
It is not possible to use a datapoint path that is also the directory name for another datapoint, i.e. a datapoint path cannot be the exact string prefix of another datapoint’s path. For example, the following two paths cannot be used at the same time: /depl1/defaults/dt1/static/sub and /depl1/defaults/dt1/static/sub/.
There is no locking mechanism implemented in the CII configuration service. Therefore users must avoid having one or more processes attempt to read from the Persistent Configuration Repository while it is being modified.