Telemetry Recorder

Overview

The Telemetry Recorder is a reusable component that needs to be customised both at compile-time and run-time. A tutorial on how to implement a specific recorder is provided in the Telemetry Recorder Tutorial.

Basic Operation

Usually, a new recording session is started with command Run and stopped with command Idle.

Once a recording session is started and the component is in state On:Operational:Running it will ingest data from various sources and write them to a set of output files located in folder $DATAROOT/component_id/session_id/.

In case the recorder is configured to stop recording on its own it will automatically transition to state On:Operational:Idle once a configurable stop-condition is met. In this case it is not necessary to send the Idle command.

In case a recording session fails the component will transition to state On:Operational:Error which can be left using command Recover.

Common Recording Infrastructure

The Telemetry Recorder and also the Metadata Collector make use of the same common recording infrastructure.

Recording Units

Telemetry Recorder and Metadata Collector components hosts one or more Recording Units, where each unit ingests data from a specific source and writes them into a corresponding FITS file. Note that the output file name corresponds to the unique identifier of the recording unit.

All recording units extend the base class RecordingUnit that defines a common interface and provides basic functionality for all recording unit types.

Common functionality implemented in the base class includes publishing the recording unit state to OLDB and implementing leader follower functionality that enables follower recording units to react to state changes of leader recording units.

To customise a recorder, recording units of different types are constructed and passed to the component via dependency injection. Recording unit configuration happens both at compile-time and at run-time.

The following compile-time configuration options are required by all units:

Parameter	Type	Description
component_id	string	name of the component the unit belongs to
unit_id	string	unique identifier of the recording unit
services	object handle	access to various services

The run-time configuration for these recording units can be found under “/component_name/unit_name” in the Runtime Configuration Repository. Depending on its type, each RecordingUnit can have different settings in this location. All RecordingUnits support the following settings:

Parameter	Type	Description
leader_list	static	list of other recording units that can trigger and stop the recording of this unit

Currently the RTC Toolkit supports the following Recording Units:

• IpcqRecordingUnit for recording IPCQ shared memory topics into FITS binary tables

• DataPointRecordingUnit for recording datapoints from the runtime repository into FITS binary tables

• JsonEventRecordingUnit and TypedEventRecordingUnit for recording events

Note that instrument RTC developers may also implement custom recording units if necessary.

IpcqRecordingUnit

This type of recording unit listens to an IPCQ shared memory topic and writes data sample to its backend.

At compile-time the unit needs to be parametrised with the Shared Memory Topic Type.

At run-time the unit supports the following configuration options:

Parameter	Type	Description
shm_queue_name	static	name of the ipcq
cpu_affinity	static	(optional) CPU affinity setting for the thread that reads from the IPCQ and writes to disk.
telemetry_subset	static	(optional) Only record the given parts of the topic.
subsample_factor	dynamic	Defines how many samples shall be skipped for each read sample.
start_at_sample_id	dynamic	This lets the recording unit wait for a minimum sample_id, before starting the recording and triggering its followers.
stop_after_num_samples	dynamic	This lets the recording unit stop automatically after having recorded the given amount of samples.

The following is an example of an IpcqRecordingUnit configuration in YAML format:

static:
  rec_units:
    ipcq_unit_1:
      shm_queue_name:
        type: RtcString
        value: exampleTelRecQueue
      cpu_affinity:
        type: RtcInt32
        value: 0
      telemetry_subset:
        type: RtcVectorString
        value:
            - intensities
            - slopes
dynamic:
  rec_units:
    ipcq_unit_1:
      subsample_factor:
        type: RtcInt64
        value: 10
      start_at_sample_id:
        type: RtcInt64
        value: 0
      stop_after_num_samples:
        type: RtcInt64
        value: 10

DataPointRecordingUnit

This kind of recording unit is used to record datapoints from the runtime repository.

At compile-time the unit needs to be parametrised with the Datapoint Type and the Datapoint Path (location of the datapoint in the repository).

At run-time the unit supports the following configuration options:

Parameter	Type	Description
capture_mask	static	bit mask to configure wheter data should be captured on_start, on_stop or on_change

The following is an example of a DataPointRecordingUnit configuration in YAML format:

static:
  rec_units:
    dp_unit_2:
      leader_list:
        type: RtcVectorString
        value:
            - ipcq_unit_1
      capture_mask:
        type: RtcInt32
        value: 3

The datapoint recording unit is configured to follow ipcq_unit_1 and to capture the datapoint value on_start (Bit 1) and on on_stop (Bit 2). To also capture data on_change Bit 3 would also need to be set, this would lead to configuration value 7.

JsonEventRecordingUnit and TypedEventRecordingUnit

These recording unit types are used to record events that are emitted by various components.

While the more low-level JsonEventRecordingUnit can record arbitrary JSON events, the more high-level Typed Event Recording Unit can record specific event types.

At compile-time the JsonEventRecordingUnit needs to be parametrised with a fixed TopicName, the TypedEventRecordingUnit needs to be parametrised with a specific EventType. In addition, both units allow the provision of a filter function that can be used to filter out unwanted samples.

Currently these units do not have any run-time configuration options.

Data Recorder

This class is an abstract class that can be used to implement an OutputStage for a Recording Unit. It provides an interface to open and close files, write a tuple and allows removing parts of the tuple from the output. This is used when only a subset of the data should be written. Currently, the only derived class is the FitsRecorder class, that implements this interface to write to a FITS binary table.

FitsRecorder

The FitsRecorder class creates a FITS file containing a single binary table. It is a templated class where the template arguments correspond to the data types to be stored in the binary table. The supported types are:

• std::string

• bool

• uint8_t

• int8_t

• int16_t

• uint16_t

• int32_t

• uint32_t

• int64_t

• float

• double

And using the above types in any of these containers:

• std::vector

• std::array

• gsl::span

• MatrixBuffer

The constructor takes 3 arguments, a discription of the columns (column name and the unit), a list of disabled outputs (can be used to disable output of some columns at runtime), and the name of the FITS binary table extension

Limitations and Known Issues

At this point, the recording infrastructure is not optimised for high performance, there is still room for optimisation to make recording viable at fast loop rates.

Currently, the only output format are FITS binary tables. Additional recording unit types that can record FITS keywords will be added in a sub-sequent release.

Not all event types are supported at the moment e.g. support for certain configuration and coordination event types is still missing. Please use the generic JsonEventRecordingUnit to record such events until the missing parts are implemented in a sub-sequent version.