Overview

The CCF project is composed by the following modules/packages:

These packages provide the following functionality:

mptk: Multiprocessing Toolkit: A small collection of classes to manage a multithreaded environment, including inter-thread synchronisation and communication.

common: Common Tools & Utilities: A set of common utilities used for implementing CCF applications, e.g. an Image Frame Class, a Frame Queue, DB access class, and more.

control: CCF Control Application: Provides implementation of the CCF Control application class and classes/callbacks for handling requests, state machine related activities and implementations of the various types of threads, running in a CCF Control application.

sim: CCF Control Simulation Deployment Module: Deployment module for a CCF Control Application, providing simulation only.

tools: Miscellaneous Tools: It provides the following features:

ccfclib: Python synchronous and asynchronous client library.

cli: Interactive CCF Control client shell (based on “ccfclib”).

pycli: Deprecated: May be removed in the future since the “ccfclib” and “cli” provide the same functionality.

dcsSend: C++ client library and command line utility.

ccfFitsCubeGen: Small tool to generate FITS cubes that can be used as input for the CCF Simulator.

rtms2ccflib: Library, handling the conversion of image frames, sent via RTMS to the internal CCF format.

stdrecipe: Standard Processing Recipes: In this version only a centroiding recipe is provided.

stdpub: Standard Data Publishers: In this version a basic FITS CCF Data Publisher, a DDT Data Publisher and an RTMS Publisher are provided.

protocols: Specific Communication Adapters & Deployment Executables: Provides implementation of CCF Communication Adapters for various protocols and the associated deployment executable for these. Note, in many cases it will be needed to provide context specific data processing and thus a context specific CCF Control Application must be generated, based on the context specific adapters.

gui: CCF Engineering GUI: Tool used to control an associated CCF instance and to inspect the status of the instance.

In the design of the component, it was foreseen to provide a separate simulator process. This design choice has been changed to providing simulation via a Communication Adapter. It is the intention to provide simulation based on this scheme until further.

Moreover, the design contains a proposal for a Super DCS Application. This has not yet been implemented and will only be assessed if specific requests for such a service will be submitted.

Architecture

The aim of the CCF Package as such, is not to so much to provide a ready-to-use set of CCF application executables, but rather to facilitate easy integration of customised (context specific) DCS solutions.

To achieve this, the CCF SDK is based on the adapter pattern to allow for context specific customisation.

The adapter types provided are:

Communication Adapter: Implements protocol specific communication.

Processing Recipe Adapter: Implements on-the-fly processing of image frames, executed in a dedicated thread in a CCF Control instance.

Data Publisher Adapter: Implements publishing of data to specific targets, which may be output files or applications subscribing to the data.

Monitoring Adapter: Will be executed periodically by the Monitoring Thread and can be used to collect run-time information about the system and monitor the health of the associated camera device.

Procedure Adapter: Provides methods that can be invoked via the “dcsif::Execute” request.

The anatomy of a CCF Control Application is depicted below.

To use the CCF it is not required to have a very detailed/low-level insight into the architecture. Nonetheless, a basic understanding is needed, and useful, to be able to use it and to implement, deploy and test the various adapters needed.

The internal threads executed in a CCF Control Application are:

RAD Event Handler Thread: The main thread in the application, which handles e.g. incoming requests.

Monitor Thread: The purpose of the Monitor Thread is to supervise the operation of the CCF Control instance. In particular it collects periodically, data about the execution of the various other CCF threads, which is used to compute statistics, which is written in the OLDB. It also invokes the Monitoring Adapter.

The Monitor Thread also handles the Telemetry Sampling, which is used to read values from the associated camera and write these to the OLDB.

Acquisition Thread: The Acquisition Thread, as indicated by the name, is responsible for acquiring the image frames from the associated camera. Once a frame has been received, the image data (pixels) and metadata information is stored in the Input Queue and the Processing Thread(s) informed about the availability of a new data frame. Note, at this point in time, it is only possible to receive complete frames and not to handle sub-frames.

Processing Thread(s): The Processing Threads applies processing in the form of one or more Processing Recipes (adapters) on the data. When informed about the availability of a new frame in the Input Queue, it reserves an image buffer in its associated Output Queue and applies the Recipe(s) specified on the image. Once the processing of an image frame has completed, the Processing Thread informs the associated Publisher Threads about the availability of a new, processed image. If no Recipes are specified (configured), the image frames are merely copied from the Input Queue buffer into the Output Queue buffer and the associated Publisher Thread(s) notified. The number of Processing Threads to execute, is configurable.

Publisher Thread(s): The Publisher Threads send the image data available in the associated Processing Thread Output Queue to the specified target(s). This will typically be files or e.g. the DDT. However, as the actual publishing is carried out by a Data Publisher Adapter, other types of targets are possible.

A Processing Thread with its associated Recipes and Publishers, is also referred to as a “Pipeline” in the context of CCF.

When inspecting the threads running in a CCF Control instance, e.g. with “htop”, the user will observe more threads than the ones mentioned above. These are threads deployed by CII and e.g. by an SDK used for communicating with the camera.

In total, the main internal elements composing a CCF Control Application are:

Communication Adapter: The Communication Adapter used for communicating with the associated camera. As this adapter may be provided by the user, it could be used for other purposes, or to collect data from other sources than cameras.

Simulation Communication Adapter: The Communication Adapter used for communicating with the simulator. For now, the provided Simulation Communication Adapter actually, is responsible for the simulation and is not communicating with an external simulator. If of relevance, a context specific Simulation Communication Adapter may be provided, either generating internally the simulated data or communicating with an external simulator.

Monitor Adapter: The Monitor Adapter can be implemented to carry specific monitoring, or control of the associated device. It can also handle other tasks, which need to be executed on a periodic basis for a given installation. The Monitor Adapter is invoked by the Monitor Thread. In the configuration it can be specified how frequently to execute a registered Monitor Adapter. It is possible to specify not to invoke the Monitor Adapter while the CCF instance is in Acquisition State, to avoid interfering with an on-going acquisition.

Procedure Adapter: The Procedure Adapter can be used to add context specific business logic to a CCF Instance. This specific logic is added in the form of procedures, which may take one or more input parameters. If a requested procedure is defined in the Procedure Adapter, it can be executed via the “dcsif::Execute()” command. The output from the procedure is returned to the caller (client) as a string.

Recipe Adapter(s): Recipe Adapters can be invoked to carry out on-the-fly processing on the images, in the Processing Thread.

Publisher Adapter(s): The Publisher Adapters are invoked to publish the possibly processed images, to the required targets, e.g. DDT or FITS files. The publishing of the data is done via the Publisher Threads.

Input Queue: Queue containing image buffer objects (“ccf::common::DataFrame”), with a preallocated set of buffers, used in a ring-buffer fashion by the Acquisition Thread to store the incoming frames, and make these available for the configured Processing Thread(s). The number of buffers in the Input Queue is configurable.

Output Queue(s): For each Processing Thread, one Output Queue is allocated to hand over processed data to the Publisher Threads. The elements in the Output Queue(s) are pre-allocated instances on the “ccf::common::DataFrame” class, which are re-used in a ring-buffer fashion. The number of image buffers in the Output Queue is configurable.

The external entities shown in the figure above, are as follows:

Pub/Sub: The Pub/Sub service supported by CII MAL. For this version, this is only used for publishing the state of the application, in accordance with the Standard Application Interface (“stdif”).

OLDB: The online database supported by the CII Services. The is the base line for CCF to publish information for the client applications. An overview of the information updated in the OLDB, is provided in one of the following sections.

Client: A client may be e.g. the Sequencer, a GUI or a command line tool, used to command the CCF Control instance.

Configuration: The main configuration file is based on the CII Configuration Service (YAML format). It is referenced when starting the application. The configuration is loaded and applied when the application is starting up, and applied to the associated camera during initialisation. The main configuration refers to a number of other files, which are loaded and applied as well, when the application starts up and initialises.

Initialisation Setup: Contains the default setup when the application starts up (YAML format). This is referenced by the main configuration. It is based on the CII Configuration Service.

Camera Name Mapping: Contains a mapping from the parameter names used internally in the Communication Adapter and the names defined in the camera name space.

Metadata Name Mapping: Maps internal names for metadata, used in the CCF Control Application to the user metadata names (not used for this release).

Dictionaries: Contains the definition of the keywords to be written in the output data products, normally FITS files.

Simulation Data File: The Simulation Data File used as basis for the simulation. Typically this will be a FITS image cube, providing a sequence of images emulating the usage on sky. The user may provide use case specific Simulation FITS Files, to obtain a realistic emulation. An example Simulation Data File is provided with the CCF Package (“ccf/control/resource/image/ccf/control/testCube16Bit.fits”). If a context specific simulation is provided, e.g. in the form of a Communication Adapter, this adapter may define how to obtain the data used as input for the simulation. A small utility is provided (on a prototype basis) to generate Simulation Data File, FITS image cubes (“ccfGenFitsCube”).

Output Data Product Files: Output files generated by a Data Publisher. Normally this will be FITS files, but if required, it will be possible to provide context specific Data Publishers to generate other formats, e.g. for more efficient storage in ‘burst mode’. A standard FITS Data Publisher is provided by the CCF Package.

DDT Broker/Viewer: The standard DDT image transportation and display facility.

State Machine

CCF Control is using the standard state machine, defined for DCS applications by the DCS ICD (ESO-305615). The state machine is described in detail in this document, and will not be described further in this manual. A diagram representation of the state machine, is provided here for convenience:

A local copy of the DCS state machine SCXML document can be found in the CCF Package (“ccf/control/resource/config/ccf/control/sm.xml”)

Online Database

The CCF Control Application updates a set of information in the OLDB to be used by client applications, e.g. for displaying the status in control GUIs, or usage in Sequencer Scripts.

Each CCF Control Application instance, has one configurable root (“oldb_prefix”), which is used for all the keys, written in the OLDB. It is recommended that this DB root name clearly identifies the CCF Control instance and a format of this form is recommended: “<system><instance ID>”, e.g. “MetisExpoMeter”.

In the following, the keys written in the OLDB are documented, also because it is (probably) mostly self-explanatory. Some keys are described elsewhere in this document.

The main categories of information updated in the OLDB are:

properties	Contains status of various configuration and setup parameters.
recording	Status about current and previous Recording Sessions.
sm	The state machine status.
statistics	Run-time statistics about the execution (performance) of the various internal threads. This is described in more details elsewhere in this document (Performance & Statistics).
status	Contains status of various configuration and setup parameters, and other system parameters read.

Example & Template Files

In addition to the modules mentioned above, the CCF package contains a number of files, which will be installed in the INTROOT.

The files, which may be used in a production environment, are (path names as they shall be given to be searchable):

config/ccf/client/log.properties: Log properties to be applied for the C-Client.
config/ccf/control/log.properties: Log properties for the CCF Control application.
config/ccf/control/sm.xml: Standard DCS SCXML state machine.
config/ccf/control/metaData.mapping.yaml: Example Meta Data Mapping - not supported in this release.
dictionary/ccf/common/ccf.did.yaml: Base dictionary for CCF.
image/ccf/control/Test_naxis3_1_512_int16.fits: Example FITS cube used for demo purposes.
config/ccf/protocols/aravis/avt.mapping.yaml: Example Camera Name Mapping file for Allied Vision GigE Vision cameras.
schema/ccf/common/server.schema.yaml: CII Configuration Service schema for the CCF Control Configuration.
schema/ccf/common/setup.schema.yaml: CII Configuration Service base schema to be used to create the specific setup schema for each individual CCF Control instance. It is possible to define new setup parameters according to the layout defined in the Configuration and for new adapters added.
schema/ccf/common/telemetry_node_list.schema.yaml: Schema defining the format for the nodes to be read by the Telemetry Sampling.
config/ccf/common/default.telemetry.yaml: Example of Telemetry Sampling nodes list. Note: The provided file shall more be seen as an example, although it may work for many devices. It only specifies one parameter, “expo.time”.

The files, which are provided only for example/demo purposes (not for production or test deployment), are:

config/ccf/control/example.cfg.yaml: Example configuration without support for DDT.
setup/ccf/control/example.setup.yaml: Initialisation Setup referenced by “config/ccf/control/example.cfg.yaml”.
schema/ccf/control/setupExample.schema.yaml: Example setup schema used for the Setup Document above.