Overview¶

The CCF project is composed by the following modules/packages:

These packages provide the following functionality:

mptk: Multiprocessing Toolkit: A small collection of classes to manage a multithreaded environment, including inter-thread syncrhonisation and communication.

common: Common Tools & Utilities: A set of common utilities used for implementing CCF applications, e.g. an Image Frame Class, a Frame Queue, DB access class, and more.

control: CCF Control Application: Provides implementation of the CCF Control application class and classes/callbacks for handling requests, state machine related activities and implementations of the various types of threads, running in a CCF Control application.

dcssend: C++ Request/Response Command Line Utility: Command line utility to submit requests to a CCF Control application and receive the responses.

pycli: Python Client Library & Command Line Utility: Python module providing a CCF Control client and a command line utility based on that.

sim: CCF Control Simulation Deployment Module: Deployment module for a CCF Control Application, providing simulation only.

tools: Miscelleneous Tools: It provides the following features:

ccfclib: Python synchroneous and asynchroneous client library.

cli: Interactive CCF Control client shell (based on “ccfclib”).

pycli: Deprecated: May be removed in the future since the “ccfclib” and “cli” provide the same functionality.

dcsSend: C++ client library and command line utility.

ccfFitsCubeGen: Small tool to generate FITS cubes that can be used as input for the CCF Simulator.

rtms2ccflib: Library, handling the conversion of image frames, sent via RTMS to the internal CCF format.

stdrecipe: Standard Processing Recipes: In this version only a centroiding recipe is provided.

stdpub: Standard Data Publishers: In this version a basic FITS CCF Data Publisher, a DDT Data Publisher and an RTMS Publisher are provided.

protocols: Specific Communication Adapters & Deployment Executables: Provides implementation of CCF Communication Adapters for various protocols and the associated deployment executable for these. Note, in many cases it will be needed to provide context specific data processing and thus a context specific CCF Control Application must be generated, based on the context specific adapters.

Two modules, “test” and “doc”. These contain the integration test and the sources for the user manual, in accordance with the standard.

In the design of the component, it was foreseen to provide a separate simulator process. This design choice has been changed to providing simulation via a Communication Adapter. It is the intention to provide simulation based on this scheme until further.

Moreover, the design contains a proposal for a Super DCS Application. This has not yet been implemented and will only be assessed if specific requests for such a service will be submitted.

Architecture¶

The aim of the CCF Package as such, is not to so much to provide a ready-to-use set of CCF application executables, but rather to facilitate easy integration of customised (context specific) DCS solutions. To achieve this, the CCF SDK is based on the adapter pattern to allow for context specific customization.

The adapter types provided are:

Communication Adapter: Implements protocol specific communication.

Processing Recipe Adapter: Implements on-the-fly processing of image frames, executed in a dedicated thread in a CCF Control instance.

Data Publisher Adapter: Implements publishing of data to specific targets, which may be output files or applications subcribing to the data.

For future versions a “Method Invocation Adapter” (for the “dcsif::Execute” request) is provided, and possibly a “Setup Handling Adapter” (for the “dcsif::Setup” request), according to needs. In addition, a “System Monitoring Adapter” will be provided; this will be executed periodically by the Monitoring Thread and can be used to collect run-time information about the system and monitor the health of the associated camera device.

The anatomy of a CCF Control Application is depicted below.

To use the CCF it is not required to have a very detailed/low-level insight into the architecture. Nonetheless, a basic understanding is needed, and useful, to be able to use it and to implement, deploy and test the various adapters needed.

The internal threads executed in a CCF Control Application are:

RAD Event Handler Thread: The main thread in the application, which handles e.g. incoming requests.

Monitor Thread: The purpose of the Monitor Thread is to supervise the operation of the CCF Control instance. In particular it collects periodically, data about the execution of the various other CCF threads, which is used to compute statistics, which is written in the OLDB.

Acquisition Thread: The Acquisition Thread, as indicated by the name, is responsible for acquiring the image frames from the associated camera. Once a frame has been received, the image data (pixels) and metadata information is stored in the Input Queue and the Processing Thread(s) informed about the availability of a new data frame. Note, at this point in time, it is only possible to receive complete frames and not to handle sub-frames.

Processing Thread(s): The Processing Threads applies processing in the form of one or more Processing Recipes (adapters) on the data. When informed about the availability of a new frame in the Input Queue, it reserves an image buffer in its associated Output Queue and applies the Recipe(s) specified on the image. Once the processing of an image frame has completed, the Processing Thread informs the associated Publisher Threads about the availability of a new, processed image. If no Recipes are specified (configured), the image frames are merely copied from the Input Queue buffer into the Output Queue buffer and the associated Publisher Thread(s) notified. The number of Processing Threads to execute, is configurable.

Publisher Thread(s): The Publisher Threads send the image data available in the associated Processing Thread Output Queue to the specified target(s). This will typically be files or e.g. the DDT. However, as the actual publishing is carried out by a Data Publisher Adapter, other types of targets are possible. In this version it is only possible to specify one Data Publisher Adapters per Publisher Thread.

When inspecting the threads running in a CCF Control instance, e.g. with “htop”, the user will observe more threads than the ones mentioned above. These are threads deployed by CII and e.g. by an SDK used for communicating with the camera.

The other internal elements composing a CCF Control Application are:

Communication Adapter: The Communication Adapter used for communicating with the associated camera. As this adapter may be provided by the user, it could be used for other purposes, or to collect data from other sources than cameras.

Simulation Communication Adapter: The Communication Adapter used for communicating with the simulator. For now, the provided Simulation Communication Adapter actually, is responsible for the simulation and is not communicating with an external simulator. If of relevance, a context specific Simulation Communication Adapter may be provided, either generating internally the simulated data or communicating with an external simulator.

Input Queue: Queue containing image buffer objects (“ccf::common::DataFrame”), with a preallocated set of buffers, used in a ring-buffer fashion by the Acquisition Thread to store the incoming frames, and make these available for the configured Processing Thread(s). The number of buffers in the Input Queue is configurable.

Output Queue: For each Processing Thread, one Output Queue is allocated to hand over processed data to the Publisher Threads. The elements in the Output Queue(s) are pre-allocated instances on the “ccf::common::DataFrame” class, which are re-used in a ring-buffer fashion. The number of image buffers in the Output Queue is configurable.

The external entities shown in the above figure, are as follows:

Pub/Sub: The Pub/Sub service supported by CII MAL. For this version, this is only used for publishing the state of the application, in accordance with the Standard Application Interface (“stdif”).

OLDB: The online database supported by the CII Services. The is the base line for CCF to publish information for the client applications. An overview of the information updated in the OLDB, is provided in one of the following sections.

Client: A client may be e.g. the Sequencer, a GUI or a command line tool, used to command the CCF Control instance.

Configuration: The main configuration file is based on the CII Configuration Service (YAML format). It is referenced when starting the application. The configuration is loaded and applied when the application is starting up, and applied to the associated camera during initialisation. The main configuration refers to a number of other files, which are loaded and applied as well, when the application starts up and initialises.

Initialisation Setup: Contains the default setup when the application starts up (YAML format). This is referenced by the main configuration. It is based on the CII Configuration Service.

Camera Name Mapping: Contains a mapping from the parameter names used internally in the Communication Adapter and the names defined in the camera name space.

Metadata Name Mapping: Maps internal names for metadata, used in the CCF Control Application to the user metadata names (not used for this release).

Dictionaries: Contains the definition of the keywords to be written in the output data products, normally FITS files.

Simulation Data File: The Simulation Data File used as basis for the simulation. Typically this will be a FITS image cube, providing a sequence of images emulating the usage on sky. The user may provide use case specific Simulation FITS Files, to obtain a realistic emulation. An example Simulation Data File is provided with the CCF Package (“ccf/control/resource/image/ccf/control/testCube16Bit.fits”). If a context specific simulation is provided, e.g. in the form of a Communication Adapter, this adapter may define how to obtain the data used as input for the simulation. A small utility is provided (on a prototype basis) to generate Simulation Data File, FITS image cubes (“ccfGenFitsCube”).

Output Data Product Files: Output files generated by a Data Publisher. Normally this will be FITS files, but if required, it will be possible to provide context specific Data Publishers to generate other formats, e.g. for more efficient storage in ‘burst mode’. A standard FITS Data Publisher is provided by the CCF Package.

DDT Broker/Viewer: The standard DDT image transportation and display facility.

State Machine¶

CCF Control is using the standard state machine, defined for DCS applications by the DCS ICD (ESO-305615). The state machine is described in detail in this document, and will not be described further in this manual. A diagram representation of the state machine, is provided here for convenience:

A local copy of the DCS state machine SCXML document can be found in the CCF Package (“ccf/control/resource/config/ccf/control/sm.xml”)

Online Database¶

The CCF Control Application updates a set of information in the OLDB to be used by client applications, e.g. for displaying the status in control GUIs, or in Sequencer Scripts.

Each CCF Control Application instance, has one configurable root (“oldb_prefix”), which is used for all the keys, written in the OLDB. It is recommened that this DB root name clearly identifies the CCF Control instance and a format of this form is recommended: “<system><instance ID>”, e.g. “MetisExpoMeter”.

In the following, the keys written in the OLDB are documented, also because it is (probably) mostly self-explanatory. Some keys are described elsewhere in this document.

The main categories of information updated in the OLDB are:

camera	Contains status values for the camera properties. This may be property names defined in the camera name space.
config	The current configuration loaded/used by CCF Control
dictionary	The dictionary keys in use.
publishers	Status of each Data Publisher deployed in the CCF Control instance.
recording	The overall status of an ongoing Data Recording.
setup	The current set of setup parameters in use.
sm	The state machine status.
statistics	Run-time statistics about the execution (performance) of the various internal threads. This is described in more details elsewhere in this document (Performance & Statistics).

Example & Template Files¶

In addition to the modules mentioned above, the CCF package contains a number of files, which will be installed in the INTROOT.

The files, which may be used in a production environment, are (path names as they shall be given to be searchable):

config/ccf/client/log.properties: Log properties to be applied for the C-Client.
config/ccf/control/log.properties: Log properties for the CCF Control application.
config/ccf/control/sm.xml: Standard DCS SCXML statemachine.
config/ccf/control/metaData.mapping.yaml: Example Meta Data Mapping - not supported in this release.
dictionary/ccf/common/ccf..yaml: Base dictionary for CCF.
image/ccf/control/Test_naxis3_1_512_int16.fits: Example FITS cube used for demo purposes.
config/ccf/protocols/aravis/avt.mapping.yaml: Example Camera Name Mapping file for Allied Vision GigE Vision cameras.
schema/ccf/common/server.schema.yaml: CII Configuration Service schema for the CCF Control Configuration.
schema/ccf/common/setup.schema.yaml: CII Configuration Service base schema to be used to create the specific setup schema for each individual CFF Control instance. It is possible to define new setup parameters according to the lay-out defined in the Configuration and for new adapters added.

The files, which are provided only for example/demo purposes (not for production or test deployment), are:

config/ccf/control/example.cfg.yaml: Example configuration without support for DDT.
setup/ccf/control/example.setup.yaml: Initialisation Setup referenced by “ccf/control/config1.yaml”.
schema/ccf/control/setupExample.schema.yaml: Example setup schema used for the Setup Document above.

More example files will be added in a future release.