Configuration¶

In this chapter, the configuration of a CCF Control instance is described. The configuration is loaded and installed during start-up and initialisation of the application. The set of configuration parameters provided is fixed.

The configuration files needed for one instance of CCF Control, are shown in the following diagram:

The logging carried out by CCF is based on the “log4cplus” tool and consequently, also a “logging properties” file is installed and used by the application.

The Configuration contains parameters that usually do not change while operating a CCF Control instance.

An Initialisation Setup is provided, which contains default values for all parameters, which depend on the lay-out of CCF Control instance, defined in the Configuration and on the parameters defined for the various adapters used – the standard ones and the user provided ones.

The Configuration and the Initialisation Setup are based on the CII Configuration Service, and uses schemas for validating the content.

It is necessary to provide a specific Setup Schema for each CFF Control instance, since the lay-out of this depends on the ‘architecture’ of the instance, e.g. the number of Processing Threads, the Data Processing Recipes and the Data Publishers.

Configuration Parameters¶

The following categories of parameters are defined:

Base (system): Parameters like URI for request/response, pub/sub etc.

Recording: Parameters defining the properties for the data recording.

Tasks: Defining properties for the various tasks (threads) running in the CCF Control instance:

Monitoring: Used for now to collect run-time performance information and update this in the OLDB.

Acquisition: Defining the properties for the Acquisition Task (thread).

Processing Pipelines (Threads): Specifying each its associated:

Data Processing Recipes: Applied on the frames to process them on-the-fly.

Data Publishers: Applied on the final output frames to deliver them to the sources required, e.g. the DDT or FITS files.

Devices (Camera Devices): Specifying the properties for each camera or data provided controlled.

An example of a configuration, is available in CCF: “config/ccf/control/example.cfg.yaml”.

The configuration is organised in a hierarchical way, based on indentation, e.g.

!cfg.include schema/ccf/common/server.schema.yaml:

server: !cfg.type:CcfServerTypeDef
  server_id:                   'TestDcs'
  req_endpoint:                'zpb.rr://127.0.0.1:12092/'
  pub_endpoint:                'zpb.ps://127.0.0.1:51234/'
  db_timeout:                  2000
  scxml:                       'config/ccf/control/sm.xml'
  log_properties:              'config/ccf/control/log.properties'
  req_timeout:                 60
  dictionaries:                ['dictionary/dit/stddid/primary.did.yaml',
                                'dictionary/ccf/common/ccf.did.yaml']
  oldb_prefix:                 'TestIns'
  fits_prefix:                 'DET1'
  simulation:                  false
  init_setup:                  'setup/ccf/control/example.setup.yaml'

  recording:
    metadata_map:              'config/ccf/control/metaData.mapping.yaml'
    rec_hist_size:             1000
    rec_hist_expiration:       2147483647

 <remaining document truncated>

In the following, the parameters are referred to as ‘keys’. E.g. for the “metadata_map” parameter above, this is referenced as “server.recording.metadata_map” to specify the complete name. The “server” element may be left out, as it applies to all configuration parameters. To indicate an element in an array, “[#]” ised used.

System Configuration Parameters¶

In the following the table the system related configuration parameters are listed. All parameters are contained in the “server” hierachical node.

Standard Parameters

Standard configuration parameters defined in the ifw-core BAT “cfg.type:CcfServerTypeDef” schema (prefix):

server_id, req_endpoint, pub_endpoint, db_timeout, scxml, log_properties, req_timeout, dictionaries, oldb_prefix, fits_prefix

simulation (String)

Defines whether to run the instance in normal or simulation mode. From the perspective of CCF, switching between normal and simulation mode, boils down to which Communication Adapter to use.

init_setup (String)

Name of the Initialisation Setup to load during application start-up and initialisation. It provides default values for the setup parameters. A specific Initialisation Setup shall normally be provided for a specific instance. Example: “system/control/init .setup.yaml”.

Recording Configuration Parameters, (“server.recording” category)¶

In the following the table the recording related configuration parameters are listed:

metadata_map (String)	Mapping used to map the internal metadata parameter names into the corresponding metadata parameter names stored in the data product output files, e.g. FITS files.
rec_hist_size (Integer)	Size of recording request buffer (number of requests to buffer).
rec_hist_expiration (Integer)	Max time in seconds to keep the recording requests buffered.

Monitoring Configuration Parameters (“server.tasks.monitoring”)¶

In the following the table the monitoring related configuration parameters are listed:

period (Double)	Period in seconds for scheduling the Monitor Thread. Recommended value is 1s. Note, this determines how frequent the thread execution statistics is computed and updated in the OLDB.
nb_of_samples (Integer)	Number of samples to use for the sliding window of samples, used for calculating the thread execution statistics.

Acquisition Configuration Parameters (“server.tasks.acquisition”)¶

In the following the table the acquisition configuration parameters are listed:

max_rate (Double)	Maximum rate for the data acquisition (Hz).
input_queue_size (Integer)	Number of frame buffer elements in the Input Queue. In the optimal case, two buffers would be sufficient. However, due to the deployment on a non real-time platform, it is necessary to allocate extra buffers to average out jittering.
allow_lost_frames (Boolean)	If true, frames lost at the level of interaction with the camera, are silently ignored.
allow_frame_skipping (Boolean)	If true, in case there are no free buffers available in the Input Queue, these are silently discarded. Otherwise a warning log is produced.

Processing Pipeline Configuration Parameters (“server.tasks.processing”)¶

In the following the table the Processing Pipeline configuration parameters are listed:

processing[#].pipeline (String)	Name of the Processing Thread.
processing[#].output_queue_size (Integer)	Number of frame buffer elements in the Output Queue allocated to the Processing Thread 1. In the optimal case, two buffers would be sufficient. However, due to the deployment on a non real-time platform, it is necessary to allocate extra buffers to average out jittering.
processing[#].allow_frame_skipping (Boolean)	If true, in case there are no free buffers available in the Processing Thread’s Output Queue, these are silently discarded. Otherwise a warning log is produced at a maximum period of 10s.
processing[#].recipes[#].name (String)	Logical name allocated to the Processing Recipe instance. Serves mainly to get meaningly and easy understandable logging output from the recipe.
processing[#].recipes[#].adapter (String)	Name of the Processing Recipe Adapter to be applied on the image frames, for processing purposes. The name must match the name allocated in the Processing Recipe Adapter implementation, e.g. “ccf::stdrecipe::RecipeCentroid”.
processing[#].publishers[#].name (String)	Logical name allocated to the Data Publisher instance. Serves mainly to get meaningly and easy understandable logging output from the recipe.
processing[#].publishers[#].adapter (String)	Name of the Data Publisher Adapter used to publish the processed image frames, for processing purposes. The name must match the name allocated in the Processing Recipe Adapter implementation, e.g. “ccf::stdpub::PubFits”.

Devices Configuration Parameters (“server.devices”)¶

In the following the table the camera device related configuration parameters are listed. The possibility of having multiple cameras in case of a ‘Super DCS’ application is foreseen, but not yet supported. Hence, “devices” is an array.

devices[#].name (String)	Name assigned to the camera.
devices[#].id (String)	ESO wide, unique ID assigned to the camera.
devices[#].type (String)	Type of camera, e.g. “CCD”, “IR, “CMOS”.
devices[#].model (String)	Camera model.
devices[#].manufacturer (String)	Camera manufacturer.
devices[#].width (Integer)	Width in pixels, of detector chip.
devices[#].height (Integer)	Height in pixels, of detector chip.
devices[#].max_resolution (Integer)	The maximum possible resolution (bits/pixel) for the given camera.
devices[#].resolution (Integer)	Resolution used in given set up.
devices[#].protocol (String)	Protocol used for communicating with the camera, e.g. “GigE Vision”.
devices[#].mapping (String)	Name mapping, mapping the internal names, used in the communication adapter, into the specific names, defined in the camera name space.
devices[#].sdk (String)	API (or SDK) used for implementation the communication with the camera. Example: “CCF/Aravis”.
devices[#].address (String)	Address used for connecting to the camera. Is typically an IP address + port number, for Ethernet/socket based communication protocols.
devices[#].sim_address (String)	Address of the CCF simulator, if an external simulator is used. Note, for the present version of CCF, simulation is provided via a Simulation Communication Adapter.
devices[#].properties[#].name (String)	Property name, e.g. “StreamBytesPerSecondMax”.
devices[#].properties[#].value (String)	Value for the property.

Note, the device properties may be used by the camera, the interface (HW), the SDK or the CFF Communication Adapter. It’s up to the Communication Adapter to implement the correct handling of the parameters.

CCF Control Configuration Schema¶

The CII Configuration Service schema for the Configuration, is defined as follows (“schema/ccf/common/server.schema.yaml”):

!cfg.include config/utils/bat/definitions.yaml:

!cfg.typedef PropertyTypeDef:
   !cfg.required name:                  !cfg.type:string
   !cfg.required value:                 !cfg.type:string
   write_on_init:                       !cfg.type:boolean false

!cfg.typedef DeviceTypeDef:
   !cfg.required name:                  !cfg.type:string
   !cfg.required id:                    !cfg.type:string
   !cfg.required type:                  !cfg.type:string
   !cfg.required model:                 !cfg.type:string
   !cfg.required manufacturer:          !cfg.type:string
   !cfg.required width:                 !cfg.type:int32
   !cfg.required height:                !cfg.type:int32
   !cfg.required resolution:            !cfg.type:int16
   !cfg.required max_resolution:        !cfg.type:int16
   !cfg.required protocol:              !cfg.type:string
   !cfg.required mapping:               !cfg.type:string
   !cfg.required sdk:                   !cfg.type:string
   !cfg.required address:               !cfg.type:string
   !cfg.required sim_address:           !cfg.type:string
   !cfg.required properties:            !cfg.type:vector_PropertyTypeDef

!cfg.typedef RecordingTypeDef:
   !cfg.required metadata_map:          !cfg.type:string
   !cfg.required rec_hist_size:         !cfg.type:int32
   !cfg.required rec_hist_expiration:   !cfg.type:int32

!cfg.typedef MonitoringTypeDef:
   !cfg.required period:                !cfg.type:int32
   !cfg.required nb_of_samples:         !cfg.type:int32

!cfg.typedef AcquisitionTypeDef:
   !cfg.required max_rate:              !cfg.type:double
   !cfg.required input_queue_size:      !cfg.type:int16
   !cfg.required allow_lost_frames:     !cfg.type:boolean false
   !cfg.required allow_frame_skipping:  !cfg.type:boolean false

!cfg.typedef RecipeTypeDef:
   !cfg.required name:                  !cfg.type:string
   !cfg.required adapter:               !cfg.type:string

!cfg.typedef PublisherTypeDef:
   !cfg.required name:                  !cfg.type:string
   !cfg.required adapter:               !cfg.type:string

!cfg.typedef PipelineTypeDef:
   !cfg.required pipeline:              !cfg.type:string
   !cfg.required output_queue_size:     !cfg.type:int16
   !cfg.required allow_frame_skipping:  !cfg.type:boolean
   !cfg.required recipes:               !cfg.type:vector_RecipeTypeDef
   !cfg.required publishers:            !cfg.type:vector_PublisherTypeDef

!cfg.typedef TasksTypeDef:
   !cfg.required monitoring:            !cfg.type:MonitoringTypeDef
   !cfg.required acquisition:           !cfg.type:AcquisitionTypeDef
   !cfg.required processing:            !cfg.type:vector_PipelineTypeDef

!cfg.typedef CcfServerTypeDef(IfwServer):
   !cfg.required simulation:            !cfg.type:boolean false
   !cfg.required init_setup:            !cfg.type:string
   !cfg.required recording:             !cfg.type:RecordingTypeDef
   !cfg.required tasks:                 !cfg.type:TasksTypeDef
   !cfg.required devices:               !cfg.type:vector_DeviceTypeDef

Initialisation Setup¶

An Initialisation Setup (or simply Setup) must be specified in the Main Configuration to define the default setup of the system it will resume after the initialisation of the application.

The Initialisation Setup merely lists the various possible Setup Parameters, defined for the given instance. The issue of the setup is handled in the next chapter.

An example of an Initialisation Setup, is provided within the CCF Package: “setup/ccf/control/example.setup.yaml”.

It is necessary to define a specific schema matching each CCF instance, as the setup is reflecting the chosen configuration of CCF Control, e.g. the number of Processing Threads and the various Processing Recipes and Data Publishers used.

Moreover, if specific adapters, e.g. Processing Recipes or Data Publishers, are provided for a specific context, the parameters for these must be defined in the specific Setup Schema.

See the chapter “Setup” for details about the Setup parameters.

Camera Name Mapping¶

The Camera Name Mapping file is a file in YAML format, used to map the names for camera properties, used in the Communciation Adapter, into the specific names defined for the associated camera.

Using a Camera Name Mapping makes sense, if a generic Communication Adapter can be re-used to control multiple cameras, if it is avoided to hardcode names in the Communication Adapter.

If the names are hardcoded in the Communication Adapter, or not used in connection with the given camera/SDK, a Camera Name Mapping is not needed.

An example of a Camera Name Mapping is provided by the CCF Package: “ifw-ccf/protocols/aravis/lib/resource/config/ccf/protocols/aravis/avt.mapping.yaml”.

Metadata Name Mapping¶

The purpose of the Metadata Name Mapping file is to map parameters registered in connection with a given image frame, into the actual metadata names, required in the Output Data Product File.

When a frame is received, the Communication Adapter may register metadata, describing the image, in its frame buffer object.

In addition, Processing Recipes may add such metadata, which should be added in the Output Data Product Files. When a Metadata Name Mapping is provided, the target Output Data Product metadata keys, shall be registered in a Dictionary, defined in the configuration.

An example of a Metadata Mapping is provided by the CCF Package: “config/ccf/control/metaData.mapping.yaml”.

Dictionaries¶

A Dictionary is used to describe the properties of the metadata keywords to be written in the Data Product Output Files. The format and tools to handle Dictionaries, is provided by the ICS Core Data Interface Tools (DIT) Package.

A base Dictionary is provided by the CCF Package: “dictionary/ccf/common/ccf.did.yaml”. This Dictionary shall normally always be specified in the configuration and used.

Note: It is the intention to replace the present format with a format based on the CII Configuration Service.

Simulation Data File¶

In the present version, the simulation is implemented within the Simulation Communication Adapter and it is based on a FITS file as input. The FITS file is then played back, according to the given setup/exposure parameters.

If a context specific simulator is provided, it may use another type of file, or no input file at all.

A small tool is provided to allow the users of CCF to create their own FITS data cubes, with an emulated star moving in the field of view. This tool is described in more detail in the section CCF Generate FITS Cube Tool (“ccfFitsCubeGen”).