Installation & Deployment

In this chapter some information in connection with the installation and usage of the CCF are provided.

Installation

The CCF Package is a standard waf/wtools build project and shall be configured, built and installed accordingly.

The CCF Control is relying on the following environment variables:

  • DATAROOT: Directory on the host machine in which Output Data Products will be generated. The storage location will be rendered as “$DATAROOT/<cfg key: server.recording.image_dir>”. The ‘image’ sub-directory in “DATAROOT” will be created automatically by CCF Control, if not existing.

  • CFGPATH: The “CFGPATH” environment variable, is a colon separated list of paths, pointing to possible Resource Directories in which resource data of different kinds are located.

  • INTROOT: In this release, the example configurations delivered with the CCF Package as well as the libraries, header files and binaries, are installed into the location pointed to by “INTROOT”. This may change in the future.

Note, in accordance with the “Instrument Software Specification”, one system, typically an instrument, shall deliver a ready-to-use Resource Tree as part of the software package. This Resource Tree shall be referenced to by the “CFGPATH” variable.

Deployment Module

Details about the configuration of a CCF Control instance are given in the “Configuration” chapter in this manual.

The CCF Package is a tool used to integrate specific DCS solutions. Normally, specific Adapters will be provided for the given context, and used to deploy (generate) a new CCF Control executable.

Two examples of deployment modules are provided by the CCF Package:

  • ifw-ccf/sim

  • ifw-ccf/protocols/aravis/exe

The example from “ifw-ccf/sim” is shown here:

#include <ccf/control/application.hpp>
#include <ccf/control/comAdptSim.hpp>
#include <ccf/stdrecipe/recipeCentroid.hpp>
#include <ccf/stdpub/pubFits.hpp>
#include <ccf/stdpub/pubDdt.hpp>
#include <ccf/stdpub/pubRtms.hpp>

int main(int argc, char* argv[]) {
  ccf::control::Application& application = ccf::control::Application::Instance();

  LOG4CPLUS_INFO(ccf::Logger(), "Application started.");

  // Normal Mode and Simulation Communication Adapter factory objects.
  ccf::control::ComAdptSim com_adapter;
  ccf::control::ComAdptSim sim_com_adapter;
  application.RegisterComAdapters(com_adapter, sim_com_adapter);

  // Register Processing Recipe Adapter factory objects.
  LOG4CPLUS_INFO(ccf::Logger(), "Allocate and register Processing Recipe Adapter factory objects ...");
  ccf::stdrecipe::RecipeCentroid centroid_recipe_object;
  ccf::stdrecipe::RecipeCentroid::AddRecipeFactoryObj(centroid_recipe_object);
  ccf::common::RecipeBase dummy_rec_object;
  ccf::common::RecipeBase::AddRecipeFactoryObj(dummy_rec_object);

  // Register Publisher Adapter factory objects.
  LOG4CPLUS_INFO(ccf::Logger(), "Allocate and register Publisher Adapter factory objects ...");
  ccf::stdpub::PubFits fits_pub_factory_object;
  ccf::common::PubBase::AddPubFactoryObj(fits_pub_factory_object);
  ccf::stdpub::PubDdt ddt_pub_factory_object;
  ccf::stdpub::PubDdt::AddPubFactoryObj(ddt_pub_factory_object);
  ccf::stdpub::PubRtms rtms_pub_factory_object;
  ccf::stdpub::PubRtms::AddPubFactoryObj(rtms_pub_factory_object);
  ccf::common::PubBase dummy_pub_factory_object;
  ccf::common::PubBase::AddPubFactoryObj(dummy_pub_factory_object);

  LOG4CPLUS_INFO(ccf::Logger(), "Execute the application ...");
  int stat = application.Execute(argc, argv);

  LOG4CPLUS_INFO(ccf::Logger(), "Application terminating ...");
  return stat;
}

As can be seen from the example of a CCF Control, C++ main function above, a deployment module consists of the following main parts:

  • Include the necessary header files.

  • Instantiate the CCF Control “Application” class.

  • Create instances of the Communication Adapters for normal and simulated operation and register these.

  • Create factory instances of the Processing Recipes to be deployed according to the configuration and register these.

  • Create factory instances of the Data Publishers to be deployed according to the configuration and register these.

  • Execute the CCF Control application by invoking the “ccf::control::Application::Execute()” method. This will handle the parsing of the command line options, loading, parsing and installation of the various configuration data, and instantiation of internal threads, queues, Recipes and Data Publishers according to the Configuration specified.

CCF Control - Execution - Example

In the following an example is shown, which can be executed directly from the shell after the ICS software package have been installed. To do this, carry out the following steps:

1. Ensure “DATAROOT”, “CFGPATH” and “INTROOT” are defined. Example:

$ echo $DATAROOT
/scratch/jknudstr/DATAROOT
$ ll /scratch/jknudstr/DATAROOT
drwxr-xr-x 2 jknudstr vlt 80 May 11 14:48 image/
$ echo $CFGPATH
/scratch/jknudstr/INTROOT/resource:/elt/ifw/resource
$ echo $INTROOT
/scratch/jknudstr/INTROOT

2. Ensure that the CII Services are up and running: These will normally be started automatically on the ELT development machines.

3. Start the DDT Broker (if required) if not already running: The parameter “<prefix>.<proc thr name>.<pub thr name>.ddt_broker” in the Initialisation Setup defines the URI of the broker, e.g. in this case:

$ nohup ddtBroker --uri zpb.rr://*:12011/broker &> /dev/null &

4. Execute the example CCF binary “ccfCtrlSim”:

$ ccfCtrlSim --config config/ccf/control/example.cfg.yaml -l INFO
INFO - ../sim/src/main.cpp:26:main:ccfCtrlSim: Application started.
INFO - ../sim/src/main.cpp:34:main:ccfCtrlSim: Allocate and register Processing Recipe Adapter factory objects ...
INFO - ../common/src/recipeBase.cpp:40:AddRecipeFactoryObj:ccfCtrlSim: Registering Processing Recipe factory object. Classname: ccf::common::RecipeBase
INFO - ../sim/src/main.cpp:39:main:ccfCtrlSim: Allocate and register Publisher Adapter factory objects ...
INFO - ../common/src/pubBase.cpp:175:AddPubFactoryObj:ccfCtrlSim: Registering Data Pub factory object. Classname: ccf::stdpub::PubFits
INFO - ../common/src/pubBase.cpp:175:AddPubFactoryObj:ccfCtrlSim: Registering Data Pub factory object. Classname: ccf::stdpub::PubDdt
INFO - ../common/src/pubBase.cpp:175:AddPubFactoryObj:ccfCtrlSim: Registering Data Pub factory object. Classname: ccf::common::PubBase
INFO - ../sim/src/main.cpp:47:main:ccfCtrlSim: Execute the application ...
INFO - ../control/src/config.cpp:127:LoadConfig:ccfCtrlSim: Loading configuration: config/ccf/control/example.cfg.yaml ...
INFO - ../control/src/config.cpp:129:LoadConfig:ccfCtrlSim: Loaded configuration: config/ccf/control/example.cfg.yaml
15:45:30.276:INFO:ccf: ../control/src/config.cpp:99:ParseOptions:ccfCtrlSim: Process name: ccfCtrlSim
15:45:30.276:INFO:ccf: ../control/src/config.cpp:106:ParseOptions:ccfCtrlSim: Configuration: config/ccf/control/example.cfg.yaml
15:45:30.304:INFO:ccf: ../control/src/application.cpp:428:LoadInitSetup:ccfCtrlSim: Initialisation Setup specified: ccf/control/initSetupDdt1.yaml
15:45:30.414:INFO:ccf: ../control/src/application.cpp:285:CreateThreads:ccfCtrlSim: Creating Monitor Thread: CcfMonThr ...
15:45:30.414:INFO:ccf: ../control/src/application.cpp:293:CreateThreads:ccfCtrlSim: Creating Acquisition Thread: CcfAcqThr ...
...
15:48:39.493:INFO:ccf: ../common/src/pubBase.cpp:193:CreatePubObj:ccfCtrlSim: Creating Data Publisher object of type: ccf::stdpub::PubFits
15:48:39.493:INFO:ccf: ../common/src/pubBase.cpp:193:CreatePubObj:ccfCtrlSim: Creating Data Publisher object of type: ccf::common::PubBase
15:48:39.494:INFO:ccf: ../common/src/db.cpp:90:UpdateSmStatus:ccfCtrlSim: State changed: || -> |On::NotOperational::NotReady/On::NotOperational/On/ |

Note, some logging information has been removed. In general the logging and error handling will be improved for future releases.

5. Start DDT Viewer if desirable: In this case:

$ nohup ddtViewer -s "zpb.rr://127.0.0.1:12011/broker/Broker1 CcfTest11" &

6. Bring Operational, Start Acquisition:

$ ccfCli --uri zpb.rr://127.0.0.1:12091
zpb.rr://127.0.0.1:12091
ccf> init
reply> = OK

ccf> enable
reply> = OK

ccf> state
reply> = On::Operational::Idle/On::Operational/On

ccf>

6. Execute a Recording Session:

ccf> recstart
Recording Status:
ID:                        RecId-25615696-ceec-461b-a45d-5b73b8ed2278
Status:                    Active
Frames Processed:          0
Frames Remaining:          7
Start Time:                2021-05-11T14:48:07.390597
Time Elapsed:              0.000000
Remaining Time:            7.000000
Estimated Completion Time: 2021-05-11T14:48:14.390597
Volume Recorded (bytes):   0
Files Generated:           0
Info:
Output Files: |

ccf> recstatus
Recording Status:
ID:                        RecId-25615696-ceec-461b-a45d-5b73b8ed2278
Status:                    Active
Frames Processed:          5
Frames Remaining:          2
Start Time:                2021-05-11T14:48:07.390597
Time Elapsed:              5.007639
Remaining Time:            1.992361
Estimated Completion Time: 2021-05-11T14:48:14.390597
Volume Recorded (bytes):   5242880
Files Generated:           2
Info:
Output Files: |
  /scratch/jknudstr/DATAROOT/image/PubFitsTest1.fits
  /scratch/jknudstr/DATAROOT/image/PubFitsTest2.fits

CCF Generate FITS Cube Tool (“ccfFitsCubeGen”)

The “ccfFitsCubeGen”, provided by the CCF Package, can be used to generate FITS cubes to be used as input for the Simulation Communication Adapter, which plays back the images in a FITS cubes. The man-page speaks for itself:

> ccfFitsCubeGen --help
usage: ccfFitsCubeGen [-h] [--size SIZE] [--noise NOISE]
                      [--magnitude MAGNITUDE]
                      [--data-type {INT16,INT32,FLOAT,DOUBLE}]
                      [--nb-of-images NB_OF_IMAGES]
                      [--max-shift-per-image MAX_SHIFT_PER_IMAGE]
                      [--max-total-shift MAX_TOTAL_SHIFT]
                      [--output-file OUTPUT_FILE]

CCF Test FITS Cube Generator

optional arguments:
  -h, --help                                 show this help message and exit
  --size SIZE                                pixels in X/Y
  --noise NOISE                              noise to apply [0; <noise>]
  --magnitude MAGNITUDE                      magnitude of simulated star (max value)
  --data-type {INT16,INT32,FLOAT,DOUBLE}     type of pixel values in output file
  --nb-of-images NB_OF_IMAGES                number of frames in the cube
  --max-shift-per-image MAX_SHIFT_PER_IMAGE  max shift of sim star in pixels per image
  --max-total-shift MAX_TOTAL_SHIFT          max total shift of sim star in pixels per image
  --output-file OUTPUT_FILE                  name of output FITS file

Note: The CCF Simulation Communication Adapter provided, only supports Int8 and Int16.

System Tuning

CCF Control has been designed to achieve maximum throughput by using three levels of parallelism for the

  • data acquisition,

  • data processing, and

  • data publishing.

In this way, the latency is reduced. However, CCF only handles entire image frames at this point in time. I.e., it is not possible to handle sub-frames (of images), which would make it possible to reduce the latency futher.

As mentioned in the “Overview” chapter, a number of internal frame buffer queues are used. These are:

  • Input Queue: Used by the Acquisition Thread, in a ring-buffer fashion, to store incoming image frames, to make them available for the Processing Pipelines.

  • Output Queues: Onces the Processing Recipes have been executed on the image frames, these are available in the Output Queues, one per Processing Thread, for the Publisher Threads.

In a perfectly balanced/calibrated system, in theory, two buffers in each queue would be enough. However, when deploying CCF on a non real-time OS, a certain amount of jitter is to be expected. In order to average out the effect of this jitter of the frame handling, it is recommended to allocate additional buffers, to compensate. If no free frames are available in a queue, the thread handling the frame object, will have to skip (drop) that frame. Whether such frame skipping is acceptable, can be configured via the configuration parameters:

  • “tasks.acquisition.allow_frame_skipping”

  • “tasks.processing.allow_frame_skipping”

If set to true, frames will be skipped/dropped silently. Otherwise, a log will be produced, indicating the frame skipping. This log will be produced with a maximum periodicity of 10 s to avoid possible flooding of the logging system.

The frames skipped is updated in the OLDB, in the statistics section for each type thread.

If frames are skipped in either type of queues (Input Queue or Output Queues), this may be an indication that the processing of the image frames or the publishing thereof, is too slow.

It could be investigated if this can be optimized to increase the throughput and reduce the amount of frames skipped.

Apart from being forced to skip frames due to lack of free space in the internal queues, the Acquisition Thread may detect that frames are lost at the level of receiving these from the camera. This is mostly caused by issues with a poor/inadequate communication channel to the camera, typically the network connection, or that the Acquisition Thread is not fast enough to pick up all frames. When this happens, a log is generated, indicating this. It is possible to suppress this log via the Configuration Parameter “tasks.acquisition.allow_lost_frames”.

Like for the frame skipping, the number of lost frames is updated in the statistics of the Acquisition Thread so it is easy to check if the system is running as expected.

Performance & Statistics

Each of the threads Acquisition, Processing and Publishing, record run-time information about the frame handling.

This information, is collected periodically by the Monitor Thread. Subsequently it computes the performance parameters for the execution of the thread, with focus on the frame handling.

The generation of this statistics, is controlled via the Configuration Parameters:

  • “tasks.monitoring.period”: Period applied for the execution of the Monitor Thread in seconds.

  • “tasks.monitoring.nb_of_samples”: Number of the samples in the sets (sliding windows) used as basis for computing the statistics; a size of 100 seems to be a reasonable value, whereby this depends on the frame rate. For very slow frame rates, it may take a long time to complete the set.

The statistics is running from the moment the data acquisition was started. It is reset when starting a new data acquisition. It is also reset when updating setup parameters, as these often may influence the statistics, e.g. when the exposure time or frame rate are changed.

An example of the statistics parameters for a thread in shown below (as displayed in the “dbbrowser”):

../_images/ccf_control_stat_acq_thread.png

The statistics data values are stored in keys with names of the form:

“<sys.db.prefix>.statistics.<thread name>[.<category>].<key>”

The values reported for each thread are (OLDB key prefix “<sys.db.prefix>.statistics.<thread name>” left out):

volume volume_mb

Indicates the accumulated volume handled by the thread in bytes/mega bytes since the data aquisition was started.

fr_handling.<key>

The Frame Handling Time refers to the time (in seconds) the frames was handled/processed by the thread, from its reception, until it was delivered to the destination. The values are:

jitter: Calculated as the average deviation from the mean value.
max: The maximum frame handling time found in the set.
mean: The mean value of the samples in the set.
min: The minimum frame handling time found in the set.
samples_in_set: The number of samples in the sliding window.
stddev: The std. deviation of the samples in the set, computed with the std. formula.

fr_rec.<key>

The Frame Reception Time refers to the absolute point in time, frames were received from the camera. The statistics is calculated from the difference in time between two consecutive receptiom time stamps. Refer to frame_handling_time.<key> for an explanation of the values computed.

frame_count frame_period frame_rate

The number of frames handled since the acquistion was started, the period of the frame handling (seconds between each set of frames) and the frame rate (frames/second).

last_update

Last time the statistics was updated (seconds since epoch).

lost_frames lost_frames_rate

Number of frames lost receicing these from the camera since the acquisition was started, the frequency of loosing the frames (frames lost/second).

samples_in_set

The size of the sliding window used as basis for calculating the statistics in samples.

skipped_frames skipped_frames_rate

Number of frames skipped (dropped), since the current image acquisition was started, due to lack of space in the queue assocated with the thread + the rate with which frames are being skipped (frames/second).

start_time

Absolute time since epoch for starting the current image acquisition.

theoretical_frame_rate theoretical_periodicity

The theoretical frame rate (frames/seconds) and periodicity (seconds/frame) according to the current setup.

through_put through_put_mbps

The average throughput in bytes/seconds and MB/second.

time_elapsed

Time elapsed in seconds since the current acqusition sequence was started.

CCF provides no service to retrieve the data, nor to generate a report from it. It is up to the client applications to record the data and store it, if of interest.