Data Acquisition Guide

This guide demonstrates different Data Acquisition use-cases with focus on the interaction with OCM to act as a guide when e.g. writing Sequencer templates/scripts.

For details on deployment please refer to section Deployment and sections for daqOcmServer and daqDpmServer.

Note

Although daqDpmServer is required to produce the final Data Product it is not a software component the end user interacts with directly, rather it is daqOcmServer that commands the daqOcmServer. As such the guide does not contain any examples of how to interact with daqDpmServer, other than how to start and stop it.

Prerequisites

The guide assumes the runtime environment has been configured and daqOcmServer and optionally daqDpmServer has been deployed and is running. This section provide the minimal number of steps to achieve that.

The following standard ICS Framework environment variables are expected to be defined:

$DATAROOT

Will be used by default by daqOcmServer and daqDpmServer for their respective workspaces:

  • $DATAROOT/ocm

  • $DATAROOT/dpm

$CFGPATH

Will be used to resolve configuration paths (see Config Path). To use configuration files when building from source add the path $PREFIX/resource to $CFGPATH, where $PREFIX is the installation prefix.

The instructions assume the following environment variables are defined (if defined the corresponding explicit option can be omitted). The endpoint URIs should reflect what is in your configuration, and using the example configuration above it would be:

  • $OCM_REQUEST_EP=zpb.rr://127.0.0.1:54162 (replaces option --rep)

  • $OCM_PUBLISH_EP=zpb.ps://127.0.0.1:54163 (replaces option --pep)

Futher, unless otherwise stated, all instructions in this section assumes OCM is deployed and is in state Operational/Ready which can be achieved with the following set of commands:

Note

Examples in this guide use the command line client daqOcmCtl provided with the ifw-daq project. To facilitate use from scripts daqOcmCtl can provide return values in JSON format with the --json option. Alternatively the MAL API daqif.OcmDaqControl may be used directly from e.g. Python scripts.

Important

Many examples in this guide include references to components with names such as “fcf1”, “fcf2”, or “dcs1” with MAL URIs such as “zpb.rr://insws:12345/daq”. These components are assumed to be deployed and operational. This means that the examples must be modified to suit your environment to instead reference your components.

Lifecycle Control

Startup

For testing purposes the default daqOcmServer sample configuration provided by the ifw-daq project may be used (config/daqOcmServer/config.yaml)

The configuration is

cfg:
    instrument_id: "UNDEF"
    req_endpoint: "zpb.rr://127.0.0.1:12081/" # IP address and port used to accept requests
    pub_endpoint: "zpb.ps://127.0.0.1:12082/" # IP address and port used to accept requests
    oldb_conn_timeout: 2 # timeout in seconds when connecting to runtime DB
    sm_scxml: "config/daqOcmServer/sm.xml"
    log_properties: "config/daqOcmServer/log.properties"

    # DPM communication configuration
    dpm:
        req_endpoint: "zpb.rr://127.0.0.1:12083/"
        pub_endpoint: "zpb.ps://127.0.0.1:12084/"

In which case the following steps can be performed to start daqOcmServer:

Start daqOcmServer server using default configuration and export environment variables to simplify daqOcmCtl interaction. The configuration must be found as a Config Path using $CFGPATH environment variable.

$ daqOcmServer &
$ export OCM_REQUEST_EP=zpb.rr://127.0.0.1:12081
$ export OCM_PUBLISH_EP=zpb.ps://127.0.0.1:12082

Optionally start daqDpmServer (if files should be merged) and bring it operational

$ daqDpmServer --rr-uri=zpb.rr://127.0.0.1:12083 --ps-uri=zpb.ps://127.0.0.1:12084 &
$ daqOcmCtl std.init
"OK"
$ daqOcmCtl std.enable
"OK"

Note

To use a custom configuration create a new configuration file using with the help of sections Configuration File and Configuration File. Then specify the configuration file when starting daqOcmServer and daqDpmServer with the --config PATH argument.

Shutdown

To shut daqOcmServer down the signal SIGINT (Ctrl-c) or the command Exit() can be sent using e.g. the daqOcmCtl application:

$ daqOcmCtl std.exit

To shut daqDpmServer down send signal SIGINT (Ctrl-c).

Observing Status Changes

To observe the published state changes in OCM it is possible to use daqOcmCtl without a command with the --status option, in which case it will subscribe and remain running until stopped with Ctrl-c while printing any received topic samples to stderr. This is an example of the output during a Data Acquisition:

$ daqOcmCtl --status
no command provided -> will subscribe indefinitely
status: Operational;Active
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=NotStarted, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Starting, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Acquiring, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Acquiring, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopping, error=false
status: Operational;Idle
daq: id=TEST.2021-05-18T14:49:03.905, state=StateAcquiring, substate=Stopped, error=false

Note

The status is published by OCM for any change, some of which are not caused by state transitions. This may lead to the same status line being repeated multiple times.

The --status option can be passed when executing a command as well, but often the commands complete before any topic samples are received. The exception to this is the daq.awaitstate which only completes when the specified state is achieved or operation times out.

Automatic Stop Sequence

Automatic stop is the typical scenario when a Data Acquisition includes one or more primary data sources (usually detectors) that are configured with a fixed integration time. When all primary data sources stop (also referred to as Completed or Stopped in recif) as observed by OCM, which monitors all the sources, OCM will proceed and stop all metadata sources automatically. This condition can then be awaited on to then trigger other activities such as configuring the instrument for the next Data Acquisition.

Note

Multiple sources in each category are provided by space-separating each source, e.g:

"fcf1@zpb.rr://insws1:12345/daq fcf2@zpb.rr://insws2:12345/daq"

Data Acquisition is started with the daq.start command which takes two list of sources.

The following example starts a new Data Acquisition with the primary source named dcs1 and metadata source named fcf1. The names are only used to give a friendly name to a possibly anonymous URI and is so far not used for anything but logging activities.

$ daqOcmCtl --json daq.start "dcs1@zpb.rr://dcsws:12345/rec" "fcf1@zpb.rr://insws:12345/daq"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

Data Acquisition status can be checked with the daq.getstatus command:

$ daqOcmCtl --json daq.getstatus "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "state": "StateAcquiring",
  "substate": "Acquiring",
  "error": false,
  "timestamp": 36142213.78997446597
}

When dcs1 completes OCM will issue the equivalent of the StopDaq command to stop all metadata sources. To await the completion of all FITS files the daq.awaitstate command is issued with states daqif.DaqState.StateAcquiring / daqif.DaqSubstate.Stopped:

$ daqOcmCtl --json daq.awaitstate \
    "TEST.2021-03-09T18:48:05.967" StateAcquiring Stopped 60
Note: Setting request timeout to 62s due to await timeout exceeding request timeout
{
  "timeout": false,
  "status": {
      "id": "TEST.2021-03-09T18:48:05.967",
      "state": "StateAcquiring",
      "substate": "Stopped",
      "error": false,
      "timestamp": 36142213.78997446597
    }
}

If the global --status option is provided, daqOcmCtl will subscribe to published status topics and print them to stderr (JSON command result goes to stdout):

$ daqOcmCtl --status --json daq.awaitstate \
    "TEST.2021-03-09T18:48:05.967" StateAcquiring Stopped 60
Note: Setting request timeout to 62s due to await timeout exceeding request timeout
daq: id=TEST.2021-03-09T18:48:05.967 state=StateAcquiring, substate=Stopping
daq: id=TEST.2021-03-09T18:48:05.967, state=StateAcquiring, substate=Stopped
Done
{
  "timeout": false,
  "status": {
      "id": "TEST.2021-03-09T18:48:05.967",
      "state": "StateAcquiring",
      "substate": "Stopped",
      "error": false,
      "timestamp": 36142213.78997446597
    }
}

At this point the data sources are finished with their contribution and the merging process can be started.

Manual Stop Sequence

Manual stop is used either when there is no fixed integration time on all primary data sources or when no primary sources are used at all (in which case there is nothing to inform OCM when to stop except for the user).

In this example data is acquired from fcf1 only, so user must decide when to stop manually.

$ daqOcmCtl --json daq.start "" "fcf1@zpb.rr://insws:12345/daq"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

At a later point the Data Acquisition is stopped by specifying the id returned by daq.start.

$ daqOcmCtl --json daq.stop "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "error": false
}

If we check status we see that it is indeed stopped.

$ daqOcmCtl --json daq.getstatus "TEST.2021-03-09T18:48:05.967"
{
  "id": "TEST.2021-03-09T18:48:05.967",
  "state": "StateAcquiring",
  "substate": "Stopped",
  "error": false,
  "timestamp": 36142213.789974466
}