Introduction

Pre-release Disclaimer

This document is part of a pre-release version with work still in progress. And as such everything must be considered subject to change.

Scope

This document is the user manual for the ELT ICS Framework products OCM and DPM and covers:

The intended audience are instrument or instrument framework developers. Readers are assumed to be familiar with core ELT technologies:

  • CII MAL (request/reply and publish/subscribe)

  • FITS (Header/Data Units, keywords, extensions)

Acronyms

CII

Core Integration Infrastructure

DAQ

Data Acquisition

DPM

Data Product Manager

In some contexts DPM may colloquially be referring to the application daqDpmServer.

DICD

Data Interface Control Document

ELT

Extremely Large Telescope

FITS

Flexible Image Transport System

DHS

Data Handling Server where DPM deliver merged products to On-Line Archive System.

OCM

Observation Coordination Manager

In some contexts OCM may colloquially be referring to the application daqOcmServer.

OCS

Observation Coordination System

OLAS

On-Line Archive System

OLDB

On-Line Database

ICD

Interface Control Document (may also refer to MAL XML Interface document)

ICS

Instrument Control System

JSON

JavaScript Object Notation

MAL

Middleware Abstraction Layer

TBC

To be confirmed

TBD

To be determined

Definitions

Config Path

Term used in this document to refer to configuration paths resolved using the environment variable $CFGPATH. The path can either be absolute or relative. In case of relative path the file is resolved by testing the combination of each path in $CFGPATH with the provided relative path.

For example if $CFGPATH=/tmp/a:/tmp/b and Config Path is the file c the tested paths are (in order):

  1. /tmp/a/c

  2. /tmp/b/c

Commentary Keyword

In this document a FITS Commentary Keyword refers to those keywords that are neither Value Keywords or ESO Keywords. Commentary Keywords are special because the same keyword name may occur multiple times in the same HDU, to e.g. continue a comment over multiple records.

Note

Strictly speaking the HIERARCH keyword is also a commentary keyword but is treated specially in applications that support the convention.

Examples are:

COMMENT  Commentary keyword names may occur multiple times in a header, it may also
COMMENT  be contextual, as in this case, where a comment is continued over multiple
COMMENT  records.
HISTORY  File modified by user 'USER' on host on 2021-09-03T01:28:26
Data Acquisition

In this document it refers to the process of acquiring data as coordinated by OCM which results in one or more FITS files that contain the primary HDU and zero or more extensions such as binary tables, and or FITS keywords only. These are then merged together used to create a Data Product by DPM.

Refer to section Conceptual Model for a high level conceptual model of the process involved and Process Overview for a concrete and detailed overview.

Data Product

In this document it refers to the FITS science data product that is produced by DPM using the acquired data (see Data Acquisition) and normally archived in the ESO Online Archive System.

Data Product Specification

Specifies how to create a Data Product from input sources. This specification is created by daqDpmServer and used by daqDpmMerge.

ESO Keyword
ESO Hierarch Keyword

Refers to FITS keywords following the ESO HIERARCH keyword conventions RD6, i.e. keywords of the form:

HIERARCH ESO INS FILT1 ENC = 2 / Filter wheel absolute position [Enc].
HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.

The first token, INS in the example above, is referred to as the category.

Keyword Type

Refers to the different FITS standard[RD5] keyword record specifications which can be any of:

Logical Keyword Name

The logical name of a FITS keyword depends on the type of keyword, but there are common traits: Trailing white spaces are insignificant and are not part of the Logical Keyword Name.

For FITS Value Keyword and Commentary Keyword the logical name is the white-space trimmed name component. The logical names for

NAXIS1  =             2048 / # of pixels in axis1
COMMENT This table was written by 'APPLICATION'

are NAXIS1 and COMMENT respectively.

For ESO Keyword the logical name is the part between HIERARCH ESO up to =. For example the logical name is INS FILT1 ID for the HIERARCH keyword:

HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.
metadaqif

Standard metadata source interface used by e.g. FCF and other components [RD3].

daqOcmServer

Name of the OCM main server executable.

daqOcmCtl

Name of the OCM command line client executable.

recif

Standard primary data source interface implemented by DCSs [RD4].

Value Keyword

A FITS Value Keyword are those keywords that have a value indicator in bytes 9 and 10 (c.f. RD5). Example keywords are:

SIMPLE  =                T / Standard FITS
BITPIX  =                8 / # of bits per pix value
NAXIS   =                0 / # of axes in data array

Style Conventions

JSON Data Structures

This manual documents a number of JSON data structures, their “schema”, using these common conventions. In some cases a more formal documentation in the form of JSON Schema is also provided.

Typically a concrete example is first provided:

Listing 1 Example JSON data structure with sum-type list
 {
   "name": "example",
   "optional": 3.0,
   "nestedAnonymous": {
     "nestedProperty": "value"
   },
   "listOfObjectsProperty": [
     {
       "type": "unionTypeA",
       "property": "value"
     },
     {
       "type": "unionTypeB",
       "anotherProperty": "value"
     }
   ]
 }

This top level structure is then documented, sometimes with named sub-objects. The types of each property uses Python Variable Annotation syntax. For example a string would be str, a JSON object would be object, a list of strings would be List[str], a union of two alternative named objects like above would use Union[type1, type2], a list of union of named types would be List[Union[type1, type2]].

Unions are used when multiple types alternatives are allowed. The way it is typically used for is to have multiple object alternatives. In this case the union variant is discriminated using a type property with a fixed literal string value. The example in Listing 1 contain a list of union type using this pattern.

Each property of an object, consisting of a <name>, <type> and <default> is then documented as follows:

<name> (<type>) <default>

Documentation for property with name “<name>” type “<type>” and optionally a default value <default>.

If <type> must have a specific value the literal value may be indicated instead.

If property is optional the <type> can specify this using Python annotation syntax: Optional[<type>]. If there is a default value <default> specifies that as [default DEFAULTVALUE]. Lists are often optional and in this case it is simply indicated by a default empty list value: [default: []]. When a property is optional and no value should be provided the property should simply be omitted from the object rather than specify a null value.

If a property is an anonymous object with nested properties it will be documented inline with nested indentation (see example below).

Documenting the example structure above would look like:

Root object

name (str)

Name property.

optional (Optional[float]) [default: 1.0]

Optional property of float type with a default value of 1.0.

nestedAnonymous (object)

Example of how nested anonymous objects are documented.

nestedProperty (str)

A nested property of this anonymous object.

listOfObjectsProperty (List[Union[UnionTypeA, UnionTypeB])

Contains a list of either UnionTypeA or UnionTypeB types. See below for their structure.

UnionTypeA

Documents named object “UnionTypeA”.

type (“unionTypeA”)

Union discriminator and must have the literal value "unionTypeA".

property (str)

Example property of UnionTypeA.

UnionTypeB

Documents named object “UnionTypeB”.

type (“unionTypeB”)

Union discriminator and must have the literal value "unionTypeB".

anotherProperty (str)

Example property of UnionTypeB.