Introduction

Acronyms

OCM

Observation Coordination Manager

DPM

Data Product Manager

Definitions

Commentary Keyword

In this document a FITS Commentary Keyword refers to those keywords that are neither Value Keywords or ESO Keywords. Commentary Keywords are special because the same keyword name may occur multiple times in the same HDU, to e.g. continue a comment over multiple records.

Note

Strictly speaking the HIERARCH ESO keyword is also a commentary keyword but is treated specially in applications that support the convention.

Examples are:

COMMENT  Commentary keyword names may occur multiple times in a header, it may also
COMMENT  be contextual, as in this case, where a comment is continued over multiple
COMMENT  records.
HISTORY  File modified by user 'USER' on host on 2021-09-03T01:28:26

ESO Keyword

ESO Hierarch Keyword

Refers to FITS keywords following the ESO HIERARCH keyword conventions RD4, i.e. keywords of the form:

HIERARCH ESO INS FILT1 ENC = 2 / Filter wheel absolute position [Enc].
HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.

The first token, INS in the example above, is referred to as the category.

Logical Keyword Name

The logical name of a FITS keyword depends on the type of keyword, but there are common traits: Trailing white spaces are insignificant and are not part of the Logical Keyword Name.

For FITS Value Keyword and Commentary Keyword the logical name is the white-space trimmed name component. The logical names for

NAXIS1  =             2048 / # of pixels in axis1
COMMENT This table was written by 'APPLICATION'

are NAXIS1 and COMMENT respectively.

For ESO Keyword the logical name is the part between HIERARCH ESO up to =. For example the logical name is INS FILT1 ID for the HIERARCH keyword:

HIERARCH ESO INS FILT1 ID = 'OUT' / Filter unique id.

Value Keyword

A FITS Value Keyword are those keywords that have a value indicator in bytes 9 and 10 (c.f. RD3). Example keywords are:

SIMPLE  =                T / Standard FITS
BITPIX  =                8 / # of bits per pix value
NAXIS   =                0 / # of axes in data array

Reference Documents

[RD1]

Central Control System Development Standards;

ESO-366378 v1

[RD2]

Data Interface Control Document;

ESO-044156 v6

[RD3]

Definition of the Flexible Image Transport System (FITS);

https://fits.gsfc.nasa.gov/standard30/fits_standard30aa.pdf

[RD4]

The ESO HIERARCH Keyword Conventions;

https://fits.gsfc.nasa.gov/registry/hierarch/hierarch.pdf

Metadata Acquisition Interface

Metadata acquisition interface, metadaqif, is a relatively small interface used by components that provide metadata to FITS data products created by OCM and DPM.

The following sections document the interface in a language agnostic manner. For data structures only the data members are documented, not the accessors generated by MAL. Similarly names are not fully qualified as it is different across the supported languages.

Interfaces

class metadaqif.MetaDaq

Metadata acquisition interface that supports acquiring arbitrary data to either FITS files and/or FITS keywords.

Interface supports concurrent acquisitions which is why the Data Acquisition id is provided in requests and replies as a handle.

Note

It is out of scope for this interface to provide behaviour modification of the application performing the acquisition. This is domain specific and needs to be provided by other means such as configuration or another MAL interface.

Success

A successful sequence looks like follows:

1. StartDaq() initiates a new acquisition, which if successful returns DaqReply as acknowledgement. State is now DaqState.Acquiring which indicates to clients that Data Acquisition has started and is in progress.

   It is not specified or required that data must be physically acquired from this point. An implementation may use another form of synchronization to determine exactly how the acquisition is performed (see DaqState.Acquiring).

2. StopDaq() is issued at the time the Data Acquisition should stop. FITS files in progress of being produced are closed and placed in a stable predetermined location for later retrieval. Only after files have been closed and keyword produced is a reply sent. Again if another form of synchronization is used it is anyway expected that that reply is sent only when Data Acquisition has completed.

Aborting

If something happens that results in that the in-progress acquisition should be aborted and data discarded the sequence looks like:

1. StartDaq() (see above).

2. AbortDaq() Data Acquisition is stopped and any acquired data can be discarded.

Error Recovery

If there is a communication problem or something else happens that the current state of a Data Acquisition is lost clients may use GetDaqStatus() to query for current status and recover from there.

StartDaq(id) → DaqReply

Start new Data Acquisition that is eventually stopped with StopDaq() or aborted with AbortDaq().

Note

If id is not provided (left empty) it is expected that the server implementing the interface will generate a unique identifier automatically.

If the same id is reused it should be considered a fatal error.

Parameters

id (str) – Optional (may be empty) unique identifier of Data Acquisition.

Returns

If id was provided it is returned as an acknowledgement, otherwise the id generated by server is returned.

Return type

DaqReply

Raises

DaqException – On fatal error.

StopDaq(id) → DaqStopReply

Stops data acquisition and returns created FITS filenames and/or keywords. Produced files should be closed so it is safe to immediately read them.

If an error occurred such that the acquisition has failed this is communicated by throwing an exception.

Parameters

id (str) – Id of data acquisition to stop.

Returns

Structure containing produced FITS files and/or FITS keywords.

Return type

DaqStopReply

Raises

DaqException – On fatal error.

AbortDaq(id) → DaqReply

Aborts data acquisition and discards any data that has been acquired.

Parameters

id (str) – Id of data acquisition to abort.

Return type

DaqReply

Raises

DaqException – On fatal error.

GetDaqStatus(id) → DaqStatus

Get status of current or past Data Acquisitions.

It is unspecified exactly how far back the history should go. But the bare minimum is to be able to provide status for last two (e.g. any current and previous).

Parameters

id (str) – Id of data acquisition to get status for.

Return type

DaqStatus

Raises

DaqException – On fatal error.

Data Structures

class metadaqif.DaqState

Enumeration of data acquisition states. The expected state transitions are as follows:

Note

There may be more internal (transitional states) but this enumeration covers the ones that are observable with metadaqif.

NotStarted

Data Acquisition is created but not yet started.

Acquiring

Logical state where data is acquired. It does not imply that at each time point data is physically being acquired. Implementations can choose to e.g.:

• Synchronize to an implementation specific event or events, such as a sequence number or time point.

• Sample at specific intervals.

• Sample once at the beginning of a Data Acquisition when MetaDaq.StartDaq() is first received and then again at the end when MetaDaq.StopDaq() is received.

• Continuously acquire time series until requested to stop.

Configuring the implementation specific behaviour is out of scope for metadaqif.

Succeeded

Final state for a successful Data Acquisition. All data has been acquired and any FITS files are completed and closed.

Aborted

Final state for an aborted Data Acquisition.

Failed

Final state for failed Data Acquisition.

class metadaqif.DaqException

Exception used by MetaDaq.

id: str

Data Acquisition identifier.

message: str

Exception message.

class metadaqif.DaqStatus

Contains the Data Acquisition reply.

id: str

Data Acquisition identifier.

state: DaqState

Data Acquisition state at the time the reply was sent.

message: str

Message, if any.

files: List[str]

List of FITS files created for this Data Acquisition in the format [user@]host:/absolute/path.

keywords: str

JSON-encoded FITS keywords, or empty. See JSON Keywords for more details.

Example:

[
   {
      "type":"valueKeyword",
      "name":"OBJECT",
      "value":"OBJECT,SKY"
   },
   {
      "type":"esoKeyword",
      "name":"OBS TPLNO",
      "value":2
   }
]

timestamp: double

Timestamp of last status update in number of seconds in TAI time standard with epoch set to 1 January 1970 00:00:00 TAI, which is 31 December 1969 23:59:51.999918 UTC RD1.

class metadaqif.DaqReply

Common reply type.

id: str

Data Acquisition identifier.

class metadaqif.DaqStopReply

Reply structure for DaqStop.

id: str

Data Acquisition identifier.

files: List[str]

List of FITS files created for this Data Acquisition in the format [user@]host:/absolute/path.

keywords: str

JSON-encoded FITS keywords, or empty. See JSON Keywords for more details.

Example:

[
   {
      "type":"valueKeyword",
      "name":"OBJECT",
      "value":"OBJECT,SKY"
   },
   {
      "type":"esoKeyword",
      "name":"OBS TPLNO",
      "value":2
   }
]

JSON Keywords

FITS keywords are provided as a JSON-array of FitsKeyword objects, where each object describe one keyword. The following example show one FITS standard value keyword, one ESO hierarchical keyword and one literal keyword:

[
   {
      "type":"valueKeyword",
      "name":"OBJECT",
      "value":"OBJECT,SKY"
   },
   {
      "type":"esoKeyword",
      "name":"OBS TPLNO",
      "value":2
   },
   {
      "type":"literalKeyword",
      "name":"ORIGIN  = 'ESO-PARANAL'    / European Southern Observatory"
   }
]

Note

The annotations use Python Variable Annotation syntax:

• str represent a string.

• number represent a JSON number.

• boolean represent a JSON boolean.

• object represents a JSON object.

• Union[A, B] represents a union where either type A or B are valid.

• Optional[A] indicates that the property of type A is optional and can be omitted.

FitsKeyword (Union[ValueKeyword, EsoKeyword, LiteralKeyword])

This union object represent any valid FITS keyword.

ValueKeyword (object)

This object represents a FITS value keyword[RD3].

type (“valueKeyword”)

This is a union discriminator and must have the literal string value "valueKeyword".

name (str)

Logical Keyword Name (up to 8 characters).

Examples:

• RA

• DEC

• OBJECT

value (Union[str, boolean, number])

For both valueKeyword and esoKeyword the value field provide the typed keyword value. See JSON to FITS type conversion for mapping between JSON and FITS.

comment (Optional[str])

Optional comment.

EsoKeyword (object)

This object represents a HIERARCH ESO FITS keyword [RD4].

type (“esoKeyword”)

This is a union discriminator and must have the literal string value "esoKeyword".

name (str)

Logical Keyword Name which does not include the HIERARCH ESO prefix.

Examples:

• DET CHIP GAIN

• TEL AIRM START

• ADA GUID RA

value (Union[str, boolean, number])

For both valueKeyword and esoKeyword the value field provide the typed keyword value. See JSON to FITS type conversion for mapping between JSON and FITS.

comment (Optional[str])

Optional comment.

LiteralKeyword (object)

This object represents a fully formatted FITS keyword record.

type (“literalKeyword”)

This is a union discriminator and must have the literal string value "literalKeyword".

value (str)

Fully formatted FITS keyword record with optional trailing spaces.

JSON to FITS type conversion

JSON	FITS	Notes	Example
string	string	Do not use single-qotes '. Timepoints are represented as strings.	"foobar" -> 'foobar', "2020-07-28T04:57:00.8836" -> '2020-07-28T04:57:00.8836'
boolean	logical		true -> T, false -> F
number	integer or float	Dictionary will be used to format the value correctly (not yet implemented). integer range: -9223372036854775807 .. 9223372036854775807 float range: -1.79769313486231e+308 .. 1.79769313486231e+308