Data Product Specification¶

For an introductory high-level overview of how data products are created see section Data Product Creation.

The Data Product Specification is the primary input to daqDpmMerge and describes how the final Data Product is created. All source files referenced in the specification must be locally accessible and relative paths are relative to a single root path which is either the directory containing the specification or an explicit root directory passed as an option to daqDpmMerge.

Merging can be done by creating a new file or to merge in-place. If no FITS file target is specified using the property target/source described below, then a new empty file will be created automatically into which the sources will be added.

Note

It is expected that daqDpmMerge will validate keywords against a set of keyword dictionaries.

Primary HDU keywords¶

There are two types of sources for primary HDU keywords:

Keywords in JSON format.

These keywords were provided directly to daqOcmServer using the available APIs. These keywords are defined by the named JSON object SourceFitsKeywords.
Keywords copied from source FITS files.

Which keywords from the source primary HDU to copy to the target primary HDU are derived using the keyword rules specified in the source.

All these keywords will be merged together:

Value Keywords and ESO Keywords keyword collisions are resolved by keeping the keyword from highest priority source.
Commentary Keywords are appended in source priority order.

And then by default sorted as specified by DICD [RD1](section 3.1.1.3):

Value keywords are sorted by source order:

The individual order of value keywords from the same source (file or list of keywords) is kept to not reorder keywords that belong together. For example it make sense to keep RA and DEC together:
OBJECT  = 'karma_cdfs_8_LP.cat' / Original target.
RA      =        53.087446 / [deg] 03:32:20.9 RA (J2000) pointing
DEC     =        -27.84692 / [deg] -27:50:48.9 DEC (J2000) pointing
Rather than e.g. sorting alphabetically:
DEC     =        -27.84692 / [deg] -27:50:48.9 DEC (J2000) pointing
OBJECT  = 'karma_cdfs_8_LP.cat' / Original target.
RA      =        53.087446 / [deg] 03:32:20.9 RA (J2000) pointing

ESO hierarchical keywords are sorted by:
1. Category1 (DPR, OBS, TPL, GEN, TEL, ADA, INS, DET, any other)
2. Keyword name in alphabetic order.

1: Category in an ESO hierarchical keyword is the first token in the logical keyword name. The category is TEL in the logical keyword name TEL MOON RA. See also glossary ESO Keyword.

HDU Extensions¶

The FITS HDU extensions are merged in order of descending priority:

Extensions from the in-place target, if any.
Extensions from other source files.

The source files in attribute sources are specified in descending priority. Like keywords the relative order between extensions in the same file is kept.

JSON Description¶

Warning

Schemas are still work-in-progress and may change between versions.

For a description of the style convention and how JSON structures are documented see section JSON Data Structures.

A full example of the structure is provided in Listing 3 with description following below. But before introducing the full example Listing 2 shows the basic structure with minimal details.

Listing 2 Incomplete example to show the basic structure of the Data Product Specification. It has placeholders for a list of sources and the target, which in turn may specify an existing source file to use as a base.¶

{
  "id": "TEST.2021-05-18T14:49:03.905",
  "sources": [

  ],
  "target": {
    "fileId": "...",
    "source": {
    }
  }
}

Listing 3 Data Product Specification example of an in-place merge to FITS file from DCS and with keywords and HDU extensions from OCM, TCS and FCS (in that order).¶

{
  "id": "TEST.2021-05-18T14:49:03.905",
  "sources": [
    {
      "type": "fitsKeywords",
      "sourceName": "ocm",
      "keywords": [
        {
          "type": "valueKeyword",
          "name": "ORIGIN",
          "value": "ESO-PARANAL"
        },
        {
          "type": "valueKeyword",
          "name": "TELESCOP",
          "value": "ESO-ELT"
        },
        {
          "type": "valueKeyword",
          "name": "OBJECT",
          "value": "OBJECT,SKY"
        },
        {
          "type": "esoKeyword",
          "name": "OCS TEMPL ID",
          "value": "template-id"
        },
        {
          "type": "literalKeyword",
          "value": "COMMENT Example of a commentary keyword."
        }
      ]
    },
    {
      "type": "fitsKeywords",
      "sourceName": "tcs",
      "keywords": [
        {
          "type": "esoKeyword",
          "name": "TEL AIRM START",
          "value": 1.072
        },
        {
          "type": "esoKeyword",
          "name": "TEL AIRM END",
          "value": 1.08
        }
      ]
    },
    {
      "type": "fitsFile",
      "sourceName": "fcs",
      "origin": "fcs-host:/path/to/somefile.fits",
      "keywordRules": [
        {
          "type": "filter",
          "selectionPatterns": [
            "+e INS *",
            "-e INS TEMP[12] *",
            "+v VALUEKW"
          ]
        },
        {
          "type": "transform",
          "selectionPatterns": [
            "+e INS *"
          ],
          "regex": "^INS ",
          "format": "INS2 "
        }
      ]
    }
  ],
  "target": {
    "fileId": "TEST.2021-05-18T14:49:03.905",
    "source": {
      "type": "fitsFile",
      "sourceName": "dcs",
      "origin": "dcs-host:/path/to/somefile.fits",
    }
  },
  "keywordDictionaries": [
    "instrument.did"
  ]
}

Root object description (`/`)¶

id (str)

Data Acquisition identifier.

sources (List[Union[SourceFitsFile, SourceFitsKeywords]])

List of input sources in descending priority order. The only source with higher priority is if a /target/source file has been specified.

Important

The order of sources files determines for the final Data Product:

Keyword order (value and commentary keywords are grouped by source order).
HDU extension order in the resulting.

SourceFitsFile

Specifies a FITS file from which:

Keywords from primary HDU, selected using keyword rules, will be copied to target.
HDU extensions that will be copied to target.

SourceFitsKeywords

List of FITS keywords that will be used to update Data Product primary HDU. The list is sorted in descending priority order where higher priority keywords have the ability to overwrite lower priority keywords.

This list includes keywords from Data Acquisition metadata sources and daqOcmServer itself.

See below for how SourceFitsFile and SourceFitsKeywords are defined.

target (object)

Specification for the target.

fileId (str)

OLAS FileId which globally identifies Data Product in the ESO archive and must be unique.

To be determined if this is required or if it can be derived from id.

Note

Uniqueness is ensured by having daqOcmServer create the id and that the following assumptions hold true:

There is only one daqOcmServer instance for each ICS.
Each ICS has a unique instrument prefix.
The time used to generate the identifier is correct.

filePrefix (Optional[str])

Optional prefix that if defined will be prefixed to the resulting filename.

source (Optional[SourceFitsFile])

Optionally specifies a source input file to be used as the in-place target when merging.

Note

This is the only file that can have a Data Unit in the primary HDU.

Using a source also as target implies the following:

It is the highest priority source.
Keywords rules are applied as with any other keyword source.
All HDU extensions (if any) are included as-is.

Important

The target source file should have enough space to hold all additional keywords being merged into it to avoid possible and likely expensive relocation of data arrays and HDU extensions.

If a target is not specified a new empty source file will be created automatically.

keywordDictionaries (List[str]): Specifies the keyword dictionaries to load when validating and formatting keywords.

Object descriptions¶

SourceFitsFile (object)

Describes a source FITS file that either acts as an in-place target or should be merged in the final Data Product.

Keywords from FITS file in origin will be processed by the keywordRules before merging them to final Data Product primary HDU.

FITS extensions will be copied as-is.

type (“fitsFile”)

This is a union discriminator and must have the literal string value "fitsFile".

sourceName (str)

Identifies the source, as provided to OCM by client.

origin (str)

Origin location as host:/path/to/file.fits format. This is the location as originally provided by the source to daqOcmServer.

keywordRules (KeywordRules) [default: []]

Specifies the keyword rules used when merging keywords from this source primary HDU to final Data Product primary HDU.

See KeywordRules for details.

SourceFitsKeywords (object)

Specifies a list of keywords from a source that should be merged to final Data Product primary HDU.

Keywords from keywords are processed by the keywordRules and then merged to Data Product.

type (“fitsKeywords”)

This is a union discriminator and must have the literal string value "fitsKeywords".

sourceName (str)

Identifies the source, as provided to OCM by client.

keywords (List[FitsKeyword])

List of FITS keywords.

See FITS Keywords for how FitsKeyword is defined.

keywordRules (KeywordRules) [default: []]

Specifies the keyword rules. If rules are not provided all non-structure keywords will be merged.

See KeywordRules for details.

KeywordRules (List[Union[KeywordRuleFilter, KeywordRuleTransform]])

Specifies which keywords from this source will merged to final Data Product primary HDU. If no rules are specified, all keywords are included by default.

Each rule can be thought of a function that accepts a list of keywords and returns a list of keywords. A list of rules is can be thought of functions that are chained. Expressed in pseudo code with two filter functions first and second it may look like:

result = second(first(keywords))

or:

result = first(keywords) | second(keywords)

By specifying a combination of the supported filter and transform rules daqDpmMerge support common operations like

Select which keywords to include in primary HDU, by Keyword Type and/or Logical Keyword Name.
Rename ESO Hierarch keyword category to avoid keyword collisions.

Each rule is executed in order with the input from the previous output. The input of the first rule are all non-structure FITS keywords.

KeywordRuleFilter (object)

Specifies a filter rule with include and exclude patterns.

type (“filter”)

This is a union discriminator and must have the literal string value "primaryHdu".

patterns (List[str])

List of pattern rules that build up a list of keywords by successively adding or subtracting keywords from a list that finally will be included.

Each rule have the following syntax:

<rule> ::= <operator><scope> ' ' <pattern>
<operator> ::= '+' | '-'
<scope> ::= 'v' | 'e' | 'c'
<pattern> ::= <ascii-text> | <wildcard> | <pattern>
<ascii-text> ::= [#x20 - #x7e] /* ' ' - '~' */
<wildcard> ::= '*' | '?' | '[' <ascii-text> ']'

<operator>

Specifies the operator to apply for matching keywords.

+: Additive operator that cause keywords matching pattern in source to be added to the keyword list.
-: Subtractive operator that cause keywords matching rule to removed from keyword list.

<scope>

Specifies the scope of the pattern matching by limiting to the specified keyword type. The pattern matching is performed against the Logical Keyword Name.

v: Denotes Value Keywords.
e: Denotes ESO Hierarch Keywords.
c: Denotes Commentary Keywords.

<wildcard>

Shell wildcard pattern (for additional details, see man fnmatch(3)).

*: Matches any string.
?: Matches any single character.
[: Starts a character class matching any character enclosed in expression [...]. The matching can be negated with [!...] so that it matches any character not in the enclosed expression.

<ascii-text>

Legal FITS ASCII characters. Characters “*”, “?” and “[” must be escaped with “” to prevent the special wildcard meaning. See manpage glob().

Example that includes the “VALUEKW” keyword as well as ESO hierarchical keywords in the INS category, excluding the INS TEMP1 * and INS TEMP2 * keywords:

["+v VALUEKW", "+e INS *", "-e INS TEMP[12] *"]

Example that includes all “COMMENT” keywords.

["+c COMMENT"]

KeywordRuleTransform

Allows the transformation the logical keyword name of a selection of keywords.

This can for example be used to change the ESO Hierarch Keyword category to prevent name collision.

TBD if this will be supported.

Example:

{
  "type": "transform",
  "selectionPatterns": [
    "+e INS *"
  ],
  "regex": "^INS ",
  "format": "INS2 "
}

Data Product Specification¶

Primary HDU keywords¶

HDU Extensions¶

JSON Description¶

Root object description (/)¶

Object descriptions¶

Root object description (`/`)¶