Note

For a description of the style convention and how JSON structures are documented see section JSON Data Structures.

Named JSON structures are indicated as ^{JSON}.

Data Product Specification Schema

Warning

Schemas are still work-in-progress and may change between versions.

For a description of the style convention and how JSON structures are documented see section JSON Data Structures.

A full example of the structure is provided in Listing 3 with description following below. But before introducing the full example Listing 2 shows the basic structure with minimal details.

Listing 2 Incomplete example to show the basic structure of the Data Product Specification. It has placeholders for a list of sources and the target, which in turn may specify an existing source file to use as a base.

{
  "id": "TEST.2021-05-18T14:49:03.905",
  "sources": [

  ],
  "target": {
    "fileId": "...",
    "source": {
    }
  }
}

Listing 3 Data Product Specification example of an in-place merge to FITS file from DCS and with keywords and HDU extensions from OCM, TCS and FCS (in that order).

{
  "id": "TEST.2021-05-18T14:49:03.905",
  "sources": [
    {
      "type": "fitsKeywords",
      "sourceName": "ocm",
      "keywords": [
        {
          "type": "valueKeyword",
          "name": "ORIGIN",
          "value": "ESO-PARANAL"
        },
        {
          "type": "valueKeyword",
          "name": "TELESCOP",
          "value": "ESO-ELT"
        },
        {
          "type": "valueKeyword",
          "name": "OBJECT",
          "value": "OBJECT,SKY"
        },
        {
          "type": "esoKeyword",
          "name": "OCS TEMPL ID",
          "value": "template-id"
        },
        {
          "type": "literalKeyword",
          "value": "COMMENT Example of a commentary keyword."
        }
      ]
    },
    {
      "type": "fitsKeywords",
      "sourceName": "tcs",
      "keywords": [
        {
          "type": "esoKeyword",
          "name": "TEL AIRM START",
          "value": 1.072
        },
        {
          "type": "esoKeyword",
          "name": "TEL AIRM END",
          "value": 1.08
        }
      ]
    },
    {
      "type": "fitsFile",
      "sourceName": "fcs",
      "location": "fcs-host:/path/to/somefile.fits",
      "keywordRules": [
        {
          "type": "filter",
          "selectionPatterns": [
            "+e INS *",
            "-e INS TEMP[12] *",
            "+v VALUEKW"
          ]
        },
        {
          "type": "transform",
          "selectionPatterns": [
            "+e INS *"
          ],
          "regex": "^INS ",
          "format": "INS2 "
        }
      ]
    }
  ],
  "target": {
    "fileId": "TEST.2021-05-18T14:49:03.905",
    "source": {
      "type": "fitsFile",
      "sourceName": "dcs",
      "location": "dcs-host:/path/to/somefile.fits",
    }
  },
  "keywordDictionaries": [
    "instrument.did"
  ]
}

DpSpecification ^{JSON}

(object)

id (str)

Data Acquisition identifier.

sources (List[Union[FitsFileSource, FitsKeywordsSource]])

List of input sources in descending priority order. The only source with higher priority is if a /target/source file has been specified.

Important

The order of sources files determines for the final Data Product:

• Keyword order (value and commentary keywords are grouped by source order).

• HDU extension order in the resulting.

FitsFileSource

Specifies a FITS file from which:

• Keywords from primary HDU, selected using keyword rules, will be copied to target.

• HDU extensions that will be copied to target.

FitsKeywordsSource

List of FITS keywords that will be used to update Data Product primary HDU. The list is sorted in descending priority order where higher priority keywords have the ability to overwrite lower priority keywords.

This list includes keywords from Data Acquisition metadata sources and daqOcmServer itself.

See also

Specification for FitsFileSource and FitsKeywordsSource.

target (object)

Specification for the target.

fileId (str)

OLAS FileId which globally identifies Data Product in the ESO archive and must be unique.

To be determined if this is required or if it can be derived from id.

Note

Uniqueness is ensured by having daqOcmServer create the id and that the following assumptions hold true:

• There is only one daqOcmServer instance for each ICS.

• Each ICS has a unique instrument prefix.

• The time used to generate the identifier is correct.

filePrefix (Optional[str])

Optional prefix that if defined will be prefixed to the resulting filename.

source (Optional[FitsFileSource])

Optionally specifies a source input file to be used as the in-place target when merging.

Note

This is the only file that can have a Data Unit in the primary HDU.

Using a source also as target implies the following:

• It is the highest priority source.

• Keywords rules are applied as with any other keyword source.

• All HDU extensions (if any) are included as-is.

Important

The target source file should have enough space to hold all additional keywords being merged into it to avoid possible and likely expensive relocation of data arrays and HDU extensions.

If a target is not specified a new empty source file will be created automatically.

keywordDictionaries (List[str])

Specifies the keyword dictionaries to load when validating and formatting keywords.