Configuration

The CalOb Configuration Set consists of the following types of documents:

  • Configuration File.

  • Calibration Schedule.

  • External Functions.

  • Static Calibration OBs.

  • Template Calibration OBs.

These documents shall be provided in order to define the CalOb scheduling properties for a given instrument. The documents are described below, providing examples.

../_images/ifwcalob-configuration.png

Configuration File

An example of the configuration is shown in the following:

!cfg.include config/ifw/calob/schema/calob_cfg.schema.yaml:

# IFW Calob Tool Example Configuration

database: !cfg.type:DatabaseDef
  db_path: "calob/db/calibration_history.db"

calob: !cfg.type:CalobDef
  calib_schedule_module: "ifw.calob.example.FORSUP_calibration_schedule"
  log_property_file: "config/ifw/calob/log/log_config.json"
  console_logger: "elt.log.test"

fits_crawler: !cfg.type:FitsCrawlerDef
  # Note: Normally data will be stored under the "DATAROOT" area and "data_path"
  # should be set to the relative path under "DATAROOT".
  data_path: "$INTROOT/resource/image/ifw/calob/example"
  file_pattern: "*.fits"
  timeframe_start: ""
  timeframe_end: ""

exporter_ob: !cfg.type:ExporterObDef
  path_dc_template: "resource/template/ifw/calob/daily_calib.obd.json"
  path_dc: "calob/daily_calibration/daily_calib.obd.json"
  obd_path: "resource/obd/ifw/calob/example"
  obdx_path: "resource/obdx/ifw/calob/example"

The parameters are as follows:

Category: “database”

db_path


Path to the name of the SQLite database file, in which the history
of previous executions of the CalOb is recorded.
It is given as a path, relative to “CFGPATH”.

Category: “calob”

General configuration parameters.

calib_schedule_module



Name of the Calibration Plan contained in a Python module.
Since this is loaded by the Python interpreter it needs
to be located in a path defined by the “PYTHONPATH”
environment variable.
log_property_file

Log properties file for the CII Log Service (“log4py”).

console_logger

The CII Logger defined in the configuration

Category: “fits_crawler”

Parameters related to the internal FITS scanning service.

data_path


Location top directory where to scan for science data
(FITS files).
Given relative to the “DATAROOT”.
file_pattern

File pattern to apply when performing the FITS file
scanning. Example: “FORSUP*.fits”.

timeframe_start

Start of timeframe to use for FITS file filtering. Example: “2023-07-01”

timeframe_end

End of timeframe to use for FITS file filtering. Example: “2023-08-01”

Category: “exporter_ob”

Parameters defining the properties for exporting (generating) the resulting Daily Calibration OB.

path_dc_template


Template JSON document, used for creating the
Calibration Observations entries in the
Daily Calibration OB.

path_dc

Name of output Daily Calibration OB document.

obd_path

Path for location where to look for Static
Calibration OBs.
obdx_path

Path for location where to look for Template
Calibration OBs.

Calibration Schedule

A Calibration Plan consists of the following elements:

  • External Functions: External Functions are user defined functions to carry out any particular procedure required in the processing of the Template Calibrations. They return a value that can be assigned to a key defined in the calibration, which will be stored in the output Daily Calibration OB. They take the headers of the current FITS file being processed as input. Implentation wise they are methods in a class, derived from “ifw.calob.calibration_plan.InstrumentCalibrationExternalFunctions”.

  • Calibration Tables: Class derived from the “ifw.calob.calibration_plan.InstrumentCalibrationTables” base class, which defines tables with key/value associations, mapping into an output value, assigned to a key in the target Daily Calibration OB.

  • Calibration Schedule: Class derived from “ifw.calob.calibration_plan.InstrumentCalibrationSchedule”, which defines the keywords for the output Daily Calibration OB and how the values of these keywords shall be processed.

A realistic example of a Calibration Schedule, can be found here.

In the following, the main structure of the FORSUP Calibration Schedule is shown.

from ifw.calob.calibration_plan import InstrumentCalibrationExternalFunctions, InstrumentCalibrationTables, InstrumentCalibrationSchedule

class FORSCalibrationExternalFunctions(InstrumentCalibrationExternalFunctions):
    def __init__(self):
        super().__init__()

    ...

class FORSCalibrationTables(InstrumentCalibrationTables):

    def __init__(self):
        super().__init__()

    ...

class FORSCalibrationSchedule(InstrumentCalibrationSchedule):

    def __init__(self,
                 cc = None):

        super().__init__(cc)
        super().set_calibration_library(FORSCalibrationLibrary())

       ...

An important element used in the tool is the “HeaderDict”, which is Python dictionary with some additional functionality. This is defined as follows:

class HeaderDict(dict):
  """
  @brief Class used to hold a FITS file header
  @details
  Subclass from dict which adds some additional handling
  when retrieving keys

  @ingroup common
  """
  def set_g(self, g):
      self.g = g

  def get(self, key):
      """
      Function that gets a key from g first and in case
      of failure, retrieves the value from the header
      """
      if key in self.g:
          return(self.g[key])
      else:
          return(self.__getitem__(key))

  def __getitem__(self,key):
      """
      @brief Override of __getitem__ that handles some variations
      @details
      in the key definition:
      1. Replaces . for spaces in the key name
      2. Tries to get the key in that form
      3. In case of failure it adds ESO to the start and tries
         to get the key again
      4. In case of failure it returns an empty string
      """
      try:
          key = key.replace("."," ").upper()
          val = dict.__getitem__(self, key)
      except KeyError:
          try:
              val = dict.__getitem__(self, f"ESO {key}")
          except KeyError:
              val = ""

      return val

  def __contains__(self, key):
      """
      @brief Override of __contains__ that handles some variations
      @details
      in the key definition:
      1. Replaces . for spaces in the key name
      2. Tries to get the key in that form
      3. In case of failure it adds ESO to the start and tries
         to get the key again
      4. In case of failure it returns False
      """

      key = key.replace("."," ").upper()
      if dict.__contains__(self, key) :
          return True
      elif dict.__contains__(self, f"ESO {key}") :
          return True
      else:
          return False

This dictionary is called “h” in the external functions and template calibrations. Used as a regular dictionary, for example:

time = h["INS.LAMP1.TIME"]

will retrieve the key “INS.LAMP1.TIME” from the header, but will try to standardise the input and also attempt to match a variation. The dictionary also add the function “get”, for example:

time = h.get("INS.LAMP1.TIME")

In this case the key will be searched on the Template Calibration keys that have been already produced, so if for example the Template Calibration had already defined the searched key by another means, then that would be the result. In case the keyword is not found, it will then fallback and look into the available headers.

Built-In Functions

The following Built-In Functions are provided to define rules for when to schedule Calibration Observations:

rule_min_days_since_last_exec

Sets the minimum days that have to pass before the CalibrationProp is executed again

rule_only_if_key_eq

Adds a rule where key in the FITS header has to match value

rule_only_if_key_diff

Adds a rule where key in the FITS header has to be different to value

External Functions

An example of a user defined (external) function is shown in the following:

def forsup_mos(self, h, clb):
    txt = ""
    mosmax = int(clb.tbl['USERVAR']['MOS_MAX_SLITS'])
    for i in range(1, mosmax + 1):
        key = f"INS.MOS{i}"
        pos = float(h[key + ".POS"])
        wid = float(h[key + ".WIDTH"])
        txt += f"{key}.POS {pos:.3f} {key}.WIDTH {wid:.3f} "

    return txt

The “clb” parameter is a reference to the Calibration Plan object.

Calibration Tables

An example of a table definition in the Calibration Table class is shown in the following:

self.tbl['DET.MODE'] = {
         ("100Kps/2ports/high_gain", 2, 2) : "100kHz,2x2,high",
         ("100Kps/2ports/high_gain", 1, 1) : "100kHz,1x1,high",
         ("200Kps/2ports/low_gain",  2, 2) : "200kHz,2x2,low",
         ("200Kps/2ports/low_gain",  1, 1) : "200kHz,1x1,low",
         ("625Kps/2ports/low_gain",  2, 2) : "200kHz,2x2,low",
         ("HIT-MS",                  2, 2) : "HIT-MS",
         ("HIT-OS1-1sec",            1, 1) : "100kHz,2x2,high",
         ("HIT-OS2-4sec",            1, 1) : "100kHz,2x2,high",
         ("HIT-OS3-16sec",           1, 1) : "100kHz,2x2,high",
         ("HIT-OS4-64sec",           1, 1) : "100kHz,2x2,high",
         ("HIT-OS5-256sec",          1, 1) : "100kHz,2x2,high",
         ("HIT-OS6-1024sec",         1, 1) : "100kHz,2x2,high",
         ("HIT-OS1-1sec",            2, 2) : "100kHz,2x2,high",
         ("HIT-OS2-4sec",            2, 2) : "100kHz,2x2,high",
         ("HIT-OS3-16sec",           2, 2) : "100kHz,2x2,high",
         ("HIT-OS4-64sec",           2, 2) : "100kHz,2x2,high",
         ("HIT-OS5-256sec",          2, 2) : "100kHz,2x2,high",
         ("HIT-OS6-1024sec",         2, 2) : "100kHz,2x2,high"
}

The tables are used in the Template Calibration OBs to map keyword values or set of keyword values into a value to assign to another key.

An example of an assignment in a Template Calibration OB, involving the table above, can be found in FORSUP_img_cal_bias.obdx.json.

In this Template Calibration OB, the key “DET.READ.CLKIND”, is assigned a value from the “DET.MODE” table above, as shown in this snippet from the template:

{
  "name": "DET.READ.CLKIND",
  "type": "string",
  "value": "<% clb.tbl['DET.MODE'][(h.get('DET.READ.CLOCK'), h.get('DET.WIN1.BINX'), h.get('DET.WIN1.BINY'))] %>"
},

The values from the keys “DET.READ.CLOCK”, “DET.WIN1.BINX” and “DET.WIN1.BINY”, are used to index the “DET.MODE” table to return the value to assign to “DET.READ.CLKIND”.

E.g., “clb.tbl[‘DET.MODE’][(‘HIT-MS’, 2, 2)]” yields the value “HIT-MS”.

Calibration Schedule Class

The Calibration Schedule Class in the Calibration Plan python module, is used to define the sequence Static and Template Calibration OBs to be scheduled in the output Daily Calibration OB.

For each Calibration Observation to consider for scheduling, it is possible to specify various conditions for when to schedule it.

The following snippet from the example Configuration Schedule illustrates this:

def init_calibration_schedule(self):
  condition  = "(h['ESO DPR CATG'] == 'SCIENCE' or "
  condition += " h['ESO DPR CATG'] == 'CALIB' or "
  condition += " h['ESO DPR CATG'] == 'ACQUISITION') and "
  condition += " h['ESO DPR TYPE'] != 'BIAS'"
  super().add_filter_condition(condition)

  # Static calibrations
  ...
  super().add_static_calib("FORSUP-focus-blue.obd.json") \
      .rule_only_if_key_eq("CCD", "BLUE") \
      .rule_min_days_since_last_exec(7)

  # Template calibrations
  ...
  super().add_template_calib("FORSUP_mos_cal_daycalib.obdx.json") \
      .rule_min_days_since_last_exec(1) \
      .rule_only_if_key_eq("INS.MODE", "MOS") \
      .rule_only_if_key_diff("TPL.ID", "FORSUP_specphot_obs_exp_fast")
  ...

The first part defines an overall filter to pass to the crawler in this case, were only the FITS files for which the condition is evaluated to True are included in the subset for which to create calibrations.

Next, the Static Calibration OBs are defined.

In the above example the FORSUP-focus-BLUE.obd.json Static Calibration OB, is scheduled if the FITS header key “CCD”, has the value “BLUE”. In addition it is specified to schedule this OB once per week (“every 7 days”).

For the Template Calibration OB example above, the FORSUP_mos_cal_daycalib.obdx.json Template Calibration OB, is scheduled on a daily basis the keyword “INS.MODE” is found in a FITS header in the set of FITS files analysed with value “MOS” and if the file is generated by executing the “FORSUP_specphot_obs_exp_fast” template.

Static Calibration OBs

The Static Calibration OBs are ready-to-use ‘sub-OBs’ which can be inserted in the Daily Calibration OB ‘as-is’, if the conditions for scheduling these are fulfilled.

An example excerpt of a Static Calibration OB is shown in the following (FORSUP-lampcheck-new.obd.json):

{
  "obId": 0,
  "itemType": "OB",
  "name": "FORSSUP_lampcheck_new",
  "executionTime": 0,
  "runId": "string",
  "instrument": "FORSUp",
  "ipVersion": "string",
  "obsDescription": {
    "name": "f2ocal-lampcheck-new",
    "userComments": "A",
    "instrumentComments": "AA"
  },
  "templates": [
    {
      "templateName": "FORSUP_img_cal_coll",
      "type": "string",
      "parameters": [
        {
          "name": "TPL.NAME",
          "type": "string",
          "value": "Collimator template"
        },
        {
          "name": "TPL.MODE",
          "type": "string",
          "value": "IMG"
        },
        {
          "name": "TPL.SEQNO",
          "type": "integer",
          "value": 2
        },
        {
          "name": "INS.COLL.NAID",
          "type": "string",
          "value": "COLL_SR+6"
        }
      ]
    },
    ...

In the example above, the template “FORSUP_img_cal_coll”, is scheduled for execution in the Daily Calibration OB. It is being assigned the fixed values as indicated.

Template Calibration OBs

Template Calibration OBs are OBs, which contain keyword values that will be computed by CalOb, when processing the Calibration Plan.

An example of a Template Calibration OB is shown in the following (FORS2_img_cal_coll.obdx.json)

{
  "obId": 0,
  "itemType": "OB",
  "name": "FORS2_img_cal_coll",
  "executionTime": 0,
  "runId": "string",
  "instrument": "FORSUp",
  "ipVersion": "string",
  "obsDescription": {
    "name": "FORS2_img_cal_coll",
    "userComments": "A",
    "instrumentComments": "AA"
  },
  "templates": [
    {
      "templateName": "FORS2_img_cal_coll",
      "type": "string",
      "parameters": [
        {
          "name": "TPL.NAME",
          "type": "string",
          "value": "Collimator template"
        },
        {
          "name": "INS.COLL.NAID",
          "type": "string",
          "value": "<% h['INS.COLL.NAME'] + h['INS.COLL.ID'] %>"
        }
      ]
    }
  ]
}

In the example above, a template invocation of the “FORSUP_img_cal_coll”, with the parameters and values listed, is scheduled in the Daily Calibration OB.

Note that the values of the key “INS.COLL.NAID” is calculated with the expression “<% h[‘INS.COLL.NAME’] + h[‘INS.COLL.ID’] %>”, whereby this is executed (evaluated) by the Python interpreter. The value of the keys “INS.COLL.NAME” and “INS.COLL.ID” are addressed in the FITS header dictionary, for the FITS file currently being processed.