Overview

The following diagram provides an overview of the CalOb, indicating the various input and output data types involved:

../_images/ifwcalob-overview.png

The elements shown in the diagram above are as follows:

  • CalOb Generator: The CalOb command line tool, “calobGenerator”, which is invoked with the CalOb Configuration as input to generate the Daily Calibration OB (CalOb Generator - Execution - Example).

  • CalOb SDK: The CalOb SDK, implementing all the services for doing the actual processing to obtain the Daily Calibration OB. It is imported by the CalOb Generator, when this is invoked (The CalOb SDK, Classes).

  • Configuration File: The configuration used as input to the CalOb Generator tool, defining the dependencies for the processing. It is implemented in YAML (Configuration File).

  • Calibration Schedule: The Calibration Schedule defines the properties of the various Calibrations to be processed for scheduling during that execution. It is an executable Python script, which is imported and executed by the CalOb SDK by invoking the Python interpreter (Calibration Schedule).

  • External Functions: An External Function is a Python function which is provided for the specific instrument to handle special logic for the given context. They are referenced from within the Calibration Plan (External Functions).

  • Static Calibration OBs: These are OB documents that can be scheduled, based on various conditions, defined in the Calibration Plan, which are based on a fixed set of paremeter values (Static Calibration OBs).

  • Template Calibration OB: These are OB documents that can be scheduled, based on various conditions, defined in the Calibration Plan, which allows for the usage of computer parameter values, in addition to fixed parameter values (Template Calibration OBs).

  • Science Image Files: Science image files (FITS), is the data generated by the instrument during the observations, normally at night-time. In the context of CalOb, it is possible to specify the data set for a given period of time, as basis for the generation of the Daily Calibration Plan. The image files are normally located in the “DATAROOT”.

  • Daily Calibration OB: The main output from CalOb. The Daily Calibration OB is a standard OB file (JSON) which can be scheduled for execution by the ELT Sequencer to generate the required Calibration Data.

  • CalOb Database: Internal database (SQLite), which is used to store information about previous days Calibration Observations scheduled. This is used to determine when to schedule certain Calibration Obervations, which are scheduled periodically. The CalOb DB is typically stored in “$DATAROOT/calob/db/calibration_history.db, this is defined in the Configuration File”.

  • Logs: Logs, generated by CalOb during execution, using the CII Log Service. The Log Properties File is defined in the CalOb Configuration File.

  • Sequencer: The ELT Sequencer tool, which executes the Daily Calibration OB.

Daily Calibration OB Processing

The processing to obtain the Daily Calibration OB can be split into the following steps:

1: Input Stage:

During this step the CalOb Configuration File is loaded, as well as the other documents in the CalOb Document Set (Configuration).

This includes loading the Calob Configuration File, which is the main entry point to access all data relevant for the execution.

The External Functions, the Calibration Library and Calibration Plan Classes (Python), defined in the specified Instrument Calibration Plan, are instantiated (example).

A scan of data files (FITS) to take into account is carried out, properties defined in the Configuration File for this (folder, filename filter/wildcard, period), and all the FITS header information of the FITS files matching, loaded into a dictionary (Python dict).

2: Processing Stage:

During this step, the method in the Instrument Calibration Plan instance to carry out the processing of the list of candidate Calibration Observations is invoked (“CalibrationPlan.init_calibration_plan()”; example).

First the Static Calibration OBs are processed. The processing involves loading the OB document (JSON) containing the predefined parameters (example).

During the processing, CalOb loops over all FITS files that were found, matching the data file search criteria, and invokes all rules an all files, to generate the Calibration OB template instances in the memory representation of the Daily Calibration OB.

As the Calibration Plan is executable code, no explicit parsing/execution of the contents is needed; the “CalibrationPlan.add_static_calib()” is automatically invoked, including the additional methods to define the conditions for scheduling the Calibration Observation.

For each invocation of “add_static_calib()”, the contents of the Static Calibration OB (OBD) is added in a temporary list which will be filtered in the following steps.

The rules to apply for scheduling the Calibration Observation, is based on periodicity of the execution, and matching of keyword values.

The processing of the Template Calibration OBs (OBDX) is done in a similar way, by invoking the method “CalibrationPlan.add_template_calib()” method to do the processing. The same rules functions apply.

The main difference between the Static and the Template Calibration OBs, is that latter may contain expressions for keyword values, which will be resolved by CalOb. The expressions are written in Python code (example - e.g. key “DET.READ.CLKIND”).

3: Filtering Stage:

During the filtering stage, the output OB, contained in memory, is cleaned for redundant Calibrations. In addition the Calibrations are ordered according to the ordering specified in the Calibration Schedule using the values entered for each calibration using set_group() and set_order().

If the history feature is enabled, the information about Calibration Observations scheduled, will be stored in the CalOb DB, where the historic about when the various candidate Calibration Observations have been scheduled, is stored.

4: Export Stage:

During this step, the output Daily Calibration OB is generated.

5: Clean-up Stage:

Final clean-up of temporary files etc.

More details, including examples, about the various data formats contained in the CalOb Configuration Set, is presented in the next chapter (Configuration).