Advanced Features¶
Creating complex data products¶
In the previous example A Complete Recipe the
utility functions of the cpl.dfs module were used to create a ESO
standards compliant image data product. While there are also utilities to
create image list and table data products, it may happen that these utilities
are not enough. For instance there is no utility function available which
would create a data product which has several extensions (e.g. data, variance
and quality information). How such products can still, easily be created is
shown in this example.
The example is again the basic_science recipe example. However here, the way
the product file is created has changed. Instead of creating a FITS product file
where the image data is stored in the FITS primary data unit, a FITS file with
an ESO standard compliant primary FITS header and an empty FITS primary data unit
is created first. The result image is then added to this product FITS file as an
image extension:
from typing import Any, Dict
from cpl import core
from cpl import ui
from cpl import dfs
from cpl.core import Msg
class ScienceDataProcessor(ui.PyRecipe):
# Fill in recipe information
_name = "basic_science_extensions"
_version = "1.0"
_author = "U.N. Owen"
_email = "unowen@somewhere.net"
_copyright = "GPL-3.0-or-later"
_synopsis = "Basic science image data processing writing products with extensions"
_description = (
"The recipe combines all science input files in the input set-of-frames using\n"
+ "the given method. For each input science image the master bias is subtracted,\n"
+ "and it is divided by the master flat. When creating the product the result\n"
+ "image is stored in an image extension demonstrating the assembly of complex\n"
+ "product files "
)
def __init__(self) -> None:
super().__init__()
# The recipe will have a single enumeration type parameter, which allows the
# user to select the frame combination method.
self.parameters = ui.ParameterList(
(
ui.ParameterEnum(
name="basic_science.stacking.method",
context="basic_science",
description="Name of the method used to combine the input images",
default="add",
alternatives=("add", "average", "median"),
),
)
)
def run(self, frameset: ui.FrameSet, settings: Dict[str, Any]) -> ui.FrameSet:
# Update the recipe paramters with the values requested by the user through the
# settings argument
for key, value in settings.items():
try:
self.parameters[key].value = value
except KeyError:
Msg.warning(
self.name,
f"Settings includes {key}:{value} but {self} has no parameter named {key}.",
)
raw_frames = ui.FrameSet()
product_frames = ui.FrameSet()
bias_frame = None
flat_frame = None
output_file = "OBJECT_REDUCED.fits"
# Go through the list of input frames, check the tag and act accordingly
for frame in frameset:
if frame.tag == "OBJECT":
frame.group = ui.Frame.FrameGroup.RAW
raw_frames.append(frame)
Msg.debug(self.name, f"Got raw frame: {frame.file}.")
elif frame.tag == "MASTER_BIAS":
frame.group = ui.Frame.FrameGroup.CALIB
bias_frame = frame
Msg.debug(self.name, f"Got bias frame: {frame.file}.")
elif frame.tag == "MASTER_FLAT":
frame.group = ui.Frame.FrameGroup.CALIB
flat_frame = frame
Msg.debug(self.name, f"Got flat field frame: {frame.file}.")
else:
Msg.warning(
self.name,
f"Got frame {frame.file!r} with unexpected tag {frame.tag!r}, ignoring.",
)
# For demonstration purposes we raise an exception here. Real world
# recipes should rather print a message (also to have it in the log file)
# and exit gracefully.
if len(raw_frames) == 0:
raise core.DataNotFoundError("No raw frames in frameset.")
# By default images are loaded as Python float data. Raw image
# data which is usually represented as 2-byte integer data in a
# FITS file is converted on the fly when an image is loaded from
# a file. It is however also possible to load images without
# performing this conversion.
bias_image = None
if bias_frame:
bias_image = core.Image.load(bias_frame.file)
Msg.info(self.name, f"Loaded bias frame {bias_frame.file!r}.")
else:
raise core.DataNotFoundError("No bias frame in frameset.")
flat_image = None
if flat_frame:
flat_image = core.Image.load(flat_frame.file)
Msg.info(self.name, f"Loaded flat frame {flat_frame.file!r}.")
else:
raise core.DataNotFoundError("No flat frame in frameset.")
# Flat field preparation: subtract bias and normalize it to median 1
Msg.info(self.name, "Preparing flat field")
flat_image.subtract(bias_image)
median = flat_image.get_median()
flat_image.divide_scalar(median)
header = None
processed_images = core.ImageList()
for idx, frame in enumerate(raw_frames):
Msg.info(self.name, f"Processing {frame.file!r}...")
if idx == 0:
header = core.PropertyList.load(frame.file, 0)
Msg.debug(self.name, "Loading image.")
raw_image = core.Image.load(frame.file)
Msg.debug(self.name, "Bias subtracting...")
raw_image.subtract(bias_image)
Msg.debug(self.name, "Flat fielding...")
raw_image.divide(flat_image)
# Insert the processed image in an image list. Of course
# there is also an append() method available.
processed_images.insert(idx, raw_image)
# Combine the images in the image list using the image stacking
# option requested by the user.
method = self.parameters["basic_science.stacking.method"].value
Msg.info(self.name, f"Combining images using method {method!r}")
combined_image = None
if method == "add":
for idx, image in enumerate(processed_images):
if idx == 0:
combined_image = image
else:
combined_image.add(image)
elif method == "average":
combined_image = processed_images.collapse_create()
elif method == "median":
combined_image = processed_images.collapse_median_create()
else:
Msg.error(
self.name,
f"Got unknown stacking method {method!r}. Stopping right here!",
)
# Since we did not create a product we need to return an empty
# ui.FrameSet object. The result frameset product_frames will do,
# it is still empty here!
return product_frames
# Create property list specifying the product tag of the processed image
product_properties = core.PropertyList()
product_properties.append(
core.Property("ESO PRO CATG", core.Type.STRING, r"OBJECT_REDUCED")
)
# Save the result image as a FITS file with an empty primary FITS
# data unit and an image extension. The primary FITS header is
# written using the DFS utility functions, which will make sure that
# the product file will be compliant with the ESO data product
# standards. The extension header and data unit is then attached
# using the basic cpl.core.Image file saving routines.
Msg.info(self.name, f"Saving product file as {output_file!r}.")
dfs.save_propertylist(
frameset,
self.parameters,
frameset,
self.name,
product_properties,
f"demo/{self.version!r}",
output_file,
header=header,
)
# Extend the now existing product file by adding the image extension to
# the end of the file. Since the extension header is just a normal FITS
# extension header, writing the image extension does not need any special
# treatment compared to the pirmary FITS header. Thus the basic cpl.core.Image
# saving function is be used.
# In this example the extension header will not contain any additional
# keywords. I contains only what is needed to be a valid FITS image extension.
# This can be achieved by passing an empty property list to the image saving
# function.
combined_image.save(output_file, core.PropertyList(), core.io.EXTEND)
# Register the created product
product_frames.append(
ui.Frame(
file=output_file,
tag="OBJECT_REDUCED",
group=ui.Frame.FrameGroup.PRODUCT,
level=ui.Frame.FrameLevel.FINAL,
frameType=ui.Frame.FrameType.IMAGE,
)
)
return product_frames
The primary FITS header, is written to a file using the utility functions from
the cpl.dfs module to make sure that the primary FITS header of the output
file meets the formal requirements of a standard ESO data product. Because an
extension header is just a normal FITS header, the basic PyCPL saving functions
can be used to add extensions to the product file.
Note that an extension header should still follow the general ESO guidelines for
data products or may need to have additional keywords to be compliant with the
ESO Science Data Producs standard. For instance the extension header should
have keywords like EXTNAME added, or other keywords characterizing and
describing the data in this extension. This has been skipped here deliberately
to illustrate the situation where no additional information should be passed,
i.e. how an empty header (except for the information needed for a valid FITS
image extension) can be created.
Error handling¶
Errors from PyCPL¶
In Python errors which occur in the code at runtime are reported through exceptions, which can be raised, catched and handled, or re-raised. In order to fit within this environment returning error codes, which is the way C libraries usually deal with errors, is not the best solution.
For this reason PyCPL provides custom exceptions which seamlessly integrate with the standard Python exceptions. These custom PyCPL exceptions are also representing the errors of the CPL C library which are translated into exceptions. Because these translated errors have become Python exceptions, they can be used in exactly the same way as any other Python exception, i.e. they can be raised, caught, handled and re-raised.
Python code using PyCPL exceptions¶
Python code can raise any of the custom PyCPL exceptions. This is illustrated in the following.
The recipe expects to find input frames of type OBJECT in the input
set-of-frames file. Therefore feeding basic_science with inappropriate
input data, for instance the BIAS frames of the gimasterbias recipe,
will raise a custom PyCPL exception, cpl.core.DataNotFoundError.
Because the exceptions is not handled the recipe will terminate showing the familiar Python traceback output.
>>> p = Pyesorex()
[ INFO ] pyesorex: This is PyEsoRex, version 1.0.0.
>>> p.recipe = "basic_science"
[ INFO ] pyesorex: Loaded recipe 'basic_science'.
>>> p.sof_location = "bias.sof"
>>> p.run()
[ INFO ] pyesorex: Running recipe 'basic_science'...
[WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:47:34.623.fits' with unexpected tag 'BIAS', ignoring.
[WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:52:45.527.fits' with unexpected tag 'BIAS', ignoring.
[WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T09:57:55.430.fits' with unexpected tag 'BIAS', ignoring.
[WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T10:03:05.335.fits' with unexpected tag 'BIAS', ignoring.
[WARNING] basic_science: Got frame '/home/dummy/test/bias/GIRAF.2019-06-25T10:08:15.239.fits' with unexpected tag 'BIAS', ignoring.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/dummy/VirtualEnvs/pycpl/lib/python3.12/site-packages/pyesorex/pyesorex.py", line 1231, in run
result = self.recipe.run(self.sof, self.recipe_parameters.as_dict())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/dummy/test/pyrecipes/pycpl_recipe_basic_science.py", line 71, in run
raise core.DataNotFoundError("No raw frames in frameset.")
cpl.core.DataNotFoundError: Most recent error last:
File "/home/dummy/test/pyrecipes/pycpl_recipe_basic_science.py", line 71, in run
No raw frames in frameset.
Errors from CPL¶
When calling a PyCPL function results in an error in the underlying CPL function, the error from the CPL library is translated into a Python exception. It can be used like any other exception in Python.
For example, trying to create a PyCPL cpl.core.Mask object with illegal,
negative dimensions, results in an cpl.core.IllegalInputError:
>>> from cpl.core import Mask, IllegalInputError, Msg
>>> mask = Mask(2, 2)
>>> print(mask)
#----- mask: 1 <= x <= 2, 1 <= y <= 2 -----
X Y value
1 1 0
1 2 0
2 1 0
2 2 0
>>> mask - Mask(2, -2)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<string>", line 87, in __new__
cpl.core.IllegalInputError: Most recent error last:
File "../../../cplcore/cpl_mask.c", line 798, in cpl_mask_new
Illegal input
Compared to the previous example where a PyCPL exception was raised directly from Python code, the exception here is actually triggered by an error reported by the underlying CPL function. This is visible in the resulting Python traceback, which shows the CPL source file, line number and function where the problem was detected.
Because CPL errors become ordinary Python exceptions they can be used like those, i.e. they can be raised, caught, handled or re-raised:
>>> def mask_maker(*args):
... try:
... mask = Mask(*args)
... except IllegalInputError as err:
... Msg.error("mask_maker", f"Tried to create Mask with illegal dimensions: {args!r}")
... raise err
... return mask
...
>>> Msg.set_config(show_domain=False, show_component=True)
>>> mask_maker(1, 3)
#----- mask: 1 <= x <= 1, 1 <= y <= 3 -----
X Y value
1 1 0
1 2 0
1 3 0
>>> mask_maker(0, 3)
[ ERROR ] mask_maker: Tried to create Mask with illegal dimensions: (0, 3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in mask_maker
File "<stdin>", line 3, in mask_maker
File "<string>", line 87, in __new__
cpl.core.IllegalInputError: Most recent error last:
File "../../../cplcore/cpl_mask.c", line 797, in cpl_mask_new
Illegal input
Errors occurring in native CPL Recipes¶
When an application runs a native CPL recipe as a plug-in and an error occurs inside the CPL recipe causing it to terminate for instance with a segmentation fault it will also terminate the the calling application because the plug-in runs as part of this process.
This is not a desirable behaviour if this happens to a user’s Python session while processing data using ESO instrument pipelines and PyCPL. In order to prevent a user’s Python session being torn down by a faulty native CPL recipe, PyCPL executes native CPL recipes in a separate child process. A failing CPL recipe will therefore only affect the child process, but not the Python interpreter that was used to call the recipe. Within the Python interpreter a forcibly terminated CPL recipe is therefore simply reported as an error.
Running Recipes from Python using Low-Level PyCPL Interfaces¶
TBD
Debugging Native CPL Recipes when using PyCPL¶
TBD
