Python support¶

Overview¶

Data Task components are envisioned to support Python to simplify the development of complicated algorithms that are being used to support the adaptive optics loop. Some of these algorithms are very large and complicated with very tight temporal requirements. Others may be complicated but have not so tight temporal requirements. For these later calculations, Python can be very powerful and allow the computation to take advantage of the cleaner syntax of Python with many mature numerical libraries.

The method we have chosen to investigate is using pybind11 (see section Using Provided Python Bindings).

Source Code Location¶

The example source code can be found in the following sub-directory of the rtctk project:

_examples/exampleDataTask/optimiser

Modules¶

The provided example is composed into the following waf modules:

app - The data task application including default configuration.
pyLib - Python library containing calculation.
scripts - Helper scripts for deployment and control.

In addition the example uses the following component framework libraries:

rtctkRtcComponent - For RtcComponent functionality
rtctkServicesPython - Framework provided Python bindings for C++

Running the Example¶

The optimiser example can be run in isolation or as part of an SRTC system. The Python computation can be invoked after bringing the component to state Operational. By sending the Optimise command with a specific JSON payload the Python interpreter is started and the computation is carried out.

The script rtctkExampleDataTaskOptimiser.sh is provided to bring the example data task and the supporting infrastructure online in a simple way.

rtctkExampleDataTaskOptimiser.sh¶

After installing the RTC Tk, run the example using the following sequence of commands:

# Deploy and start the example applications
rtctkExampleDataTaskOptimiser.sh deploy
rtctkExampleDataTaskOptimiser.sh start

# Use the client to step through the life-cycle of the respective component instance
rtctkExampleDataTaskOptimiser.sh send Init
rtctkExampleDataTaskOptimiser.sh send Enable
rtctkExampleDataTaskOptimiser.sh send Optimise '{"algorithm":"PythonInversion"}'
rtctkExampleDataTaskOptimiser.sh send Disable
rtctkExampleDataTaskOptimiser.sh send Reset

# Gracefully terminate the applications and clean-up
rtctkExampleDataTaskOptimiser.sh stop
rtctkExampleDataTaskOptimiser.sh undeploy

Note

The commands indicated above, e.g. when using deploy, may generate the following output:

rtctkExampleDataTaskTelemetry: no process found
rtctkExampleShmPub: no process found

This is expected and should not be treated as an indication of failure.

Using Provided Python Bindings¶

PyBind11 is a powerful tool to link Python and C++ together. It can be used either to embed C++ in Python or to embed an Python interpreter in C++.

The RTC Toolkit library rtctkServicesPython provides functionality for passing vectors and matrices between C++ and Python code without internal data being copied. To use the framework-provided Python bindings link against this library and include the headers shown below.

#include <rtctk/componentFramework/matrixBufferPy.hpp>
#include <rtctk/componentFramework/vectorPy.hpp>

Note

Currently we only support no copy passing of data in Vectors and MatrixBuffers formats. (see Runtime Configuration Repository).

Note

The version of pybind11 used in the ELT dev env does not support booleans.

Before Python can be used in a Data Task application it needs to be set up accordingly. This setup can either be done once in the constructor of the computation class or on demand before the respective Python computation starts.

py::scoped_interpreter py_guard;
auto py_matrix_buffer_import = LoadMatrixBufferPyBinds();
auto py_compute_module = py::module::import("rtctkExampleDataTaskPyLib");
auto py_inversion = py_compute_module.attr("inversion");

In the first line a new Python interpreter is started. Then the bindings for the MatrixBuffer class are loaded. In the last line a handle to the actual computation is retrieved.

Note

To allow for easy swapping of Python modules the names of the modules and functions can also be retrieved from configuration. This may provided additional flexibility and may eliminate the need for recompilation.

To run the Python computation the previously retrieved function handle is invoked with references to the data buffers of the input and output data as the provided arguments.

py_inversion(&m_IM, &m_CM);

Limitations and Known Issues¶

This section has been created to document the first attempt to implement a Data Task with Python support. The section will be expanded as the use of pybind11 is further investigated and as additional Python support libraries are being added to the toolkit.