5. MAL Python Mappings

Document ID:

Revision:

1.1

Last modification:

March 18, 2024

Status:

Released

Repository:

https://gitlab.eso.org/cii/srv/cii-srv

File:

mal_pythonmappings.rst

Project:

ELT CII

Owner:

Marcus Schilling
Document History

Revision

Date

Changed/ reviewed

Section(s)

Modification

0.1

2018-07-25

msekoranja

Created.

1.0

2018-08-10

msekoranja

Minor reviewer changes, released.

1.1

18.03.2024

mschilli

0

Public doc

Confidentiality

This document is classified as Public.

Scope

This document is a manual for the Middleware python bindings of the ELT Core Integration Infrastructure software.

Audience

This document is aimed at Users and Maintainers of the ELT Core Integration Infrastructure software.

Glossary of Terms

AMI

Asynchronous Method Invocation

API

Application Programmers Interface

CII

Core Integration Infrastructure

DDS

Data Distribution Service

GC

Garbage Collector

GIL

Global Interpreter Lock

MAL

Middleware Abstraction Layer

QoS

Quality Of Service

STL

Standard Template Library

URI

Uniform Resource Identifier

ZPB

ZeroMQ and Protocol Buffers

References

  1. ESO, E-ELT Control System Architecture Concepts, ESO-283831_1

  1. Cosylab, MAL ZPB Design Document, CSL-DOC-18-150609, version 1.0

  2. Cosylab, MAL DDS Design Document, CSL-DOC-18-150607, version 1.0

  3. https://pybind11.readthedocs.io/

5.1. Introduction

This document provides information on use of Python MAL mappings interface also referred to as Python MAL API.

Python MAL API builds on MAL C++ API foundation. It provides programming interface that is very similar to the programming interface that is exposed by MAL C++ API.

Python MAL API is compatible with Python 3.5 as provided with ESO E-ELT development environment v2.1.8.

Python applications use Python MAL API and are not aware of the actual MAL mappings details. The CiiFactory interface is the primary means to access MAL functionality.

5.2. Prerequisites

In order to use Python MAL API the following prerequisites must be fulfilled:

  • ESO E-ELT Development Environment v2.1.8 must be properly set up.

  • Directory hierarchy with installed MAL products must be established (commonly referred to as DATAROOT).

  • MAL and MAL COMMON products must be installed under DATAROOT.

  • Additional C++ MAL mappings must be installed (for example MAL DDS) under DATAROOT.

  • Interface modules for specific data types, generated by ICD GENERATOR or created manually must be installed under DATAROOT.

  • Linker path for dynamic loading of libraries must be properly set. On Linux this means that LD_LIBRARY_PATH environment variable should contain reference to the lib64 subdirectory under DATAROOT. In more general terms, linker path must be setup in such a way that shared libraries required by Python modules and MAL mapping in use are within that path.

  • PYTHONPATH environment variable must point to directory lib/python3.5/site-packages directory within DATAROOT hierarchy, for example on Linux:

# when using bash

::

# Define DATAROOT

export DATAROOT=~/INTROOT

::

# Do other stuff …

::

# Define LD_LIBRARY_PATH

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$DATAROOT/lib64

::

# Define PYTHONPATH

export PYTHONPATH=$PYTHONPATH:$DATAROOT/lib/python3.5/site-packages

5.3. Using Python MAL API

This chapter provides usage information for Python MAL API.

Python MAL API is implemented as language binding to C++ MAL mapping therefore it closely follows C++ API in its resemblance and operation.

5.3.1. Importing top level Python module, obtaining CiiFactory reference

Top level Python MAL API module is elt.pymal. It is recommended to be imported with alias import to mal.

Once module is imported, reference to CiiFactory can be obtained with call to mal.CiiFactory.getInstance. This call returns reference to the shared CiiFactory instance.

Alternatively, malCiiFactory.createInstance call creates new instance of the CiiFactory.

The following example provides basic steps:

#!/usr/bin/env python

# import top level Python MAL API module
import elt.pymal as mal

# use mal interface to obtain regerence to shared CiiFactory instance
ciiFactory = mal.CiiFactory.getInstance()

elt.pymal module provides the following interfaces (classes or methods):

  • CiiFactory, CII Factory class interface

  • Exceptions, MalException, TimeoutException, SchemeNotSupportedException, IllegalUriException, IllegalStateExceptions

  • Uri, wrapper for C++ ::elt::mal::Uri class implementation. Not needed by general applications, as methods within Python API that require uri as one of the parameters take a string as input.

  • loadMal, method for loading MAL mappings

  • PsQoSList, python class helper for creating list of Publish-Subscribe QoS options required by Publisher-Subscriber interface.

  • RrQoSList, python helper for creating list of Request-Reply QoS options, required by Request-Reply interface.

5.3.2. Loading and registering MAL mappings

When imported, Python MAL API does not load any MAL mapping by default. Application must load required MAL mappings dynamically, by using loadMal method. loadMal method takes two required arguments:

  • malName, string, name of the MAL mapping. Supported values are:

    • dds, for DDS MAL

    • zpb, for ZPB MAL

    • opcua, for OPCUA MAL

  • malProperties, python dictionary with properties, used to initialize MAL mapping.

loadMal method returns opaque handle to the MAL mapping. In case MAL mapping could not be loaded, exception will be raised.

Opaque handle to the loaded MAL mapping must then be passed to the CiiFactory.registerMal class method as second argument. First argument to this method is name of the URI scheme that this particular MAL mapping covers.

Example that follows, illustrates loading and registering two separate MAL mappings:

# !/usr/bin/env python

# import top level python MAL API module
import elt.pymal as mal

# obtain reference to CiiFactory
ciiFactory = mal.CiiFactory.getInstance()

# Prepare MAL properties. Note that different MAL mappings
# require specific properties to be initialy set. Refer to
# documentation for specific MAL mapping for details.
malProperties = {}

# Dynamically load DDS MAL mapping
ddsmal = mal.loadMal('dds',  malProperties)

# Register DDS MAL for two schemes with CiiFactory
ciiFactory.registerMal('dds.ps', ddsmal)
ciiFactory.registerMal('dds.rr', ddsmal)

# Dynamically load ZPB MAL mapping
zpbmal = mal.loadMal('zpb', malProperties)

# Register ZPB MAL with CiiFactory
ciiFactory.registerMAL('zpb.ps', zpbmal)

5.3.3. Python MAL API relative time parameters (timeouts/durations)

Many API calls require relative time parameters, mostly for timeouts. Relative time parameters are also used within constructors of QoS parameters.

Python MAL API expects instance of datetime.timedelta class when relative time parameter is required. For example:

import datetime
# ...
publishTimeout = datetime.timedelta(milliseconds=500)
publisher.publish(sample, publishTimeout)

5.3.4. Using Publisher-Subscriber interface

Publisher-Subscriber interface relies on generated or handcrafted mapping module that provides Python interface for specific data entity type. Data entity type is typically represented as Python class within mapping module that must be imported. This Python class is a key to obtaining correct Subscriber or Publisher instances from the CiiFactory. Both CiiFactory.getSubscriber and CiiFactory.getPublisher methods expect reference to the data entity class (not instance) as second parameter. Both methods take URI as the first argument, properties dictionary as third and list of QoS parameters as fourth argument. References obtained with these methods should be checked to be valid (not None).

Code to obtain reference to the subscriber generally looks like:

#!/usr/bin/env python
# import top level python MAL API module
import elt.pymal as mal

# Import data entity class. This usually resides in the interface module that
# is either handcrafted or generated.
# In this case TestSample class is our data entity class. Data entity classes
# must provide specific interface to the rest of the Python bindings,
# so they are usually generated automatically.
# Public interface provided by data entity class is specified by data entity
# definition for generator.

from TestSampleIf import TestSample

# obtain reference to CiiFactory
ciiFactory = mal.CiiFactory.getInstance()

# Prepare MAL properties
malProperties = {}

# load and register MAL with CiiFactory
zpbmal = mal.loadMal('zpb', malProperties)
ciiFactory.registerMal('zpb.ps', malProperties)

# prepare subscriber specific MAL properties
subscriberMalProperties = {}

# obtain reference to Subscriber

subscriber = factory.getSubscriber(
 'zpb.ps://localhost:13455/test', TestSample, mal.ps.qos.DEFAULT,
 subscriberMalProperties)
if subscriber is None:
     throw RuntimeError('could not obtain subscriber')

# Read all available samples
for sample in subscriber.read(0):
     # Use interface specific to data entity type, to get values
     print(sample.getSampleId())
# Close the subscriber
subscriber.close()

Code to obtain reference to the publisher is similar:

#!/usr/bin/env python

import datetime
# import top level python MAL API module
import elt.pymal as mal

# Import data entity class. This usually resides in the interface module that
# is either handcrafted or generated.
# In this case TestSample class is our data entity class. Data entity classes
# must provide specific interface to the rest of the Python bindings,
# so they are usually generated automatically.
# Public interface provided by data entity class is specified by data entity
# definition for generator.

from TestSampleIf import TestSample

# obtain reference to CiiFactory
ciiFactory = mal.CiiFactory.getInstance()

# Prepare MAL properties
malProperties = {}

# load and register mAL with CiiFactory
zpbmal = mal.loadMal('zpb', malProperties)
ciiFactory.registerMal('zpb.ps', malProperties)

# prepare publisher specific MAL properties
publisherMalProperties = {}

# obtain reference to Publisher

publisher = factory.getPublisher(
 'zpb.ps://localhost:13455/test', TestSample, mal.ps.qos.DEFAULT,
 publisherMalProperties)
if publisher is None:
     throw RuntimeError('Could not obtain publisher')

# Use the publisher to create new data entity
dataEntity = publisher.createDataEntity()

# Use interface specific to data entity type, to set values
dataEntity.setSampleId(100)
dataEntity.setText('test')
# Publish sample
publisher.publish(dataEntity, datetime.timedelta(seconds=1))
# Close the publisher
publisher.close()

5.3.4.1. Using Subscriber/Publisher as a Context Manager

Subscriber and Publisher can be closed explicitly by calling close method or implicitly when they are destroyed by Python garbage collector. Alternative is to use them as Context Managers. In this case, underlying objects are automatically closed as soon as containing context is destroyed.

See the following example for illustration:

# ...setup...

uri = 'zpb.ps:/localhost:13455/test'
properties = {}
qos = mal.ps.qos.DEFAULT
# Use subscriber as Context Maanager
with ciiFactory.getSubscriber(uri, TestSample, qos, properties) as subscriber:
     samples = subscriber.read(0)
# subscriber was automatically closed at the end of previous context

5.3.4.2. Constructing list of QoS parameters

Construction of Subscriber or Publisher object requires list of QoS parameters. Due to underlying implementation this list cannot be simple Python list.

In case list contains no elements, predefined symbol mal.ps.qos.DEFAULT should be used.

To construct non empty list, special helper class mal.PsQoSList has to be used. Constructor to this class takes either individual QoS parameters or Python list with individual QoS parameters:

import elt.pymal as mal

# ...

# Individual QoS parameters
deadline = mal.ps.qos.Deadline(datetime.timedelta(0, 30))
latency = mal.ps.qos.Latency(datetime.timedelta(0, 20))

# Construct QoS list from individual elements
qosList1 = mal.PsQoSlist(deadline, latency)

# Construct QoS list from Python list
lst = [deadline, latency]
qosList2 = mal.PsQoSlist(lst)

5.3.4.3. Constructing Data Event filters

Data Event filters are related to type of data entity that is to be filtered. Filter must be constructed for specific data entity class (not instance). To construct specific event filter, class methods mal.ps.DataEventFilter.new or mal.ps.DataEventFilter.all must be used with data entity class as argument as in following example:

import elt.pymal as mal

# Import data entity class definition
from TestSampleIf import TestSample

# ...
# Obtain reference to CiiFactory
ciiFactory = mal.CiiFactory.getInstance()
# Load and register MAL implementation to be used
# ...

# Create a subscriber
subscriber = factory.getSubscriber(
 'zpb.ps://localhost:13455/test', TestSample, mal.ps.qos.DEFAULT, {})

# create instance for filtering
filterInstance = subscriber.createDataEntity()

# set parameters in filter instance, use API supported by specific data entity

# Create Data event filter. Default filter performs no filtering.
# Parameter is data entity class for which the filter is created.

eventFilter = mal.ps.DataEventFilter.new(TestSample)

# Set filter instance
eventFilter.setFilterInstance(filterInstance)

# Set filter events, input parameter is Python set of events
eventFilter.setInstanceEvents(set(mal.ps.InstanceEvent.UPDATED,
  mal.ps.InstanceEvent.REMOVED))

# use event filter...

Both methods return default filter for that data entity.

5.3.5. Using Request-Reply interface

As with Publisher-Subscriber, Request-Reply interface also relies on generated or handcrafted binding module that provides Python interface for specific Request-Reply client. Client can be implemented as either synchronous or asynchronous. Request-Reply client is typically represented as Python class within binding module that must be imported. This python class is key to obtaining correct Client instance from the CiiFactory. Method CiiFactory.getClient expects reference to the data Request-Reply client class (not instance) as second parameter. Method takes URI as the first argument, list of QoS parameters as third argument and dictionary of properties as the fourth argument.

The following example illustrates the concept:

#!/usr/bin/env python

# Import top level Python API module
import elt.pymal as mal

# from mapping module implementing specific Request-Reply interface,
# import client class
from MathModule import MathSyncClient

# obtain reference to CiiFactory, load MAL implementaton and register MAL with
# CiiFactory
ciiFactory = mal.CiiFactory.getInstance()
# ...

# Obtain synchronous client for Math RR interface
client = factory.getClient(uri, MathSyncClient, mal.rr.qos.DEFAULT, {})

# issue explicit connect request. This is asynchronous, therefore
# we must wait on future returned from connect() method.

client.connect().get()

if client:
    # issue explicit connect request. This is asynchronous, therefore
    # we must wait on future returned from connect() method.

    client.connect().get()

    # This is synchronous client, client will wait for result to arrive
    sum = client.sum(10.1, 15.10)
    print(sum)
    # once done, close the client
    client.close()

5.3.5.1. Using Request-Reply client as Context Manager

Request-Reply clients also support Context Manager protocol to allow for closing client automatically on context exit.

# setup...
#
#
qos = mal.rr.qos.DEFAULT

with ciiFactory.getClient(uri, MathSyncClient, qos, {}) as client:
     client.connect().get()
     sum = client.sum(10.1, 15.10)

# Client was automatically closed by now

5.3.5.2. Constructing list of QoS parameters

Construction of Client object requires list of QoS parameters. Due to underlying implementation this list cannot be simple Python list.

In case list contains no elements, predefined symbol mal.rr.qos.DEFAULT should be used.

To construct non empty list, special helper class mal.RrQoSList has to be used. Constructor to this class takes either individual QoS parameters or Python list with individual QoS parameters:

import elt.pymal as mal

# ...

# Individual QoS parameters
ct = mal.rr.qos.ConnectionTime(datetime.timedelta(0, 5))
rt = mal.rr.qos.ReplyTime(datetime.timedelta(0, 1))

# Construct QoS list from individual elements
qosList1 = mal.RsQoSlist(ct, rt)

# Construct QoS list from Python list
lst = [ct, rt]
qosList2 = mal.RrQoSlist(lst)