4.3.6. Bindings

Shiboken2 is a Python binding generator. It uses CLang to parse and analyze the structure of the C++ code (both from Qt, and our own widget), and typesystem to create the generated code through instructions.

../_images/qtforpython-underthehood.png

Qt uses Shiboken2 to create its Python bindings, and already provides typesystem files for all its libraries. This is how PySide2 is built.

The mayor issue of Shiboken2, is that it uses CMake3 and a huge CMakeList.txt that is very difficult to understand. There is no other build system supported. In this page, I will explain how Shiboken2 works, and how the CMakeLists.txt file was translated.

4.3.6.1. Input for Bindings

Input files for binding generation in Shiboken2 are simple. Two files are needed:

  • C++ header file, that includes every class we need to bind.

  • Typesetting XML file, that indicates which will be the name of the target Python module, and which classes will be bind into said module.

The header file is just a list of #include sentences, no extra code is needed. Every class that needs binding has to be included in this file, through the use of common #include macro instructions. If two classes are defined in the same file, then only one include is needed.

1#ifndef BINDINGS_H
2#define BINDINGS_H
3#define QT_ANNOTATE_ACCESS_SPECIFIER(a) __attribute__((annotate(#a)))
4#include "QeWidgetExample.h"
5#endif // BINDINGS_H

There are only two important lines, 3 and 4.

  • Line 3 indicates to the C++ preprocessor that it needs to make classifiers used by Qt visible. These classifiers are used in QObjects. (signals:, slots: in the header files).

  • In Line 4, you should include every widget you need. In here, we recommend to use only file named as the resulting target name. This will ensure compabitility with Python module names, allowing both C++ and Python version of the widget to provide correct include/import statements.

The typesystem file is a bit more complex.

 1<?xml version="1.0"?>
 2<typesystem package="QeWidgetExample">
 3    <load-typesystem name="typesystem_core.xml" generate="no"/>
 4    <load-typesystem name="typesystem_core_common.xml" generate="no" />
 5    <load-typesystem name="typesystem_core_x11.xml" generate="no" />
 6    <load-typesystem name="typesystem_widgets.xml" generate="no" />
 7    <load-typesystem name="typesystem_gui_common.xml" generate="no" />
 8    <primitive-type name="double"/>
 9    <object-type name="QeDartBoard">
10        <modify-function signature="paintEvent(QPaintEvent*)">
11            <modify-argument index="1" invalidate-after-use="yes">
12                <rename to="event"/>
13            </modify-argument>
14        </modify-function>
15    </object-type>
16</typesystem>

  • As you see here, the XML defines in line 2 a package called QeWidgetExamlple. This will be the name of the resulting Python package.

  • Lines 3 to 7 indicate the typesystem engine to load more typesystem definitions. These ones are the base set of Qt needed for widgets.

  • Then one primitive is declare in line 7, as it is used in slots and signals in the QeDartBoard class. If any other primitive is used, but it is not in the signature, then no definition for them is needed.

  • In line 9 an object is declared. This object is a C++ class named QeDartBoard. This should be the same class name as in the C++ code declaration. The resulting Python library will have a class in package QeWidgetExample named QeDartBoard.

  • Inside the object-type, several lines (10 to 14), give special instructions to typesetting. This indicates that the function, with signature paintEvent(QPaintEvent*) should be modified:

    • The first (index=1) argument gets invalidated after use, and it is renamed to event.

    • Lets remember that Event is a object generated in Python, but is being passed as argument to a C++ class. Since python keep references counters, we need to avoid leaking memory. This is how it is done in this case.

4.3.6.2. Code generation

In Qt’s examples, CMake is used. I will indicate here how to do this without a build system (and eventually, how to do it in WAF).

The Code generation is roughly this command, which invokes Shiboken2 with a bunch of options, and passes two mandatory arguments, the C++ header file, and the typesystem XML file:

$SHIBOKEN_CMD $SHIBOKEN_OPT ${SRC_DIR}/bindings.h ${SRC_DIR}/bindings.xml

The SHIBOKEN_CMD variable is path to the code generator. We use pyside2_config.py which is part of PySide2 package. It offers information about the installed libraries and includes, so that a build system can use those strings.

First we find out where the Python site-packages are located (pyside2 shiboken2 and shiboken2-generator are installed in this directory), and then we construct the path to Shiboken2 executable:

PYTHON_SP=$(python3 -c 'import site; print(site.getsitepackages()[0])')
SHIBOKEN_CMD="$(python3 $PYTHON_SP/PySide2/examples/utils/pyside2_config.py --shiboken2-generator-path)/shiboken2 "

The options are not supposed to change much, expect for the highlighted section (lines 8, 9, 10 and 11). Each of the -I entries is a path to the include files for the libraries mentioned in your source class to be binded.

  • Line 5 indicates where the includes for the original C++ widgets are.

  • Line 6 indicates where the bindings.h file is located

  • Line 7 $INTROOT/include are there for already installed files.

  • Lines 8 to 11 give access to QtWidgets, QtCore and QtGui classes.

  • Line 12 Also the PySide2/include headers files are needed

  • Lined 13 as well as the C++ STL headers.

 1SHIBOKEN_OPT="--generator-set=shiboken --enable-parent-ctor-heuristic "\
 2"--enable-return-value-heuristic "\
 3"--enable-pyside-extensions --use-isnull-as-nb_nonzero "\
 4--avoid-protected-hack "\
 5"-I${SRC_DIR}/../../widgets/src/include "\
 6"-I${SRC_DIR} "\
 7"-I$INTROOT/include "\
 8"-I/opt/Qt/5.14.0/gcc_64/include "\
 9"-I/opt/Qt/5.14.0/gcc_64/include/QtCore "\
10"-I/opt/Qt/5.14.0/gcc_64/include/QtWidgets "\
11"-I/opt/Qt/5.14.0/gcc_64/include/QtGui "\
12"-I$PYTHON_SP/PySide2/include "\
13"-I/opt/llvm/include/c++/v1/ "\
14"-T$PYTHON_SP/PySide2/typesystems "\
15"-T${SRC_DIR} "\
16"--output-directory=${DEST_DIR}"

The ones in -T entries are special. -T indicates where the typesettings files are located, in case we need to “include” other libraries.

  • Line 14 indicates the location of all typesettings files from Qt. In this case, I do not want to specify how to bind a QWidget, QPen, QBrush classes, so I just indicate that my project uses typesettings files from Qt project.

  • Lined 15 indicate where our typesetting file is located.

Tip

If you need more example, we suggest to look at typesystem files from Qt:

As a result, this will generate a directory in DEST_DIR, that has 2 files for each binded class, and 2 more for each Python module indicated as target.

4.3.6.3. Compilation

The compilation of these are more or less straight C++ objects, that need to be linked together as a shared library.

We use WAF custom script to conduct compilation:

1def configure(cnf):
2  paths = [cnf.path.abspath(), os.path.join(cnf.path.abspath(), "bin")]
3  cnf.find_program('generate', path_list=paths)
4  cnf.env.DEST_DIR = cnf.path.find_or_declare('src').abspath()
5  cnf.env.SRC_DIR =  cnf.path.find_node('src').abspath()

The configure method preparation during “waf configure” several variables and check for us. In this case, we tell WAF to find the generate script, and to declare the DEST_DIR and SRC_DIR variable. One is on the source, and the other in the build directory WAF uses for compilation products.

 1def build(bld):
 2  code_generation = bld(
 3    rule='${GENERATE} ${SRC_DIR} ${DEST_DIR}',
 4    source=['src/bindings.xml', 'src/bindings.h'],
 5    target=[
 6        'src/QeExamplesWidgets/qeexampleswidgets_module_wrapper.cpp',
 7        'src/QeExamplesWidgets/qeexampledmslabel_wrapper.cpp'
 8        ],
 9    name='bindings-generate'
10  )
11
12  compilation = bld(
13    features='pyext cxx cxxshlib wdep',
14    includes=[
15      '/usr/include/shiboken2',
16      '/usr/include/PySide2',
17      '/usr/include/PySide2/QtCore',
18      '/usr/include/PySide2/QtGui',
19      '/usr/include/PySide2/QtWidgets',
20      '/usr/include/python{}'.format(bld.env.PYTHON_VERSION),
21      '/usr/include/qt5/QtCore',
22      '/usr/include/qt5/QtWidgets',
23      '/usr/include/qt5/QtGui',
24      '/usr/include/qt5/QtSvg',
25      '/usr/include/qt5/QtUiTools',
26    ],
27    lib=[':libpython{}.so'.format(bld.env.PYTHON_VERSION),
28         ':libshiboken2.cpython-{}-x86_64-linux-gnu.so'.format(bld.env.PYTHON_VERSION.replace(".","")),
29         ':libpyside2.cpython-{}-x86_64-linux-gnu.so'.format(bld.env.PYTHON_VERSION.replace(".","")),],
30    source=[
31        'src/QeExamplesWidgets/qeexampleswidgets_module_wrapper.cpp',
32        'src/QeExamplesWidgets/qeexampledmslabel_wrapper.cpp'
33        ],
34    target='QeExamplesWidgets',
35    use=['widgets','QT5CORE', 'QT5GUI', 'QT5WIDGETS', 'QT5SVG', 'Qt5Designer', 'Qt5Xml', 'Qt5UiPlugin'],
36    name='bindings',
37    # Fix for ELTDEV-853
38    pytest_path  = [ bld.path.get_bld().abspath() ],
39  )
40  compilation.cxxflags = ["-Wno-pedantic"]

The build method has two tasks, the code_generation executes the file:generate.sh script. It passes as argument the SRC_DIR and DEST_DIR variable from configuration. We use WAF source and target variables to indicate what it should expect as input, and output. This allows WAF to track what needs recompilation in case of changes, and the dependency tree.

The rest of the build method is the compilation task. This is quite large, so lets see its contents:

  • The first argument indicates to WAF which features will use to compile this target. In particular cxxshlib is of interest for us, as this is a shared library.

  • The includes list is expanded from the usual ones to add the shiboken2, PySide2, python and qt libraries header files. These are extra “-I” entries to the C++ compiler.

    • lib argument indicates the library needs to link against python, and shiboken2 libraries. Qt is already on the link list.

    • source argument uses the same target list from the code_generation. The product from the code_generation tstep is used as source in the compilation of the python bindings.

  • target indicates the resulting name of the library. Since it is a shared library, it will be: libQeExamplesWidgets.so

  • use indicates which dependencies are to be added to this task compilation. This means that the listed entries will add lib and includes to the ones specified by us.

    • “widgets” target is the name of the directory in the WAF project structure that this module depends on. Not the name of the target. In case of nested modules, use a point separate notation (module.submodule.subsubmodule).

    • This module depends on “widgets”, as we are producing Python version of widgets defined in “widgets” module.

4.3.6.4. Bindings WAF Module

The bindings module has no WAF support at this point, but this is the expected structure. The product of this module is a C++ shared library that acts as a python module. Its installation target is different, as it is intended to be loaded and imported by python scripts: $INTROOT/lib/python$PYTHON_VERSION/site-packages/.

Warning

This module is here for illustration purposes only. When WAF support is added, we will review this document.

This is how the module would look like with bindings added:

bindings
├── bin
│   └── generate
├── src
│   ├── bindings.h
│   └── bindings.xml
└── wscript