Interfacing Python with C++ using ctypes: classes and arrays
There are many reasons why you may find yourself wanting to call C or C++ code from Python. Perhaps you would like to use a library that doesn’t have a Python interface yet. Or perhaps you would like to compile an expensive algorithm to improve its performance.
As a laboratory technician, I use Python to rapidly build control applications for various instruments. These instruments — cameras, mechanical stages, microscopes, etc. — usually come with an API that’s used to control them. The vendor won’t always provide a Python library that works out-of-the-box; oftentimes these APIs are in the form of C functions exposed by a dynamic-link library.
Whatever the case may be, we’ll need to wrap C/C++ functions in Python code that allows arguments and return types to be shared. Luckily, the ctypes module (included in the Standard Library) makes this simple. Even arrays can be easily passed to and returned from wrapped functions with a little help from NumPy.
In the following demo we’ll compile a dynamic-link library that is subsequently loaded and used by a Python script. There are a few limitations to this approach I’ll lay out before getting started:
It only works on Windows. You can use ctypes in an analogous manner with .so libraries on other platforms, but I’ll be using Visual Studio for this demo.
You can only wrap procedural C functions. You cannot directly expose a C++ class in Python using ctypes — as the name implies, ctypes can only wrap C objects. This isn’t a big deal if you’re writing the C++ yourself: I’ll demonstrate that you can easily provide a C interface for your C++ objects. If you really want to expose classes in Python directly, look into Boost.Python.
Creating a dynamic library
I’ll be using Microsoft Visual Studio 2019 to create the dynamic-link library, which is available for free. Other versions of Visual Studio should work just as well. You should be able to follow this tutorial even if you haven’t used Visual Studio before.
Setting up Visual Studio
First, create a new solution using the dynamic-link library template.
Next, ensure that the DLL being compiled matches the bitness of your operating system and the bitness of your Python environment. The DLL project template should include config files for both x86 and x64 architectures by default. I’ll switch the build configuration to x64 because I am using a 64-bit version of Windows and Python:
I’ve named my project ‘myclib’ — when I build it, a DLL called ‘myclib.dll’ will be created. Where this DLL appears on disk depends on how Visual Studio is configured. It’s important is that we know where it is so we can provide ctypes with a path to it from our Python script.
Simple functions
Add a new .cpp source file to your project using the file menu, the Solution Explorer, or CTRL+SHIFT+A.
In this file we can write functions that we will later access from Python. Let’s start with a very simple add function.
add functionNote that the function is included in an extern “C” block. This instructs the compiler to provide C linkage for the C++ function. Again, ctypes cannot interface with C++ objects.
Note also that the function is proceeded by the __declspec(dllexport) attribute. This will add the function to the dynamic symbol table of the DLL explicitly so that no module-definition file is needed to use it.
The add function will now be available by name once the DLL is loaded using ctypes.
The #include “pch.h” import is required for the template to compile correctly.
Let’s try using our add function in Python —compile the library from the Build menu or by pressing F6.
Using ctypes
Let’s create a simple Python script that loads the library and calls our function.
First, we load our library. This Python script is in the same directory as the DLL, otherwise we would need to pass its path to ctypes.CDLL().
All the functions in the dynamic symbol table are available as methods of the CDLL object in Python. We can access our add function simply via lib.add. Before we can use it, we need to specify the function’s argument types and return type.
ctypes can deal with all the common C types like int, char, float, etc — here, we set lib.add’s argtypes attribute equal to a tuple of two ctypes.c_int, and its restype attribute also equal to ctypes.c_int. This mirrors the function’s prototype. In fact, all you need to know about a function in order to use it is its prototype, making it easy to wrap third-party APIs.
And that’s it. Now we can use the function from Python. Try it out for yourself!
C++ classes
Now for a more complicated example involving classes and arrays.
If your library includes more than just static procedural functions, you’ll need to wrap the object-oriented code in C functions which are exported by the DLL, and then wrap these again using ctypes. I’ll demonstrate this using a dummy class I’ll create called a Sorter.
The Sorter class will keep track of an array of int for the user called array, who can access it using methods getArray and setArray, which take arrays passed by reference. If the user calls sortArray, the std::sort() function will be used to sort the array in place. The user will specify the size of the array when creating a new instance of Sorter.
Here’s the Sorter prototype:
Because our Sorter will manage a dynamically allocated array, we will need to implement a destructor to make sure it gets released when the Sorter is destroyed.
The getArray and setArray methods will use memcpy to copy the contents of array into or out of an array passed in by reference.
Here’s the Sorter definition:
Wrapping a class
Now we need to write C functions that will allow calls from Python to interface with our class. We will export these functions to the dynamic symbol table like we did in the earlier example.
The user will need a way to create a new Sorter or get rid of an old one. Since we can’t instantiate or delete the Sorter directly from Python, we will write a function for our library that creates a Sorter and returns a pointer to it, and another that takes said pointer and deletes it.
We will likewise provide functions that take a Sorter instance in the form of a pointer and invoke one of its methods.
Our wrapper will look like this:
More advanced ctypes
We will need to use more advanced features of ctypes to interface with our Sorter class and its wrapper functions.
We need to be equipped to deal with C pointers to arrays and objects from Python. We’ll define SorterHandle as a char pointer — really it will point to an instance of Sorter, which appears as a struct, but this trick works fine as long as we don’t care about accessing any of Sorter’s fields.
Next, we’ll define c_int_array using NumPy’s ndpointer. It goes without saying that arrays are dealt with very differently in C and in Python, but NumPy provides an interface that makes it easy to pass arrays back and forth.
Now we can specify the argument and return types of our C functions, like before:
Now we’re ready to use Sorter from Python!
Here’s a quick demo (a continuation of the gist above):
Finally, if you want to restore the original, object-oriented interface of Sorter to your user in Python, you can write yet another wrapper:
Wrapping up
It’s especially important to remember how memory is managed by Python versus how it’s managed by the dynamic library. In the Python wrapper above, we actually had to implement an explicit destructor that disposes of the Sorter object: Python isn’t responsible for it, and Python’s garbage collector won’t get rid of it automatically.
You can also run into trouble when using arrays: keeping track of dimensions and types is important to prevent runtime problems that wouldn’t occur with pure Python arrays.
I hope this demo shows that using C/C++ from Python can be really simple, if a little time-consuming. There are a lot of ways to go about extending Python, and while this technique has proven useful to me, it might not be perfect for other applications. If you need more direct interoperability between C and Python, or want to compile existing Python code to improve its performance, the Cython language might be for you. If you have a huge amount of code written in C or another language you need to interface with, look into SWIG rather than churning out endless wrapper code using the technique I described here.
Happy wrapping!
