The ctypes library makes it easily possible to call functions from C libraries, but you do need to be careful with memory access and data types. Python is generally very lenient in memory allocation and type casting; C is, most definitely, not that forgiving.
Even though all platforms will have a standard C library available somewhere, the location and the method of calling it differs per platform. For the purpose of having a simple environment that is easily accessible to most people, I will assume the use of an Ubuntu (virtual) machine. If you don't have a native Ubuntu available, you can easily run it through VirtualBox on Windows, Linux, and OS X.
Since you will often want to run examples on your native system instead, we will first show the basics of loading printf from the standard C library.
One problem of calling C functions from Python is that the default libraries are platform-specific. While the following example will work just fine on Windows systems, it won't run on other platforms:
>>> import ctypes >>> ctypes.cdll <ctypes.LibraryLoader object at 0x...> >>> libc = ctypes.cdll.msvcrt >>> libc <CDLL 'msvcrt', handle ... at ...> >>> libc.printf <_FuncPtr object at 0x...>
Because of these limitations, not all examples can work for every Python version and distribution without requiring manual compilation. The basic premise of calling functions from external libraries functions is to simply access their names as properties of the ctypes import. There is a difference, however; on Windows, the modules will generally be auto-loaded, while on Linux/Unix systems, you will need to load them manually.
Calling standard system libraries from Linux/Unix does require manual loading, but it's nothing too involved luckily. Fetching the printf function from the standard C library is quite simple:
>>> import ctypes >>> ctypes.cdll <ctypes.LibraryLoader object at 0x...> >>> libc = ctypes.cdll.LoadLibrary('libc.so.6') >>> libc <CDLL 'libc.so.6', handle ... at ...> >>> libc.printf <_FuncPtr object at 0x...>
For OS X, explicit loading is also required, but beyond that, it is quite similar to how everything works on regular Linux/Unix systems:
>>> import ctypes >>> libc = ctypes.cdll.LoadLibrary('libc.dylib') >>> libc <CDLL 'libc.dylib', handle ... at 0x...> >>> libc.printf <_FuncPtr object at 0x...>
Besides the way libraries are loaded, there are more differences—unfortunately—but these examples at least give you the standard C library. It allows you to call functions such as printf straight from your C implementation. If, for some reason, you have trouble loading the right library, there is always the ctypes.util.find_library function. As always, I recommend explicit over implicit declarations, but things can be made easier using this function. Let's illustrate a run on an OS X system:
>>> from ctypes import util >>> from ctypes import cdll >>> libc = cdll.LoadLibrary(util.find_library('libc')) >>> libc <CDLL '/usr/lib/libc.dylib', handle ... at 0x...>
Calling a function through
ctypes is nearly as simple as calling native Python functions. The notable difference is the arguments and return statements. These should be converted to native C variables:
>>> spam = ctypes.create_string_buffer(b'spam') >>> ctypes.sizeof(spam) 5 >>> spam.raw b'spamx00' >>> spam.value b'spam' >>> libc.printf(spam) 4 spam>>>
As you can see, to call the printf function you must—and I cannot stress this enough—convert your values from Python to C explicitly. While it might appear to work without this initially, it really doesn't:
>>> libc.printf(123) segmentation fault (core dumped) python3
Note
Remember to use the faulthandler module from Chapter 11, Debugging – Solving the Bugs to debug segfaults.
Another thing to note from the example is that ctypes.sizeof(spam) returns 5 instead of 4. This is caused by the trailing null character, which C strings require. This is visible in the raw property of the C string. Without it, the printf function won't know where the string will end.
To pass along other types (such as integers) towards libc functions, we have to use some conversion as well. In some cases, it is optional:
>>> format_string = ctypes.create_string_buffer(b'Number: %d ') >>> libc.printf(format_string, 123) Number: 123 12 >>> x = ctypes.c_int(123) >>> libc.printf(format_string, x) Number: 123 12
But not in all cases, so it's definitely recommended that you convert your values explicitly in all cases:
>>> format_string = ctypes.create_string_buffer(b'Number: %.3f ') >>> libc.printf(format_string, 123.45) Traceback (most recent call last): File "<stdin>", line 1, in <module> ctypes.ArgumentError: argument 2: <class 'TypeError'>: Don't know how to convert parameter 2 >>> x = ctypes.c_double(123.45) >>> libc.printf(format_string, x) Number: 123.450 16
It's important to note that even though these values are usable as native C types, they are still mutable through the value attribute:
>>> x = ctypes.c_double(123.45) >>> x.value 123.45 >>> x.value = 456 >>> x c_double(456.0)
However, this is not the case if the original object was immutable, which is a very important distinction to make. The create_string_buffer object creates a mutable string object, whereas c_wchar_p, c_char_p, and c_void_p create references to the actual Python string. Since strings are immutable in Python, these values are also immutable. You can still change the value property, but it will only assign a new string. Actually, passing one of these to a C function that mutates the internal value will cause problems.
The only values that should convert to C without any issues are integers, strings, and bytes, but I personally recommend that you always convert all of your values so that you are certain of which type you will get and how to treat it.
We have already seen that we can't just pass along Python values to C, but what if we need more complex objects? That is, not just bare values that are directly translatable to C but complex objects containing multiple values. Luckily, we can easily create (and access) C structures using ctypes:
>>> class Spam(ctypes.Structure): ... _fields_ = [ ... ('spam', ctypes.c_int), ... ('eggs', ctypes.c_double), ... ] ...>>> spam = Spam(123, 456.789) >>> spam.spam 123 >>> spam.eggs 456.789
Within Python, we generally use a list to represent a collection of objects. These are very convenient in that you can easily add and remove values. Within C, the default collection object is the array, which is just a block of memory with a fixed size.
The size of the block in bytes is decided by multiplying the number of items with the size of the type. In the case of a char, this is 8 bits, so if you wish to store 100 chars, you would have 100 * 8 bits = 800 bits = 100 bytes.
This is literally all it is—a block of memory—and the only reference you receive from C is a pointer to the memory address where the block of memory begins. Since the pointer does have a type, char* in this case, C will know how many bytes to jump ahead when trying to access a different item. Effectively, when trying to access item 25 in a char array, you simply need to do array_pointer + 25 * sizeof(char). This has a convenient shortcut: array_pointer[25].
Note that C does not store the number of items in the array, so even though our array has only 100 items, it won't block us from doing array_pointer[1000] and reading other (random) memory.
If you take all of that into account, it is definitely usable, but mistakes are quickly made and C is unforgiving. No warnings, just crashes and strangely behaving code. Beyond that, let's see how easily we can declare an array with ctypes:
>>> TenNumbers = 10 * ctypes.c_double >>> numbers = TenNumbers() >>> numbers[0] 0.0
As you can see, because of the fixed sizes and the requirement of declaring the type before using it, its usage is slightly awkward. However, it does function as you would expect, and the values are initialized to zero by default. Obviously, this can be combined with the previously discussed structures as well:
>>> Spams = 5 * Spam >>> spams = Spams() >>> spams[0].eggs = 123.456 >>> spams <__main__.Spam_Array_5 object at 0x...> >>> spams[0] <__main__.Spam object at 0x...> >>> spams[0].eggs 123.456 >>> spams[0].spam 0
Even though you cannot simply append to these arrays to resize them, they are actually resizable with a few constraints. Firstly, the new array needs to be larger than the original array. Secondly, the size needs to be specified in bytes, not items. To illustrate, we have this example:
>>> TenNumbers = 10 * ctypes.c_double >>> numbers = TenNumbers() >>> ctypes.resize(numbers, 11 * ctypes.sizeof(ctypes.c_double)) >>> ctypes.resize(numbers, 10 * ctypes.sizeof(ctypes.c_double)) >>> ctypes.resize(numbers, 9 * ctypes.sizeof(ctypes.c_double)) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: minimum size is 80 >>> numbers[:5] = range(5) >>> numbers[:] [0.0, 1.0, 2.0, 3.0, 4.0, 0.0, 0.0, 0.0, 0.0, 0.0]
Besides the obvious memory allocation issues and mixing mutable and immutable objects, there is one more strange memory mutability issue:
>>> class Point(ctypes.Structure): ... _fields_ = ('x', ctypes.c_int), ('y', ctypes.c_int) ... >>> class Vertex(ctypes.Structure): ... _fields_ = ('a', Point), ('b', Point), ('c', Point) ... >>> v = Vertex() >>> v.a = Point(0, 1) >>> v.b = Point(2, 3) >>> v.c = Point(4, 5) >>> v.a.x, v.a.y, v.b.x, v.b.y, v.c.x, v.c.y (0, 1, 2, 3, 4, 5) >>> v.a, v.b, v.c = v.b, v.c, v.a >>> v.a.x, v.a.y, v.b.x, v.b.y, v.c.x, v.c.y (2, 3, 4, 5, 2, 3) >>> v.a.x = 123 >>> v.a.x, v.a.y, v.b.x, v.b.y, v.c.x, v.c.y (123, 3, 4, 5, 2, 3)
Why didn't we get 2, 3, 4, 5, 0, 1? The problem is that these objects are copied to a temporary buffer variable. In the meantime, the values of that object are being changed because it contains separate objects internally. After that, the object is transferred back, but the values have already changed, giving the incorrect results.