The libraries that we have used so far only showed us how to access a C/C++ library within our Python code. Now we are going to look at the other side of the story—how C/C++ functions/modules within Python are actually written and how modules such as cPickle and cProfile are created.
Before we can actually start with writing and using native C/C++ extensions, we have a few prerequisites. First of all, we need the compiler and Python headers; the instructions in the beginning of this chapter should have taken care of this for us. After that, we need to tell Python what to compile. The setuptools package mostly takes care of this, but we do need to create a setup.py file:
import setuptools
spam = setuptools.Extension('spam', sources=['spam.c'])
setuptools.setup(
name='Spam',
version='1.0',
ext_modules=[spam],
)This tells Python that we have an Extension object named Spam that will be based on spam.c.
Now, let's write a function in C that sums all perfect squares (2*2, 3*3, and so on) up to a given number. The Python code will look like this:
def sum_of_squares(n):
sum = 0
for i in range(n):
if i * i < n:
sum += i * i
else:
break
return sumThe raw C version of this code would look something like this:
long sum_of_squares(long n){
long sum = 0;
/* The actual summing code */
for(int i=0; i<n; i++){
if((i * i) < n){
sum += i * i;
}else{
break;
}
}
return sum;
}And the Python C version looks like this:
#include <Python.h>
static PyObject* spam_sum_of_squares(PyObject *self, PyObject
*args){
/* Declare the variables */
int n;
int sum = 0;
/* Parse the arguments */
if(!PyArg_ParseTuple(args, "i", &n)){
return NULL;
}
/* The actual summing code */
for(int i=0; i<n; i++){
if((i * i) < n){
sum += i * i;
}else{
break;
}
}
/* Return the number but convert it to a Python object first
*/
return PyLong_FromLong(sum);
}
static PyMethodDef spam_methods[] = {
/* Register the function */
{"sum_of_squares", spam_sum_of_squares, METH_VARARGS,
"Sum the perfect squares below n"},
/* Indicate the end of the list */
{NULL, NULL, 0, NULL},
};
static struct PyModuleDef spam_module = {
PyModuleDef_HEAD_INIT,
"spam", /* Module name */
NULL, /* Module documentation */
-1, /* Module state, -1 means global. This parameter is
for sub-interpreters */
spam_methods,
};
/* Initialize the module */
PyMODINIT_FUNC PyInit_spam(void){
return PyModule_Create(&spam_module);
}It looks quite complicated, but it's really not that hard. There is just a lot of overhead in this case because we only have a single function. Generally, you would have several functions, in which case you only need to expand the spam_methods array and create the functions. The next paragraph will explain the code in more detail, but first let's look at how to run our first example. We need to build and install the module:
# python setup.py build install running build running build_ext running install running install_lib running install_egg_info Removing lib/python3.5/site-packages/Spam-1.0-py3.5.egg-info Writing lib/python3.5/site-packages/Spam-1.0-py3.5.egg-info
Now, let's create a little test script to time the difference between the Python version and the C version:
import sys
import spam
import timeit
def sum_of_squares(n):
sum = 0
for i in range(n):
if i * i < n:
sum += i * i
else:
break
return sum
if __name__ == '__main__':
c = int(sys.argv[1])
n = int(sys.argv[2])
print('%d executions with n: %d' % (c, n))
print('C sum of squares: %d took %.3f seconds' % (
spam.sum_of_squares(n),
timeit.timeit('spam.sum_of_squares(n)', number=c,
globals=globals()),
))
print('Python sum of squares: %d took %.3f seconds' % (
sum_of_squares(n),
timeit.timeit('sum_of_squares(n)', number=c,
globals=globals()),
))# python3 test_spam.py 10000 1000000 10000 executions with n: 1000000 C sum of squares: 332833500 took 0.008 seconds Python sum of squares: 332833500 took 1.778 seconds
Perfect! Exactly the same results but more than 200 times faster!
The Python language makes programming so easy that you might forget about the underlying data structures at times; with C, you can't afford to do that. Just take our example from the previous chapter but with different parameters:
# python3 test_spam.py 1000 10000000 1000 executions with n: 10000000 C sum of squares: 1953214233 took 0.002 seconds Python sum of squares: 10543148825 took 0.558 seconds
It's still very fast, but what happened to the numbers? The Python and C versions give different results, 1953214233 versus 10543148825. This is caused by integer overflows in C. Whereas Python numbers can essentially have any size, with C, a regular number has a fixed size. How much you get depends on the type you use (int, long, and so on) and your architecture (32-bit, 64-bit, and so on), but it's definitely something to be careful with. It might be hundreds of times faster in some cases, but that is meaningless if the results are incorrect.
We can increase the size a bit, of course. This makes it better:
static PyObject* spam_sum_of_squares(PyObject *self, PyObject *args){
/* Declare the variables */
unsigned long long int n;
unsigned long long int sum = 0;
/* Parse the arguments */
if(!PyArg_ParseTuple(args, "K", &n)){
return NULL;
}
/* The actual summing code */
for(unsigned long long int i=0; i<n; i++){
if((i * i) < n){
sum += i * i;
}else{
break;
}
}
/* Return the number but convert it to a Python object first */
return PyLong_FromUnsignedLongLong(sum);
}If we test it now, we realize that it works great:
# python3 test_spam.py 1000 100000001000 executions with n: 10000000 C sum of squares: 10543148825 took 0.002 seconds Python sum of squares: 10543148825 took 0.635 seconds
Unless we make the number even larger:
# python3 test_spam.py 1 100000000000000 ~/Dropbox/Mastering Python/code/h14 1 executions with n: 100000000000000 C sum of squares: 1291890006563070912 took 0.006 seconds Python sum of squares: 333333283333335000000 took 2.081 seconds
So how can you fix this? The simple answer is that you can't. The complex answer is that you can if you use a different data type to store your data. The C language by itself doesn't have the "big number support" that Python has. Python supports infinitely large numbers by combining several regular numbers in the actual memory. Within C, there are no commonly available provisions for this, so there is simply no easy way to get this working. But we can check for errors instead:
static unsigned long long int get_number_from_object(int* overflow, PyObject* some_very_large_number){
return PyLong_AsLongLongAndOverflow(sum, overflow);
}Note that this only works for PyObject*, which means it doesn't work for internal C overflows. But you can, of course, just keep the original Python long around and perform operations on that instead. So, you do have big number support in C without too much effort.
We have seen the results from our example, but if you're not familiar with the Python C API, you might be confused as to why the function parameters look the way they do. The basic calculations within spam_sum_of_squares are identical to the regular C sum_of_squares function, but there are a few small differences. Firstly, the type definition for a function using the Python C API should look something like this:
static PyObject* spam_sum_of_squares(PyObject *self, PyObject *args)
This means that the function is static. A function that's static can be called only from the same translation unit within the compiler. This effectively results in a function that cannot be linked from other modules, which allows the compiler to optimize a bit further. Since functions in C are global by default, this can be very useful to prevent collisions. Just to be sure, however, we have prefixed the function name with spam_ to indicate that this function comes from the spam module.
Be careful not to confuse the word static here with the static before a variable. They are completely different beasts. A static variable means that the variable that will exist for the entire runtime of the program instead of the runtime of just the function.
The PyObject type is the basic type for Python data types, which means that all Python objects can be cast to PyObject* (the PyObject pointer). Effectively, it only tells the compiler what kind of properties to expect, which can be used later for type identification and memory management. Instead of direct access to PyObject*, it is generally a better idea to use the available macros, such as Py_TYPE(some_object). Internally, this expands to (((PyObject*)(o))->ob_type), which is why the macro is generally a better idea. Besides being unreadable, a typo can easily happen.
The list of properties is long and depends greatly on the type of object. For those, I would like to refer to the Python documentation:
https://docs.python.org/3/c-api/typeobj.html
The entire Python C API could fill a book of its own, but it is luckily well documented within the Python manual. The usage, on the other hand, might be less obvious.
With regular C and Python, you specify the arguments explicitly, since variable-sized arguments are a bit tricky with C. This is because they need to be parsed separately. PyObject* args is the reference to objects containing the actual values. To parse these, you need to know how many and which type of variables to expect. In the example, we used the PyArg_ParseTuple function, which parses the arguments as positional arguments only, but it is quite easily possible to parse named arguments as well using PyArg_ParseTupleAndKeywords or PyArg_VaParseTupleAndKeywords. The difference between the last two is that the first one uses a variable number of arguments to specify the destination and the latter uses a va_list to set the values to. But first, let's analyze the code from the actual example:
if(!PyArg_ParseTuple(args, "i", &n)){
return NULL;
}We know that args is the object containing the reference to the actual arguments. The "i" is a format string, which in this case will try to parse a single integer. And &n tells the function to store the value at the memory address of the n variable.
The format string is the important part here. Depending on the character, you get a different data type, but there are many; i specifies a regular integer, and s converts your variable to a c-string (actually a char*, which is a null-terminated character array). It should be noted that this function is, luckily, smart enough to take overflows into consideration as well.
Parsing multiple arguments is quite similar; you simply need to add multiple characters to the format string and multiple destination variables:
PyObject* callback;
int n;
/* Parse the arguments */
if(!PyArg_ParseTuple(args, "Oi", &callback, &n)){
return NULL;
}The version with keyword arguments is similar but requires a few more code changes as the list of methods needs to be informed that the function takes keyword arguments. Otherwise, the kwargs parameter would never arrive:
static PyObject* function(
PyObject *self,
PyObject *args,
PyObject *kwargs){
/* Declare the variables */
int sum = 0;
PyObject* callback;
int n;
static char* keywords[] = {"callback", "n", NULL};
/* Parse the arguments */
if(!PyArg_ParseTupleAndKeywords(args, kwargs, "Oi", keywords,
&callback, &n)){
return NULL;
}
Py_RETURN_NONE;
}
static PyMethodDef methods[] = {
/* Register the function with kwargs */
{"function", function, METH_VARARGS | METH_KEYWORDS,
"Some kwargs function"},
/* Indicate the end of the list */
{NULL, NULL, 0, NULL},
};Note that this still supports normal arguments, but keyword arguments are also supported now.
As we saw in the previous example, integer overflows are not something you will generally notice, and unfortunately there's no good cross-platform way to catch them. However, those are actually the easier errors to handle; the worst one is generally memory management. With Python, if you get an error, you will get an exception that you can catch. But with C, you can't really handle it gracefully. Take a division by zero for example:
# python3 -c '1/0' Traceback (most recent call last): File "<string>", line 1, in <module> ZeroDivisionError: division by zero
This is simple enough to catch with try: ... except ZeroDivisionError: .... With C on the other hand, if you get a bad error, it will kill your entire process. But debugging C code is what C compilers have debuggers for, and to find the cause of the error, you can use the faulthandler module discussed in Chapter 11, Debugging – Solving the Bugs. Right now, let's see how we can properly throw errors from C. Let's use the spam module from earlier, but for brevity, we will omit the rest of the C code:
static PyObject* spam_eggs(PyObject *self, PyObject *args){
PyErr_SetString(PyExc_RuntimeError, "Too many eggs!");
return NULL;
}
static PyMethodDef spam_methods[] = {
/* Register the function */
{"eggs", spam_eggs, METH_VARARGS,
"Count the eggs"},
/* Indicate the end of the list */
{NULL, NULL, 0, NULL},
};Here is the execution:
# python3 setup.py clean build install ... # python3 -c 'import spam; spam.eggs()' Traceback (most recent call last): File "<string>", line 1, in <module> RuntimeError: Too many eggs!
The syntax is slightly different—PyErr_SetString instead of raise—but it's the same basic principle, luckily.
We have seen how to call C functions from Python, but now let's try Python from C and back. Instead of using the readily available sum function, we will build one of our own with a callback and handling of any type of iterable. While this sounds simple enough, it does actually require a bit of type meddling as you can only expect PyObject* as arguments. This is contrary to the simple types, such as integers, chars, and strings, which are immediately converted to the native Python version:
static PyObject* spam_sum(PyObject* self, PyObject* args){
/* Declare all variables, note that the values for sum and
* callback are defaults in the case these arguments are not
* specified */
long long int sum = 0;
int overflow = 0;
PyObject* iterator;
PyObject* iterable;
PyObject* callback = NULL;
PyObject* value;
PyObject* item;
/* Now we parse a PyObject* followed by, optionally
* (the | character), a PyObject* and a long long int */
if(!PyArg_ParseTuple(args, "O|OL", &iterable, &callback,
&sum)){
return NULL;
}
/* See if we can create an iterator from the iterable. This is
* effectively the same as doing iter(iterable) in Python */
iterator = PyObject_GetIter(iterable);
if(iterator == NULL){
PyErr_SetString(PyExc_TypeError,
"Argument is not iterable");
return NULL;
}
/* Check if the callback exists or wasn't specified. If it was
* specified check whether it's callable or not */
if(callback != NULL && !PyCallable_Check(callback)){
PyErr_SetString(PyExc_TypeError,
"Callback is not callable");
return NULL;
}
/* Loop through all items of the iterable */
while((item = PyIter_Next(iterator))){
/* If we have a callback available, call it. Otherwise
* just return the item as the value */
if(callback == NULL){
value = item;
}else{
value = PyObject_CallFunction(callback, "O", item);
}
/* Add the value to sum and check for overflows */
sum += PyLong_AsLongLongAndOverflow(value, &overflow);
if(overflow > 0){
PyErr_SetString(PyExc_RuntimeError,
"Integer overflow");
return NULL;
}else if(overflow < 0){
PyErr_SetString(PyExc_RuntimeError,
"Integer underflow");
return NULL;
}
/* If we were indeed using the callback, decrease the
* reference count to the value because it is a separate
* object now */
if(callback != NULL){
Py_DECREF(value);
}
Py_DECREF(item);
}
Py_DECREF(iterator);
return PyLong_FromLongLong(sum);
}Make sure you note the PyDECREF calls, which ensure that you don't leak these objects. Without them, the objects will stay in use and the Python interpreter won't be able to clear them.
This function is callable in three different ways:
>>> import spam >>> x = range(10) >>> spam.sum(x) 45 >>> spam.sum(x, lambda y: y + 5) 95 >>> spam.sum(x, lambda y: y + 5, 5) 100
Another important issue is that even though we catch overflow errors when converting to long long int, this code is still not safe. If we sum even two very large numbers (close to the long long int limit), we will still have an overflow:
>>> import spam >>> n = (2 ** 63) - 1 >>> x = n, >>> spam.sum(x) 9223372036854775807 >>> x = n, n >>> spam.sum(x) -2