20.1. Introduction/Motivation

20.1.1. What Are Extensions?

In general, any code that you write that can be integrated or imported into another Python script can be considered an “extension.” This new code can be written in pure Python or in a compiled language like C and C++ (or Java for JPython). However, a more “strict” definition of an extension is relegated to the latter category, the topic of this chapter.

One great feature of Python is that its extensions interact with the interpreter in exactly the same way as the regular Python modules. Python was designed so that the abstraction of module import hides the underlying implementation details from the code which uses such extensions. Unless the client programmer searches the file system, he or she simply cannot tell whether a module is written in Python or in a compiled language.

20.1.2. Why Extend Python?

Throughout the brief history of software engineering, programming languages have always been taken at face value. What you see is what you get; it was impossible to add new functionality to an existing language. In today's programming environment however, the ability to customize one's programming environment is now a desired feature; it also promotes code reuse. Languages such as TCL and Python are among the first languages to provide the ability to extend the base language. So why would you want to extend a language like Python which is already feature-rich? There are several good reasons:

• Added/extra (non-Python) functionality

  One reason for extending Python is the need to have new functionality not provided by the core part of the language. This can be accomplished in either pure Python or as a compiled extension, but there are certain things such as creating new data types or embedding Python in an existing application.

• Bottleneck performance improvement

  It is well-known that interpreted languages do not perform as fast as compiled languages due to the fact that translation must happen on-the-fly and during runtime. In general, moving a body of code into an extension will improve overall performance. The problem is that it is sometimes not advantageous if the cost is high in terms of resources.

  Percentage-wise, it is a wiser bet to do some simple profiling of the code to identify what the bottlenecks are, and move those pieces of code out to an extension. The gain can be seen more quickly and without expending as much in terms of resources.

• Keep proprietary source code private

  Another important reason to create extensions is due to one side effect of having a scripting language. For all the ease-of-use such languages bring to the table, there really is no privacy as far as source code is concerned because the executable is the source code.

  Code that is moved out of Python and into a compiled language helps keep proprietary code private because you ship a binary object. Because these objects are compiled, they are not as readily able to be reverse-engineered; thus, the source remains more private. This is key when it involves special algorithms, encryption or software security, etc.

  Another alternative to keeping code private is to only ship pre-compiled .pyc files only. It serves as a good middle ground between releasing the actual source (.py files) and having to migrate that code to extensions.

20.2. Extending Python by Writing Extensions

Creating extensions for Python involve three main steps:

In this section, we will break out all three pieces and expose it all to you.

20.2.1. Create Your Application Code

First, before any code becomes an extension, create a standalone “library.” In other words, create your code keeping in mind that it is going to turn into a Python module. Design your functions and objects with the vision that Python code will be communicating and sharing data with your C code and vice versa.

Next, create test code to bulletproof your software. You may even use the “Pythonic” development method of designating your main() function in C as the testing application so that if your code is compiled, linked, and loaded into an executable (as opposed to just a shared object), that invocation of such an executable will result in a regression test of your software library. For our extension example below, this is exactly what we do.

The test case involves two C functions which we want to bring to the world of Python programming. The first is the recursive factorial function, fac(). The second, reverse(), is a simple string reverse algorithm, whose main purpose is to reverse a string “in place,” that is, to return a string whose characters are all reversed from their original positions, all without allocating a separate string to copy in reverse order. Because this involves the use of pointers, we need to carefully design and debug our code before bringing Python into the picture.

Our first version, Extest1.c, is presented in Example 20.1.

Listing 20.1. Pure C Version of Library(Extest1.c)

1  #include <stdio.h>
2  #include <stdlib.h>
3  #include <string.h>
4
5  int fac(int n)
6  {
7      if (n < 2) return(1);
8      return((n)*fac(n-1));
9  }
10
11 char *reverse(char *s)
12 {
13    register char t,
14            *p = s,
15            *q = (s + (strlen(s) - 1));
16
17    while (s && (p < q))
18    {
19        t = *p;
20        *p++ = *q;
21        *q-- = t;
22    }
23    return s;
24 }
25
26 void main()
27 {
28    char s[BUFSIZ];
29    printf("4! == %d
", fac(4));
30    printf("8! == %d
", fac(8));
31    printf("12! == %d
", fac(12));
32    strcpy(s, "abcdef");
33    printf("reversing 'abcdef', we get '%s'
", 
34        reverse(s));
35    strcpy(s, "madam");
36    printf("reversing 'madam', we get '%s'
", 
37        reverse(s));
38 }

This code consists of a pair of functions, fac() and reverse(), which are implementations of the functionality we described above. fac() takes a single integer argument and recursively calculates the result, which is eventually returned to the caller once it exits the outermost call.

The last piece of code is the required main() function. We use it to be our tester, sending various arguments to fac() and reverse(). With this function, we can actual tell whether our code works (or not).

Now we should compile the code. For many versions of Unix with the gcc compiler, we use the following command:

% gcc Extest1.c -o Extest
%

To run our program, we issue the following command and get the output:

% Extest
4! == 24
8! == 40320
12! == 479001600
reversing 'abcdef', we get 'fedcba'
reversing 'madam', we get 'madam'
%

We stress again that you should try to complete your code as much as possible, because you do not want to mix debugging of your library with potential bugs when integrating with Python. In other words, keep the debugging of your core code separate from the debugging of the integration. The closer you write your code to Python interfaces, the sooner your code will be integrated and work correctly.

Each of our functions takes a single value and returns a single value. It's pretty cut and dried, so there shouldn't be a problem integrating with Python. Note that so far, there no connection or relationship with Python as of now. We are simply creating a standard C or C++ application.

20.2.2. Wrap Your Code in Boilerplate

The entire implementation of an extension primarily revolves around the “wrapping” concept which we introduced earlier in Section 13.15.1. You should design your code in such a way that there is a smooth transition between the world of Python and your implementing language. This interfacing code is commonly called “boilerplate” code because it is a necessity if your code is to talk to the Python interpreter.

There are 4 main pieces to the boilerplate software:

Include Python header file

The first thing you should do is to find out where your Python include files are and make sure your compiler has access to that directory, which is usually /usr/local/include/python1.x, or /usr/include/python1.x, where the “1.x” is your version of Python. (It is probably 1.5 or 2.0.) If you compiled and installed your Python interpreter, then you shouldn't have a problem because the system generally knows where your files are installed.

Add the inclusion of the Python.h header file to your source. The line will look something like:

#include "Python.h"

That's the easy part. Now you have to add the rest of the boilerplate software.

Add PyObject* Module_func() Python wrappers for each function

This part is the trickiest. For each function which you want accessible to the Python environment, you will create a static PyObject* function with the module name (along with an underscore [ _ ]) prepended to it.

For example, we want fac() to be one of the functions available for import from Python and will use Extest as the name of our final module, so we create a “wrapper” called Extest_fac(). So in the client Python script, there will be an “import Extest” and an “Extest.fac()” call somewhere (or just “fac()” for “from Extest import fac”).

The job of the wrapper is to take Python values, convert them to C, then make a call to the appropriate function with what we want. When our function has completed, and it is time to return to the world of Python, it is also the job of this wrapper to take whatever return values we designate, convert them to Python, and then perform the return, passing back any values as necessary.

In the case of fac(), when the client program invokes Extest.fac(), our wrapper will be called. We will accept a Python integer, convert it to a C integer, call our C function fac() and obtain another integer result. We then have to take that return value, convert it back to a Python integer, then return from the call. (In your head, try to keep in mind that you are writing the code that will proxy for a “def fac(n)” declaration. When you are returning, it is as if that imaginary Python fac() function is completing.)

So, you're asking, how does this conversion take place? The answer is with the PyArg_Parse*() functions when going from C to Python, and Py_BuildValue() when returning from C to Python.

The PyArg_Parse*() functions are similar to the C sscanf() function. It takes a stream of bytes, and, according to some format string, parcels them off to corresponding container variables, which, as expected, take pointer addresses. They both return 1 on successful parsing and 0 otherwise.

Py_BuildValue() works like sprintf(), taking a format string and converting all arguments to a single returned object containing those values in the formats that you requested.

You will find a summary of these functions below in Table 20.1:

A set of conversion codes is used to convert data objects between C and Python; they are given in Table 20.2.

These conversion codes are the ones given in the respective format strings that dictate how the values should be converted when moving between both languages. NOTE: the conversion types are different for Java since all data types are classes. Consult the JPython documentation to obtain the corresponding Java types for Python objects.

Here we show you our completed Extest_fac() wrapper function:

								static PyObject *
Extest_fac(PyObject *self, PyObject *args) {

     int res;              // parse result
     int num;              // arg for fac()
     PyObject* retval;     // return value

     res = PyArg_ParseTuple(args, "i", &num);
     if (!res)             // TypeError
         return NULL;
     }
     res = fac(num);
     retval = (PyObject*)Py_BuildValue("i", res);
     return retval;
}

The first step is to parse the data received from Python. It should be a regular integer, so we use the “i” conversion code to indicate as such. If the value was indeed an integer, then it gets stored in the num variable. Otherwise, PyArg_ParseTuple() will return a NULL, in which case we also return one. In our case, it will generate a TypeError exception that tells the client user that we are expecting an integer.

We then call fac() with the value stored in num and put the result in res, reusing that variable. Now we build our return object, a Python integer, again using a conversion code of “i”. Py_BuildValue() creates an integer Python object which we then return. That's all there is to it!

In fact, once you have created wrapper after wrapper, you tend to shorten your code somewhat to avoid extraneous use of variables. Try to keep your code legible, though. We take our Extest_fac() function and reduce it to its smaller version given here, using only one variable, num:

								static PyObject *
Extest_fac(PyObject *self, PyObject *args) {
    int num;
    if (!PyArg_ParseTuple(args, "i", &num)) return NULL;
        return (PyObject*)Py_BuildValue("i", fac(num));
}

What about reverse()? Well, since you already know how to return a single value, we are going to change our reverse() example somewhat, returning two values instead of one. We will return a pair of strings as a tuple, the first element being the string as passed in to us, and the second being the newly-reversed string.

To show you that there is some flexibility, we will call this function Extest.doppel() to indicate that its behavior differs from reverse(). Wrapping our code into an Extest_doppel() function, we get:

								static PyObject *
Extest_doppel(PyObject *self, PyObject *args) {
    char *orig_str;
    if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL;
    return (PyObject*)Py_BuildValue("ss", orig_str, 
        reverse(strdup(orig_str)));
}

As in Extest_fac(), we take a single input value, this time a string, and store it into orig_str. Notice that we use the “s” conversion code now. We then call strdup() to create a copy of the string. (Since we want to return the original one as well, we need a string to reverse, so the best candidate is just a copy of the string.) strdup() creates and returns a copy which we immediate dispatch to reverse(). We get back a reversed string.

As you can see, Py_BuildValue() puts together both strings using a conversion string of “ss.” This creates a tuple of two strings, the original string and the reversed one. End of story, right? Unfortunately, no.

We got caught by one of the perils of C programming: the memory leak, that is, when memory is allocated but not freed. Memory leaks are analogous to borrowed books from the library but not returning them. You should always release resources which you have acquired when you no longer require them. How did we commit such a crime with our code (which looks innocent enough)?

When Py_BuildValue() puts together the Python object to return, it makes copies of the data it has been passed. In our case here, that would be a pair of strings. The problem is that we allocated the memory for the second string, but we did not release that memory when we finished, leaking it. What we really want to do is to build the return object and then free the memory that we allocated in our wrapper. We have no choice but to lengthen our code to:

								static PyObject *
Extest_doppel(PyObject *self, PyObject *args) {
    char *orig_str;                 // original string
    char *dupe_str;                 // reversed string
    PyObject* retval;

    if (!PyArg_ParseTuple(args, "s", &orig_str)) return NULL;
    retval = (PyObject*)Py_BuildValue("ss", orig_str, 
        dupestr=reverse(strdup(orig_str)));
    free(dupe_str);
    return retval;
}

We introduced the dupe_str variable to point to the newly-allocated string, built the return object and referenced it to retval. Then we free() the memory allocated and finally return back to the caller. Now we are done.

Add PyMethodDef ModuleMethods[]array/table for each module function

Now that both of our wrappers are complete, we want to list them somewhere so that the Python interpreter knows how to import and access them. This is the job of the ModuleMethods[] array.

It is made up of an array of arrays, with each individual array containing information about each function, terminated by a NULL array marking the end of the list. For our Extest module, we create the following ExtestMethods[] array:

								static PyMethodDef
ExtestMethods[] = {
     { "fac", Extest_fac, METH_VARARGS },
     { "doppel", Extest_doppel, METH_VARARGS },
     { NULL, NULL },
};

The Python-accessible names are given, followed by the corresponding wrapping functions. The constant METH_VARARGS is given, indicating a set of arguments in the form of a tuple. If we are using PyArg_ParseTupleAndKeywords() with keyworded arguments, we would logically OR this flag with the METH_KEYWORDS constant. Finally, a pair of NULLs properly terminates our list of two functions.

Add void initModule() module initializer function

The final piece to our puzzle is the module initializer function. This code is called when our module is imported for use by the interpreter. In this code, we make one call to Py_InitModule() along with the module name and the name of the ModuleMethods[] array so that the interpreter can access our module functions. For our Extest module, our initExtest() procedure looks like this:

    void initExtest() {
          Py_InitModule("Extest", ExtestMethods);
    }

We are now done with all our wrapping. We add all this code to our original code from Extest1.c and merge the results into a new file called Extest2.c, concluding the development phase of our example.

Another approach to creating an extension would be to make your wrapping code first, using “stubs” or test or dummy functions which will, during the course of development, be replaced by the fully functional pieces of implemented code. That way you can ensure that your interface between Python and C is correct, and then use Python to test your C code.

20.2.3. Compilation

Now we are on to the compilation phase. In order to get your new wrapper Python extension to build, you need to get it to compile with the Python library. This task has finally been standardized across platforms to make life a lot easier for extension designers.

Copy Misc/Makefile.pre.in

The first step is to copy the Makefile.pre.in file from the Misc directory of the Python distribution to your local directory where your extension is to be compiled. In fact, all steps take place in this same directory or folder.

Create Setup

The next step is to create a Setup file. The first line should contain the string “*shared*.” It is followed by line after line of module names followed by source files and compiler options which need to come together to build the module. If you have only one module, then it should be only one line. The format of these lines is the following:

								modName modFile[1, ***modFile2…][compiler_opts][linker_opts]

So for our Extest example, our Setup file consists of the following pair of lines:

*shared*
Extest Extest.2c

The “*shared*” string at the top of the Setup file means to create a shared library (.so object file), i.e., Extest.so. This file can then be imported by any Python module just as if your module was written in pure Python.

Create Makefile

Now we need to create the Makefile. We do this by issuing the make command:

% make -f Makefile.pre.in boot

This step usually gives a good amount of output, most of which is not important to you. It basically takes the information provided in the Setup file, adds its knowledge of where all the Python files are, and generates a Makefile so that you can build your module object file.

Compile and link your code by running make

% make
gcc -fpic  -O2 -m486 -fno-strength-reduce -I/usr/
include/python1.5 -I/usr/include/python1.5 -
DHAVE_CONFIG_H -c ./Extest2.c
gcc -shared  Extest2.o  -o Extestmodule.so

Import your module from Python

Now we can test out our module from the interpreter:

>>> import Extest
>>> Extest.fac(5)
120
>>> Extest.fac(9)
362880
>>> Extest.doppel('abcdefgh')
('abcdefgh', 'hgfedcba')
>>> Extest.doppel("Madam, I'm Adam.")
("Madam, I'm Adam.", ".madA m'I ,madaM")

Test function

The one last thing we want to do is to add a test function. In fact, we already have one, in the form of the main() function. Now it is potentially dangerous to have a main() function in our code because there should only be one main() in the system. We remove this danger by changing the name of our main() to test() and wrapping it, adding Extest_test() and updating ExtestMethods array so that they both look like this:

								static PyObject *
Extest_test(PyObject *self, PyObject *args) {
    test();
    return (PyObject*)Py_BuildValue("");
}
static PyMethodDef
ExtestMethods[] = {
    { "fac", Extest_fac, METH_VARARGS },
    { "doppel", Extest_doppel, METH_VARARGS },
    { "test", Extest_test, METH_VARARGS },
    { NULL, NULL },
};

The Extest_test() module function just runs test() and returns an empty string, resulting in a Python value of None being returned to the caller.

Now we can run the same test from Python:

>>> Extest.test()
4! == 24
8! == 40320
12! == 479001600
reversing 'abcdef', we get 'fedcba'
reversing 'madam', we get 'madam'
>>>

Below, we present the final version of Extest2.c (Example 20.2) that was used to generate the output we just witnessed.

Listing 20.2. Python-wrapped Version of C Library (Extest2.c)

1  #include <stdio.h>
2  #include <stdlib.h>
3  #include <string.h>
4
5  int fac(int n)
6  {
7      if (n < 2) return(1);
8      return ((n)*fac(n-1));
9  }
10
11 char *reverse(char *s)
12 {
13     register char t,
14                  *p = s,
15                  *q = (s + (strlen(s) - 1));
16
17     while (s && (p < q))
18     {
19         t = *p;
20         *p++ = *q;
21         *q-- = t;
22     }
23     return(s);
24 }
25
26 void test()
27 {
28     char s[BUFSIZ];
29     printf("4! == %d
", fac(4));
30     printf("8! == %d
", fac(8));
31     printf("12! == %d
", fac(12));
32     strcpy(s, "abcdef");
33     printf("reversing 'abcdef', we get '%s'
", 
34         reverse(s));
35     strcpy(s, "madam");
36     printf("reversing 'madam', we get '%s'
", 
37         reverse(s));
38 }
39
40 #include "Python.h"
41
42 static PyObject *
43 Extest_fac(PyObject *self, PyObject *args)
44 {
45     int num;
46     if (!PyArg_ParseTuple(args, "i", &num))
47         return NULL;
48     return (PyObject*)Py_BuildValue("i", fac(num));
49 }
50
51 static PyObject *
52 Extest_doppel(PyObject *self, PyObject *args)
53 {
54     char *orig_str;
55     char *dupe_str;
56     PyObject* retval;
57
58     if (!PyArg_ParseTuple(args, "s", &orig_str))
59         return NULL;
60     retval = (PyObject*)Py_BuildValue("ss", orig_str, 
61         dupe_str=reverse(strdup(orig_str)));
62     free(dupe_str);
63     return retval;
64 }
65
66 static PyObject *
67 Extest_test(PyObject *self, PyObject *args)
68 {
69     test();
70     return (PyObject*)Py_BuildValue("");
71 }
72
73 static PyMethodDef
74 ExtestMethods[] =
75 {
76     { "fac", Extest_fac, METH_VARARGS },
77     { "doppel", Extest_doppel, METH_VARARGS },
78     { "test", Extest_test, METH_VARARGS },
79     { NULL, NULL },
80 };
81
82 void initExtest()
83 {
84     Py_InitModule("Extest", ExtestMethods);
85 }

In this example, we chose to segregate our C code from our Python code. It just kept things easier to read and is no problem with our short example. In practice, these source files tend to get large, and some choose to implement their wrappers completely in a different source file, i.e., ExtestWrappers.c or something of that nature.

20.2.4. Reference Counting

You may recall that Python uses reference counting as a means of keeping track of objects and deallocating objects no longer referenced as part of the garbage collection mechanism. When creating extensions, you must pay extra special attention to how you manipulate Python objects because you must be mindful of whether or not you need to change the reference count for such objects.

There are two types of references you may have to an object, one of which is an owned reference, meaning that the reference count to the object is incremented by one to indicate your ownership. One place where you would definitely have an owned reference is where you create a Python object from scratch.

When you are done with a Python object, you must dispose of your ownership, either by decrementing the reference count, transferring your ownership by passing it on, or storing the object. Failure to dispose of an owned reference creates a memory leak.

You may also have a borrowed reference to an object. Somewhat lower on the responsibility ladder, this is where you are passed the reference of an object, but otherwise do not manipulate the data in any way nor do you have to worry about its reference count, so long as you do not hold onto this reference after its reference count has decreased to zero. You may convert your borrowed reference to an owned reference simply by incrementing an object's reference count.

Python provides a pairs of C macros which are used to change the reference count to a Python object. They are given in Table 20.3:

In our above Extest_test() function, we return None by building a PyObject with an empty string; however, it can also be accomplished by becoming an owner of the None object, PyNone, incrementing your reference count to it, and returning it explicitly, as in the following alternative piece of code:

							static PyObject *
Extest_test(PyObject *self, PyObject *args) {
    test();
    Py_INCREF(Py_None);
    return PyNone;
}

Py_INCREF() and Py_DECREF() also have versions which check for NULL objects, and they are Py_XINCREF() and Py_XDECREF(), respectively.

We strongly urge the reader to consult the Python documentation regarding extending and embedding Python for all the details with regards to reference counting (see the documentation reference in the Appendix).

20.2.5. Threading and GIL Awareness

Extension writers must be aware that their code may be executed in a multithreaded Python environment. Back in Section 17.3.1, we introduced the Python Virtual Machine (PVM) and the Global Interpreter Lock (GIL) and described how only one thread of execution can be running at any given time in the PVM, and that the GIL is responsible for keeping other threads from running. Furthermore, we indicated that code calling external functions such as in extension code would keep the GIL locked until the call returns.

We also hinted that there was a remedy, a way for the extension programmer to release the GIL, for example before performing a system call. This accomplished by “blocking” your code off to where threads may (and may not) run safely using another pair of C macros, Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS. A block of code bounded by these macros will permit other threads to run.

As with the reference counting macros, we urge you consult with the documentation regarding extending and embedding Python as well as the Python/C API reference manual.