The arguments to os.execlp specify the program to be run by giving command-line arguments used to start the program (i.e., what Python scripts know as sys.argv). If successful, the new program begins running and the call to os.execlp itself never returns (since the original program has been replaced, there’s really nothing to return to). If the call does return, an error has occurred, so we code an assert after it that will always raise an exception if reached.

There are a handful of os.exec variants in the Python standard library; some allow us to configure environment variables for the new program, pass command-line arguments in different forms, and so on. All are available on both Unix and Windows, and they replace the calling program (i.e., the Python interpreter). exec comes in eight flavors, which can be a bit confusing unless you generalize:

So when the script in Example 5-3 calls os.execlp, individually passed parameters specify a command line for the program to be run on, and the word python maps to an executable file according to the underlying system search-path setting environment variable (PATH). It’s as if we were running a command of the form python child.py 1 in a shell, but with a different command-line argument on the end each time.

Just as when typed at a shell, the string of arguments passed to os.execlp by the fork-exec script in Example 5-3 starts another Python program file, as shown in Example 5-4.

import os, sys
print('Hello from child', os.getpid(), sys.argv[1])

Here is this code in action on Linux. It doesn’t look much different from the original fork1.py, but it’s really running a new program in each forked process. More observant readers may notice that the child process ID displayed is the same in the parent program and the launched child.py program; os.execlp simply overlays a program in the same process:

[C:...PP4ESystemProcesses]$ python fork-exec.py
Child is 4556
Hello from child 4556 1

Child is 5920
Hello from child 5920 2

Child is 316
Hello from child 316 3
q

There are other ways to start up programs in Python besides the fork/exec combination. For example, the os.system and os.popen calls and subprocess module which we explored in Chapters 2 and 3 allow us to spawn shell commands. And the os.spawnv call and multiprocessing module, which we’ll meet later in this chapter, allow us to start independent programs and processes more portably. In fact, we’ll see later that multiprocessing’s process spawning model can be used as a sort of portable replacement for os.fork in some contexts (albeit a less efficient one) and used in conjunction with the os.exec* calls shown here to achieve a similar effect in standard Windows Python.

We’ll see more process fork examples later in this chapter, especially in the program exits and process communication sections, so we’ll forego additional examples here. We’ll also discuss additional process topics in later chapters of this book. For instance, forks are revisited in Chapter 12 to deal with servers and their zombies—dead processes lurking in system tables after their demise. For now, let’s move on to threads, a subject which at least some programmers find to be substantially less frightening…

    ./configure
    make
    make test
    sudo make install

Let’s start off by experimenting with a script that demonstrates the main thread interfaces. The script in Example 5-5 spawns threads until you reply with a q at the console; it’s similar in spirit to (and a bit simpler than) the script in Example 5-1, but it goes parallel with threads instead of process forks.

"spawn threads until you type 'q'"

import _thread

def child(tid):
    print('Hello from thread', tid)

def parent():
    i = 0
    while True:
        i += 1
        _thread.start_new_thread(child, (i,))
        if input() == 'q': break

parent()

This script really contains only two thread-specific lines: the import of the _thread module and the thread creation call. To start a thread, we simply call the _thread.start_new_thread function, no matter what platform we’re programming on.^[14] This call takes a function (or other callable) object and an arguments tuple and starts a new thread to execute a call to the passed function with the passed arguments. It’s almost like Python’s function(*args) call syntax, and similarly accepts an optional keyword arguments dictionary, too, but in this case the function call begins running in parallel with the rest of the program.

Operationally speaking, the _thread.start_new_thread call itself returns immediately with no useful value, and the thread it spawns silently exits when the function being run returns (the return value of the threaded function call is simply ignored). Moreover, if a function run in a thread raises an uncaught exception, a stack trace is printed and the thread exits, but the rest of the program continues. With the _thread module, the entire program exits silently on most platforms when the main thread does (though as we’ll see later, the threading module may require special handling if child threads are still running).

In practice, though, it’s almost trivial to use threads in a Python script. Let’s run this program to launch a few threads; we can run it on both Unix-like platforms and Windows this time, because threads are more portable than process forks—here it is spawning threads on Windows:

C:...PP4ESystemThreads> python thread1.py
Hello from thread 1

Hello from thread 2

Hello from thread 3

Hello from thread 4
q

Each message here is printed from a new thread, which exits almost as soon as it is started.

Although the preceding script runs a simple function, any callable object may be run in the thread, because all threads live in the same process. For instance, a thread can also run a lambda function or bound method of an object (the following code is part of file thread-alts.py in the book examples package):

import _thread                                       # all 3 print 4294967296

def action(i):                                       # function run in threads
    print(i ** 32)

class Power:
    def __init__(self, i):
        self.i = i
    def action(self):                                # bound method run in threads
        print(self.i ** 32)

_thread.start_new_thread(action, (2,))               # simple function

_thread.start_new_thread((lambda: action(2)), ())    # lambda function to defer

obj = Power(2)
_thread.start_new_thread(obj.action, ())             # bound method object

As we’ll see in larger examples later in this book, bound methods are especially useful in this role—because they remember both the method function and instance object, they also give access to state information and class methods for use within and during the thread.

More fundamentally, because threads all run in the same process, bound methods run by threads reference the original in-process instance object, not a copy of it. Hence, any changes to its state made in a thread will be visible to all threads automatically. Moreover, since bound methods of a class instance pass for callables interchangeably with simple functions, using them in threads this way just works. And as we’ll see later, the fact that they are normal objects also allows them to be stored freely on shared queues.

To really understand the power of threads running in parallel, though, we have to do something more long-lived in our threads, just as we did earlier for processes. Let’s mutate the fork-count program of the prior section to use threads. The script in Example 5-6 starts 5 copies of its counter function running in parallel threads.

"""
thread basics: start 5 copies of a function running in parallel;
uses time.sleep so that the main thread doesn't die too early--
this kills all other threads on some platforms; stdout is shared:
thread outputs may be intermixed in this version arbitrarily.
"""

import _thread as thread, time

def counter(myId, count):                        # function run in threads
    for i in range(count):
        time.sleep(1)                            # simulate real work
        print('[%s] => %s' % (myId, i))

for i in range(5):                               # spawn 5 threads
    thread.start_new_thread(counter, (i, 5))     # each thread loops 5 times

time.sleep(6)
print('Main thread exiting.')                    # don't exit too early

Each parallel copy of the counter function simply counts from zero up to four here and prints a message to standard output for each count.

Notice how this script sleeps for 6 seconds at the end. On Windows and Linux machines this has been tested on, the main thread shouldn’t exit while any spawned threads are running if it cares about their work; if it does exit, all spawned threads are immediately terminated. This differs from processes, where spawned children live on when parents exit. Without the sleep here, the spawned threads would die almost immediately after they are started.

This may seem ad hoc, but it isn’t required on all platforms, and programs are usually structured such that the main thread naturally lives as long as the threads it starts. For instance, a user interface may start an FTP download running in a thread, but the download lives a much shorter life than the user interface itself. Later in this section, we’ll also see different ways to avoid this sleep using global locks and flags that let threads signal their completion.

Moreover, we’ll later find that the threading module both provides a join method that lets us wait for spawned threads to finish explicitly, and refuses to allow a program to exit at all if any of its normal threads are still running (which may be useful in this case, but can require extra work to shut down in others). The multiprocessing module we’ll meet later in this chapter also allows spawned children to outlive their parents, though this is largely an artifact of its process-based model.

Now, when Example 5-6 is run on Windows 7 under Python 3.1, here is the output I get:

C:...PP4ESystemThreads> python thread-count.py
[1] => 0
[1] => 0
[0] => 0
[1] => 0
[0] => 0
[2] => 0
[3] => 0
[3] => 0

[1] => 1
[3] => 1
[3] => 1
[0] => 1[2] => 1
[3] => 1
[0] => 1[2] => 1
[4] => 1

[1] => 2
[3] => 2[4] => 2
[3] => 2[4] => 2
[0] => 2
[3] => 2[4] => 2
[0] => 2
[2] => 2
[3] => 2[4] => 2
[0] => 2
[2] => 2

...more output omitted...
Main thread exiting.

If this looks odd, it’s because it should. In fact, this demonstrates probably the most unusual aspect of threads. What’s happening here is that the output of the 5 threads run in parallel is intermixed—because all the threaded function calls run in the same process, they all share the same standard output stream (in Python terms, there is just one sys.stdout file between them, which is where printed text is sent). The net effect is that their output can be combined and confused arbitrarily. In fact, this script’s output can differ on each run. This jumbling of output grew even more pronounced in Python 3, presumably due to its new file output implementation.

More fundamentally, when multiple threads can access a shared resource like this, their access must be synchronized to avoid overlap in time—as explained in the next section.

One of the nice things about threads is that they automatically come with a cross-task communications mechanism: objects and namespaces in a process that span the life of threads are shared by all spawned threads. For instance, because every thread runs in the same process, if one Python thread changes a global scope variable, the change can be seen by every other thread in the process, main or child. Similarly, threads can share and change mutable objects in the process’s memory as long as they hold a reference to them (e.g., passed-in arguments). This serves as a simple way for a program’s threads to pass information—exit flags, result objects, event indicators, and so on—back and forth to each other.

The downside to this scheme is that our threads must sometimes be careful to avoid changing global objects and names at the same time. If two threads may change a shared object at once, it’s not impossible that one of the two changes will be lost (or worse, will silently corrupt the state of the shared object completely): one thread may step on the work done so far by another whose operations are still in progress. The extent to which this becomes an issue varies per application, and sometimes it isn’t an issue at all.

But even things that aren’t obviously at risk may be at risk. Files and streams, for example, are shared by all threads in a program; if multiple threads write to one stream at the same time, the stream might wind up with interleaved, garbled data. Example 5-6 of the prior section was a simple demonstration of this phenomenon in action, but it’s indicative of the sorts of clashes in time that can occur when our programs go parallel. Even simple changes can go awry if they might happen concurrently. To be robust, threaded programs need to control access to shared global items like these so that only one thread uses them at once.

Luckily, Python’s _thread module comes with its own easy-to-use tools for synchronizing access to objects shared among threads. These tools are based on the concept of a lock—to change a shared object, threads acquire a lock, make their changes, and then release the lock for other threads to grab. Python ensures that only one thread can hold a lock at any point in time; if others request it while it’s held, they are blocked until the lock becomes available. Lock objects are allocated and processed with simple and portable calls in the _thread module that are automatically mapped to thread locking mechanisms on the underlying platform.

For instance, in Example 5-7, a lock object created by _thread.allocate_lock is acquired and released by each thread around the print call that writes to the shared standard output stream.

"""
synchronize access to stdout: because it is shared global,
thread outputs may be intermixed if not synchronized
"""

import _thread as thread, time

def counter(myId, count):                        # function run in threads
    for i in range(count):
        time.sleep(1)                            # simulate real work
        mutex.acquire()
        print('[%s] => %s' % (myId, i))          # print isn't interrupted now
        mutex.release()

mutex = thread.allocate_lock()                   # make a global lock object
for i in range(5):                               # spawn 5 threads
    thread.start_new_thread(counter, (i, 5))     # each thread loops 5 times

time.sleep(6)
print('Main thread exiting.')                    # don't exit too early

Really, this script simply augments Example 5-6 to synchronize prints with a thread lock. The net effect of the additional lock calls in this script is that no two threads will ever execute a print call at the same point in time; the lock ensures mutually exclusive access to the stdout stream. Hence, the output of this script is similar to that of the original version, except that standard output text is never mangled by overlapping prints:

C:...PP4ESystemThreads> thread-count-mutex.py
[0] => 0
[1] => 0
[3] => 0
[2] => 0
[4] => 0
[0] => 1
[1] => 1
[3] => 1
[2] => 1
[4] => 1
[0] => 2
[1] => 2
[3] => 2
[4] => 2
[2] => 2
[0] => 3
[1] => 3
[3] => 3
[4] => 3
[2] => 3
[0] => 4
[1] => 4
[3] => 4
[4] => 4
[2] => 4
Main thread exiting.

Though somewhat platform-specific, the order in which the threads check in with their prints may still be arbitrary from run to run because they execute in parallel (getting work done in parallel is the whole point of threads, after all); but they no longer collide in time while printing their text. We’ll see other cases where the lock idiom comes in to play later in this chapter—it’s a core component of the multithreading model.

Besides avoiding print collisions, thread module locks are surprisingly useful. They can form the basis of higher-level synchronization paradigms (e.g., semaphores) and can be used as general thread communication devices.^[15] For instance, Example 5-8 uses a global list of locks to know when all child threads have finished.

"""
uses mutexes to know when threads are done in parent/main thread,
instead of time.sleep; lock stdout to avoid comingled prints;
"""

import _thread as thread
stdoutmutex = thread.allocate_lock()
exitmutexes = [thread.allocate_lock() for i in range(10)]

def counter(myId, count):
    for i in range(count):
        stdoutmutex.acquire()
        print('[%s] => %s' % (myId, i))
        stdoutmutex.release()
    exitmutexes[myId].acquire()    # signal main thread

for i in range(10):
    thread.start_new_thread(counter, (i, 100))

for mutex in exitmutexes:
    while not mutex.locked(): pass
print('Main thread exiting.')

A lock’s locked method can be used to check its state. To make this work, the main thread makes one lock per child and tacks them onto a global exitmutexes list (remember, the threaded function shares global scope with the main thread). On exit, each thread acquires its lock on the list, and the main thread simply watches for all locks to be acquired. This is much more accurate than naïvely sleeping while child threads run in hopes that all will have exited after the sleep. Run this on your own to see its output—all 10 spawned threads count up to 100 (they run in arbitrarily interleaved order that can vary per run and platform, but their prints run atomically and do not comingle), before the main thread exits.

Depending on how your threads run, this could be even simpler: since threads share global memory anyhow, we can usually achieve the same effect with a simple global list of integers instead of locks. In Example 5-9, the module’s namespace (scope) is shared by top-level code and the threaded function, as before. exitmutexes refers to the same list object in the main thread and all threads it spawns. Because of that, changes made in a thread are still noticed in the main thread without resorting to extra locks.

"""
uses simple shared global data (not mutexes) to know when threads
are done in parent/main thread; threads share list but not its items,
assumes list won't move in memory once it has been created initially
"""

import _thread as thread
stdoutmutex = thread.allocate_lock()
exitmutexes = [False] * 10

def counter(myId, count):
    for i in range(count):
        stdoutmutex.acquire()
        print('[%s] => %s' % (myId, i))
        stdoutmutex.release()
    exitmutexes[myId] = True  # signal main thread

for i in range(10):
    thread.start_new_thread(counter, (i, 100))

while False in exitmutexes: pass
print('Main thread exiting.')

The output of this script is similar to the prior—10 threads counting to 100 in parallel and synchronizing their prints along the way. In fact, both of the last two counting thread scripts produce roughly the same output as the original thread_count.py, albeit without stdout corruption and with larger counts and different random ordering of output lines. The main difference is that the main thread exits immediately after (and no sooner than!) the spawned child threads:

C:...PP4ESystemThreads> python thread-count-wait2.py
...more deleted...
[4] => 98
[6] => 98
[8] => 98
[5] => 98
[0] => 99
[7] => 98
[9] => 98
[1] => 99
[3] => 99
[2] => 99
[4] => 99
[6] => 99
[8] => 99
[5] => 99
[7] => 99
[9] => 99
Main thread exiting.

Notice how the main threads of both of the last two scripts fall into busy-wait loops at the end, which might become significant performance drains in tight applications. If so, simply add a time.sleep call in the wait loops to insert a pause between end tests and to free up the CPU for other tasks: this call pauses the calling thread only (in this case, the main one). You might also try experimenting with adding sleep calls to the thread function to simulate real work.

Passing in the lock to threaded functions as an argument instead of referencing it in the global scope might be more coherent, too. When passed in, all threads reference the same object, because they are all part of the same process. Really, the process’s object memory is shared memory for threads, regardless of how objects in that shared memory are referenced (whether through global scope variables, passed argument names, object attributes, or another way).

And while we’re at it, the with statement can be used to ensure thread operations around a nested block of code, much like its use to ensure file closure in the prior chapter. The thread lock’s context manager acquires the lock on with statement entry and releases it on statement exit regardless of exception outcomes. The net effect is to save one line of code, but also to guarantee lock release when exceptions are possible. Example 5-10 adds all these coding alternatives to our threaded counter script.

"""
passed in mutex object shared by all threads instead of globals;
use with context manager statement for auto acquire/release;
sleep calls added to avoid busy loops and simulate real work
"""

import _thread as thread, time
stdoutmutex = thread.allocate_lock()
numthreads  = 5
exitmutexes = [thread.allocate_lock() for i in range(numthreads)]

def counter(myId, count, mutex):                     # shared object passed in
    for i in range(count):
        time.sleep(1 / (myId+1))                     # diff fractions of second
        with mutex:                                  # auto acquire/release: with
            print('[%s] => %s' % (myId, i))
    exitmutexes[myId].acquire()                      # global: signal main thread

for i in range(numthreads):
    thread.start_new_thread(counter, (i, 5, stdoutmutex))

while not all(mutex.locked() for mutex in exitmutexes): time.sleep(0.25)
print('Main thread exiting.')

When run, the different sleep times per thread make them run more independently:

C:...PP4ESystemThreads> thread-count-wait3.py
[4] => 0
[3] => 0
[2] => 0
[4] => 1
[1] => 0
[3] => 1
[4] => 2
[2] => 1
[3] => 2
[4] => 3
[4] => 4
[0] => 0
[1] => 1
[2] => 2
[3] => 3
[3] => 4
[2] => 3
[1] => 2
[2] => 4
[0] => 1
[1] => 3
[1] => 4
[0] => 2
[0] => 3
[0] => 4
Main thread exiting.

Of course, threads are for much more than counting. We’ll put shared global data to more practical use in Adding a User Interface, where it will serve as completion signals from child processing threads transferring data over a network to a main thread controlling a tkinter GUI display, and again later in Chapter 10’s threadtools and Chapter 14’s PyMailGUI to post results of email operations to a GUI (watch for Preview: GUIs and Threads for more pointers on this topic). Global data shared among threads also turns out to be the basis of queues, which are discussed later in this chapter; each thread gets or puts data using the same shared queue object.

The Thread class can also be used to start a simple function, or any other type of callable object, without coding subclasses at all—if not redefined, the Thread class’s default run method simply calls whatever you pass to its constructor’s target argument, with any provided arguments passed to args (which defaults to () for none). This allows us to use Thread to run simple functions, too, though this call form is not noticeably simpler than the basic _thread module. For instance, the following code snippets sketch four different ways to spawn the same sort of thread (see four-threads*.py in the examples tree; you can run all four in the same script, but would have to also synchronize prints to avoid overlap):

import threading, _thread
def action(i):
    print(i ** 32)

# subclass with state
class Mythread(threading.Thread):
    def __init__(self, i):
        self.i = i
        threading.Thread.__init__(self)
    def run(self):                                        # redefine run for action
        print(self.i ** 32)
Mythread(2).start()                                       # start invokes run()

# pass action in
thread = threading.Thread(target=(lambda: action(2)))     # run invokes target
thread.start()

# same but no lambda wrapper for state
threading.Thread(target=action, args=(2,)).start()        # callable plus its args

# basic thread module
_thread.start_new_thread(action, (2,))                    # all-function interface

As a rule of thumb, class-based threads may be better if your threads require per-thread state, or can leverage any of OOP’s many benefits in general. Your thread classes don’t necessarily have to subclass Thread, though. In fact, just as in the _thread module, the thread’s target in threading may be any type of callable object. When combined with techniques such as bound methods and nested scope references, the choice between coding techniques becomes even less clear-cut:

# a non-thread class with state, OOP
class Power:
    def __init__(self, i):
        self.i = i
    def action(self):
        print(self.i ** 32)

obj = Power(2)
threading.Thread(target=obj.action).start()        # thread runs bound method

# nested scope to retain state
def action(i):
    def power():
        print(i ** 32)
    return power

threading.Thread(target=action(2)).start()         # thread runs returned function

# both with basic thread module
_thread.start_new_thread(obj.action, ())           # thread runs a callable object
_thread.start_new_thread(action(2), ())

As usual, the threading APIs are as flexible as the Python language itself.

Earlier, we saw how print operations in threads need to be synchronized with locks to avoid overlap, because the output stream is shared by all threads. More formally, threads need to synchronize their changes to any item that may be shared across thread in a process—both objects and namespaces. Depending on a given program’s goals, this might include:

For instance, even simple global variables can require coordination if concurrent updates are possible, as in Example 5-12.

"prints different results on different runs on Windows 7"

import threading, time
count = 0

def adder():
    global count
    count = count + 1             # update a shared name in global scope
    time.sleep(0.5)               # threads share object memory and global names
    count = count + 1

threads = []
for i in range(100):
    thread = threading.Thread(target=adder, args=())
    thread.start()
    threads.append(thread)

for thread in threads: thread.join()
print(count)

Here, 100 threads are spawned to update the same global scope variable twice (with a sleep between updates to better interleave their operations). When run on Windows 7 with Python 3.1, different runs produce different results:

C:...PP4ESystemThreads> thread-add-random.py
189

C:...PP4ESystemThreads> thread-add-random.py
200

C:...PP4ESystemThreads> thread-add-random.py
194

C:...PP4ESystemThreads> thread-add-random.py
191

This happens because threads overlap arbitrarily in time: statements, even the simple assignment statements like those here, are not guaranteed to run to completion by themselves (that is, they are not atomic). As one thread updates the global, it may be using the partial result of another thread’s work in progress. The net effect is this seemingly random behavior. To make this script work correctly, we need to again use thread locks to synchronize the updates—when Example 5-13 is run, it always prints 200 as expected.

"prints 200 each time, because shared resource access synchronized"

import threading, time
count = 0

def adder(addlock):                 # shared lock object passed in
    global count
    with addlock:
        count = count + 1           # auto acquire/release around stmt
    time.sleep(0.5)
    with addlock:
        count = count + 1           # only 1 thread updating at once

addlock = threading.Lock()
threads = []
for i in range(100):
    thread = threading.Thread(target=adder, args=(addlock,))
    thread.start()
    threads.append(thread)

for thread in threads: thread.join()
print(count)

Although some basic operations in the Python language are atomic and need not be synchronized, you’re probably better off doing so for every potential concurrent update. Not only might the set of atomic operations change over time, but the internal implementation of threads in general can as well (and in fact, it may in Python 3.2, as described ahead).

Of course, this is an artificial example (spawning 100 threads to add twice isn’t exactly a real-world use case for threads!), but it illustrates the issues that threads must address for any sort of potentially concurrent updates to shared object or name. Luckily, for many or most realistic applications, the queue module of the next section can make thread synchronization an automatic artifact of program structure.

Before we move ahead, I should point out that besides Thread and Lock, the threading module also includes higher-level objects for synchronizing access to shared items (e.g., Semaphore, Condition, Event)—many more, in fact, than we have space to cover here; see the library manual for details. For more examples of threads and forks in general, see the remainder this chapter as well as the examples in the GUI and network scripting parts of this book. We will thread GUIs, for instance, to avoid blocking them, and we will thread and fork network servers to avoid denying service to clients.

We’ll also explore the threading module’s approach to program exits in the absence of join calls in conjunction with queues—our next topic.

Notice how the queue is assigned to a global variable; because of that, it is shared by all of the spawned threads (all of them run in the same process and in the same global scope). Since these threads change an object instead of a variable name, it would work just as well to pass the queue object in to the threaded functions as an argument—the queue is a shared object in memory, regardless of how it is referenced (see queuetest2.py in the examples tree for a full version that does this):

dataQueue = queue.Queue()             # shared object, infinite size

def producer(idnum, dataqueue):
    for msgnum in range(nummessages):
        time.sleep(idnum)
        dataqueue.put('[producer id=%d, count=%d]' % (idnum, msgnum))

def consumer(idnum, dataqueue): ...

if __name__ == '__main__':
    for i in range(numproducers):
        thread.start_new_thread(producer, (i, dataQueue))
    for i in range(numproducers):
        thread.start_new_thread(producer, (i, dataQueue))

Also notice how this script exits when the main thread does, even though consumer threads are still running in their infinite loops. This works fine on Windows (and most other platforms)—with the basic _thread module, the program ends silently when the main thread does. This is why we’ve had to sleep in some examples to give threads time to do their work, but is also why we do not need to be concerned about exiting while consumer threads are still running here.

In the alternative threading module, though, the program will not exit if any spawned threads are running, unless they are set to be daemon threads. Specifically, the entire program exits when only daemon threads are left. Threads inherit a default initial daemonic value from the thread that creates them. The initial thread of a Python program is considered not daemonic, though alien threads created outside this module’s control are considered daemonic (including some threads created in C code). To override inherited defaults, a thread object’s daemon flag can be set manually. In other words, nondaemon threads prevent program exit, and programs by default do not exit until all threading-managed threads finish.

This is either a feature or nonfeature, depending on your program—it allows spawned worker threads to finish their tasks in the absence of join calls or sleeps, but it can prevent programs like the one in Example 5-14 from shutting down when they wish. To make this example work with threading, use the following alternative code (see queuetest3.py in the examples tree for a complete version of this, as well as thread-count-threading.py, also in the tree, for a case where this refusal to exit can come in handy):

import threading, queue, time

def producer(idnum, dataqueue): ...

def consumer(idnum, dataqueue): ...

if __name__ == '__main__':
    for i in range(numconsumers):
        thread = threading.Thread(target=consumer, args=(i, dataQueue))
        thread.daemon = True  # else cannot exit!
        thread.start()

    waitfor = []
    for i in range(numproducers):
        thread = threading.Thread(target=producer, args=(i, dataQueue))
        waitfor.append(thread)
        thread.start()

    for thread in waitfor: thread.join()    # or time.sleep() long enough here
    print('Main thread exit.')

We’ll revisit the daemons and exits issue in Chapter 10 while studying GUIs; as we’ll see, it’s no different in that context, except that the main thread is usually the GUI itself.

Now, as coded in Example 5-14, the following is the output of this example when run on my Windows machine. Notice that even though the queue automatically coordinates the communication of data between the threads, this script still must use a lock to manually synchronize access to the standard output stream; queues synchronize data passing, but some programs may still need to use locks for other purposes. As in prior examples, if the safeprint lock is not used, the printed lines from one consumer may be intermixed with those of another. It is not impossible that a consumer may be paused in the middle of a print operation:

C:...PP4ESystemThreads> queuetest.py
consumer 1 got => [producer id=0, count=0]
consumer 0 got => [producer id=0, count=1]
consumer 1 got => [producer id=0, count=2]
consumer 0 got => [producer id=0, count=3]
consumer 1 got => [producer id=1, count=0]
consumer 1 got => [producer id=2, count=0]
consumer 0 got => [producer id=1, count=1]
consumer 1 got => [producer id=3, count=0]
consumer 0 got => [producer id=1, count=2]
consumer 1 got => [producer id=2, count=1]
consumer 1 got => [producer id=1, count=3]
consumer 1 got => [producer id=3, count=1]
consumer 0 got => [producer id=2, count=2]
consumer 1 got => [producer id=2, count=3]
consumer 1 got => [producer id=3, count=2]
consumer 1 got => [producer id=3, count=3]
Main thread exit.

Try adjusting the parameters at the top of this script to experiment with different scenarios. A single consumer, for instance, would simulate a GUI’s main thread. Here is the output of a single-consumer run—producers still add to the queue in fairly random fashion, because threads run in parallel with each other and with the consumer:

C:...PP4ESystemThreads> queuetest.py
consumer 0 got => [producer id=0, count=0]
consumer 0 got => [producer id=0, count=1]
consumer 0 got => [producer id=0, count=2]
consumer 0 got => [producer id=0, count=3]
consumer 0 got => [producer id=1, count=0]
consumer 0 got => [producer id=2, count=0]
consumer 0 got => [producer id=1, count=1]
consumer 0 got => [producer id=3, count=0]
consumer 0 got => [producer id=1, count=2]
consumer 0 got => [producer id=2, count=1]
consumer 0 got => [producer id=1, count=3]
consumer 0 got => [producer id=3, count=1]
consumer 0 got => [producer id=2, count=2]
consumer 0 got => [producer id=2, count=3]
consumer 0 got => [producer id=3, count=2]
consumer 0 got => [producer id=3, count=3]
Main thread exit.

In addition to the basics used in our script, queues may be fixed or infinite in size, and get and put calls may or may not block; see the Python library manual for more details on queue interface options. Since we just simulated a typical GUI structure, though, let’s explore the notion a bit further.

Some concurrent updates might work without locks if the thread-switch interval is set high enough to allow each thread to finish without being swapped out. The sys.setcheckinterval(N) call sets the frequency with which the interpreter checks for things like thread switches and signal handlers.

This interval defines the number of bytecode instructions before a switch. It does not need to be reset for most programs, but it can be used to tune thread performance. Setting higher values means switches happen less often: threads incur less overhead but they are less responsive to events. Setting lower values makes threads more responsive to events but increases thread switch overhead.

Because of the way Python uses the GIL to synchronize threads’ access to the virtual machine, whole statements are not generally thread-safe, but each bytecode instruction is. Because of this bytecode indivisibility, some Python language operations are thread-safe—also called atomic, because they run without interruption—and do not require the use of locks or queues to avoid concurrent update issues. For instance, as of this writing, list.append, fetches and some assignments for variables, list items, dictionary keys, and object attributes, and other operations were still atomic in standard C Python; others, such as x = x+1 (and any operation in general that reads data, modifies it, and writes it back) were not.

As mentioned earlier, though, relying on these rules is a bit of a gamble, because they require a deep understanding of Python internals and may vary per release. Indeed, the set of atomic operations may be radically changed if a new free-threaded implementation ever appears. As a rule of thumb, it may be easier to use locks for all access to global and shared objects than to try to remember which types of access may or may not be safe across multiple threads.

Finally, if you plan to mix Python with C, also see the thread interfaces described in the Python/C API standard manual. In threaded programs, C extensions must release and reacquire the GIL around long-running operations to let the Python language portions of other Python threads run during the wait. Specifically, the long-running C extension function should release the lock on entry and reacquire it on exit when resuming Python code.

Also note that even though the Python code in Python threads cannot truly overlap in time due to the GIL synchronization, the C-coded portions of threads can. Any number may be running in parallel, as long as they do work outside the scope of the Python virtual machine. In fact, C threads may overlap both with other C threads and with Python language threads run in the virtual machine. Because of this, splitting code off to C libraries is one way that Python applications can still take advantage of multi-CPU machines.

Still, it may often be easier to leverage such machines by simply writing Python programs that fork processes instead of starting threads. The complexity of process and thread code is similar. For more on C extensions and their threading requirements, see Chapter 20. In short, Python includes C language tools (including a pair of GIL management macros) that can be used to wrap long-running operations in C-coded extensions and that allow other Python language threads to run in parallel.

By now, you should have a basic grasp of parallel processes and threads, and Python’s tools that support them. Later in this chapter, we’ll revisit both ideas to study the multiprocessing module—a standard library tool that seeks to combine the simplicity and portability of threads with the benefits of processes, by implementing a threading-like API that runs processes instead of threads. It seeks to address the portability issue of processes, as well as the multiple-CPU limitations imposed in threads by the GIL, but it cannot be used as a replacement for forking in some contexts, and it imposes some constraints that threads do not, which stem from its process-based model (for instance, mutable object state is not directly shared because objects are copied across process boundaries, and unpickleable objects such as bound methods cannot be as freely used).

Because the multiprocessing module also implements tools to simplify tasks such as inter-process communication and exit status, though, let’s first get a handle on Python’s support in those domains as well, and explore some more process and thread examples along the way.

Let’s look at the case of the shell commands first—the following, run on Linux, spawns Example 5-15, and Example 5-16 reads the output streams through pipes and fetches their exit status codes:

[mark@linux]$ python
>>> import os
>>> pipe = os.popen('python testexit_sys.py')
>>> pipe.read()
'Bye sys world12'
>>> stat = pipe.close()              # returns exit code
>>> stat
10752
>>> hex(stat)
'0x2a00'
>>> stat >> 8                        # extract status from bitmask on Unix-likes
42

>>> pipe = os.popen('python testexit_os.py')
>>> stat = pipe.close()
>>> stat, stat >> 8
(25344, 99)

This code works the same under Cygwin Python on Windows. When using os.popen on such Unix-like platforms, for reasons we won’t go into here, the exit status is actually packed into specific bit positions of the return value; it’s really there, but we need to shift the result right by eight bits to see it. Commands run with os.system send their statuses back directly through the Python library call:

>>> stat = os.system('python testexit_sys.py')
Bye sys world
>>> stat, stat >> 8
(10752, 42)

>>> stat = os.system('python testexit_os.py')
Bye os world
>>> stat, stat >> 8
(25344, 99)

All of this code works under the standard version of Python for Windows, too, though exit status is not encoded in a bit mask (test sys.platform if your code must handle both formats):

C:...PP4ESystemExits> python
>>> os.system('python testexit_sys.py')
Bye sys world
42
>>> os.system('python testexit_os.py')
Bye os world
99

>>> pipe = os.popen('python testexit_sys.py')
>>> pipe.read()
'Bye sys world
'
>>> pipe.close()
42
>>>
>>> os.popen('python testexit_os.py').close()
99

Notice that the last test in the preceding code didn’t attempt to read the command’s output pipe. If we do, we may have to run the target script in unbuffered mode with the -u Python command-line flag or change the script to flush its output manually with sys.stdout.flush. Otherwise, the text printed to the standard output stream might not be flushed from its buffer when os._exit is called in this case for immediate shutdown. By default, standard output is fully buffered when connected to a pipe like this; it’s only line-buffered when connected to a terminal:

>>> pipe = os.popen('python testexit_os.py')
>>> pipe.read()                                     # streams not flushed on exit
''

>>> pipe = os.popen('python -u testexit_os.py')     # force unbuffered streams
>>> pipe.read()
'Bye os world
'

Confusingly, you can pass mode and buffering argument to specify line buffering in both os.popen and subprocess.Popen, but this won’t help here—arguments passed to these tools pertain to the calling process’s input end of the pipe, not to the spawned program’s output stream:

>>> pipe = os.popen('python testexit_os.py', 'r', 1)   # line buffered only
>>> pipe.read()                                        # but my pipe, not program's!
''

>>> from subprocess import Popen, PIPE
>>> pipe = Popen('python testexit_os.py', bufsize=1, stdout=PIPE)   # for my pipe
>>> pipe.stdout.read()                                              # doesn't help
b''

Really, buffering mode arguments in these tools pertain to output the caller writes to a command’s standard input stream, not to output read from that command.

If required, the spawned script itself can also manually flush its output buffers periodically or before forced exits. More on buffering when we discuss the potential for deadlocks later in this chapter, and again in Chapters 10 and 12 where we’ll see how it applies to sockets. Since we brought up subprocess, though, let’s turn to its exit tools next.

The alternative subprocess module offers exit status in a variety of ways, as we saw in Chapters 2 and 3 (a None value in returncode indicates that the spawned program has not yet terminated):

C:...PP4ESystemExits> python
>>> from subprocess import Popen, PIPE, call
>>> pipe = Popen('python testexit_sys.py', stdout=PIPE)
>>> pipe.stdout.read()
b'Bye sys world
'
>>> pipe.wait()
42

>>> call('python testexit_sys.py')
Bye sys world
42

>>> pipe = Popen('python testexit_sys.py', stdout=PIPE)
>>> pipe.communicate()
(b'Bye sys world
', None)
>>> pipe.returncode
42

The subprocess module works the same on Unix-like platforms like Cygwin, but unlike os.popen, the exit status is not encoded, and so it matches the Windows result (note that shell=True is needed to run this as is on Cygwin and Unix-like platforms, as we learned in Chapter 2; on Windows this argument is required only to run commands built into the shell, like dir):

[C:...PP4ESystemExits]$ python
>>> from subprocess import Popen, PIPE, call
>>> pipe = Popen('python testexit_sys.py', stdout=PIPE, shell=True)
>>> pipe.stdout.read()
b'Bye sys world
'
>>> pipe.wait()
42

>>> call('python testexit_sys.py', shell=True)
Bye sys world
42

Since they are more traditional, let’s start with a look at anonymous pipes. To illustrate, the script in Example 5-19 uses the os.fork call to make a copy of the calling process as usual (we met forks earlier in this chapter). After forking, the original parent process and its child copy speak through the two ends of a pipe created with os.pipe prior to the fork. The os.pipe call returns a tuple of two file descriptors—the low-level file identifiers we met in Chapter 4—representing the input and output sides of the pipe. Because forked child processes get copies of their parents’ file descriptors, writing to the pipe’s output descriptor in the child sends data back to the parent on the pipe created before the child was spawned.

import os, time

def child(pipeout):
    zzz = 0
    while True:
        time.sleep(zzz)                          # make parent wait
        msg = ('Spam %03d' % zzz).encode()       # pipes are binary bytes
        os.write(pipeout, msg)                   # send to parent
        zzz = (zzz+1) % 5                        # goto 0 after 4

def parent():
    pipein, pipeout = os.pipe()                  # make 2-ended pipe
    if os.fork() == 0:                           # copy this process
        child(pipeout)                           # in copy, run child
    else:                                        # in parent, listen to pipe
        while True:
            line = os.read(pipein, 32)           # blocks until data sent
            print('Parent %d got [%s] at %s' % (os.getpid(), line, time.time()))

parent()

If you run this program on Linux, Cygwin, or another Unix-like platform (pipe is available on standard Windows Python, but fork is not), the parent process waits for the child to send data on the pipe each time it calls os.read. It’s almost as if the child and parent act as client and server here—the parent starts the child and waits for it to initiate communication.^[17] To simulate differing task durations, the child keeps the parent waiting one second longer between messages with time.sleep calls, until the delay has reached four seconds. When the zzz delay counter hits 005, it rolls back down to 000 and starts again:

[C:...PP4ESystemProcesses]$ python pipe1.py
Parent 6716 got [b'Spam 000'] at 1267996104.53
Parent 6716 got [b'Spam 001'] at 1267996105.54
Parent 6716 got [b'Spam 002'] at 1267996107.55
Parent 6716 got [b'Spam 003'] at 1267996110.56
Parent 6716 got [b'Spam 004'] at 1267996114.57
Parent 6716 got [b'Spam 000'] at 1267996114.57
Parent 6716 got [b'Spam 001'] at 1267996115.59
Parent 6716 got [b'Spam 002'] at 1267996117.6
Parent 6716 got [b'Spam 003'] at 1267996120.61
Parent 6716 got [b'Spam 004'] at 1267996124.62
Parent 6716 got [b'Spam 000'] at 1267996124.62
Parent 6716 got [b'Spam 001'] at 1267996125.63
...etc.: Ctrl-C to exit...

Notice how the parent received a bytes string through the pipe. Raw pipes normally deal in binary byte strings when their descriptors are used directly this way with the descriptor-based file tools we met in Chapter 4 (as we saw there, descriptor read and write tools in os always return and expect byte strings). That’s why we also have to manually encode to bytes when writing in the child—the string formatting operation is not available on bytes. As the next section shows, it’s also possible to wrap a pipe descriptor in a text-mode file object, much as we did in the file examples in Chapter 4, but that object simply performs encoding and decoding automatically on transfers; it’s still bytes in the pipe.

If you look closely at the preceding output, you’ll see that when the child’s delay counter hits 004, the parent ends up reading two messages from the pipe at the same time; the child wrote two distinct messages, but on some platforms or configurations (other than that used here) they might be interleaved or processed close enough in time to be fetched as a single unit by the parent. Really, the parent blindly asks to read, at most, 32 bytes each time, but it gets back whatever text is available in the pipe, when it becomes available.

To distinguish messages better, we can mandate a separator character in the pipe. An end-of-line makes this easy, because we can wrap the pipe descriptor in a file object with os.fdopen and rely on the file object’s readline method to scan up through the next separator in the pipe. This also lets us leverage the more powerful tools of the text-mode file object we met in Chapter 4. Example 5-20 implements this scheme for the parent’s end of the pipe.

# same as pipe1.py, but wrap pipe input in stdio file object
# to read by line, and close unused pipe fds in both processes

import os, time

def child(pipeout):
    zzz = 0
    while True:
        time.sleep(zzz)                          # make parent wait
        msg = ('Spam %03d
' % zzz).encode()     # pipes are binary in 3.X
        os.write(pipeout, msg)                   # send to parent
        zzz = (zzz+1) % 5                        # roll to 0 at 5

def parent():
    pipein, pipeout = os.pipe()                  # make 2-ended pipe
    if os.fork() == 0:                           # in child, write to pipe
        os.close(pipein)                         # close input side here
        child(pipeout)
    else:                                        # in parent, listen to pipe
        os.close(pipeout)                        # close output side here
        pipein = os.fdopen(pipein)               # make text mode input file object
        while True:
            line = pipein.readline()[:-1]        # blocks until data sent
            print('Parent %d got [%s] at %s' % (os.getpid(), line, time.time()))

parent()

This version has also been augmented to close the unused end of the pipe in each process (e.g., after the fork, the parent process closes its copy of the output side of the pipe written by the child); programs should close unused pipe ends in general. Running with this new version reliably returns a single child message to the parent each time it reads from the pipe, because they are separated with markers when written:

[C:...PP4ESystemProcesses]$ python pipe2.py
Parent 8204 got [Spam 000] at 1267997789.33
Parent 8204 got [Spam 001] at 1267997790.03
Parent 8204 got [Spam 002] at 1267997792.05
Parent 8204 got [Spam 003] at 1267997795.06
Parent 8204 got [Spam 004] at 1267997799.07
Parent 8204 got [Spam 000] at 1267997799.07
Parent 8204 got [Spam 001] at 1267997800.08
Parent 8204 got [Spam 002] at 1267997802.09
Parent 8204 got [Spam 003] at 1267997805.1
Parent 8204 got [Spam 004] at 1267997809.11
Parent 8204 got [Spam 000] at 1267997809.11
Parent 8204 got [Spam 001] at 1267997810.13
...etc.: Ctrl-C to exit...

Notice that this version’s reads also return a text data str object now, per the default r text mode for os.fdopen. As mentioned, pipes normally deal in binary byte strings when their descriptors are used directly with os file tools, but wrapping in text-mode files allows us to use str strings to represent text data instead of bytes. In this example, bytes are decoded to str when read by the parent; using os.fdopen and text mode in the child would allow us to avoid its manual encoding call, but the file object would encode the str data anyhow (though the encoding is trivial for ASCII bytes like those used here). As for simple files, the best mode for processing pipe data in is determined by its nature.

Although the os.fork call required by the prior section’s examples isn’t available on standard Windows Python, os.pipe is. Because threads all run in the same process and share file descriptors (and global memory in general), this makes anonymous pipes usable as a communication and synchronization device for threads, too. This is an arguably lower-level mechanism than queues or shared names and objects, but it provides an additional IPC option for threads. Example 5-21, for instance, demonstrates the same type of pipe-based communication occurring between threads instead of processes.

# anonymous pipes and threads, not processes; this version works on Windows

import os, time, threading

def child(pipeout):
    zzz = 0
    while True:
        time.sleep(zzz)                              # make parent wait
        msg = ('Spam %03d' % zzz).encode()           # pipes are binary bytes
        os.write(pipeout, msg)                       # send to parent
        zzz = (zzz+1) % 5                            # goto 0 after 4

def parent(pipein):
    while True:
        line = os.read(pipein, 32)                   # blocks until data sent
        print('Parent %d got [%s] at %s' % (os.getpid(), line, time.time()))

pipein, pipeout = os.pipe()
threading.Thread(target=child, args=(pipeout,)).start()
parent(pipein)

Since threads work on standard Windows Python, this script does too. The output is similar here, but the speakers are in-process threads, not processes (note that because of its simple-minded infinite loops, at least one of its threads may not die on a Ctrl-C—on Windows you may need to use Task Manager to kill the python.exe process running this script or close its window to exit):

C:...PP4ESystemProcesses> pipe-thread.py
Parent 8876 got [b'Spam 000'] at 1268579215.71
Parent 8876 got [b'Spam 001'] at 1268579216.73
Parent 8876 got [b'Spam 002'] at 1268579218.74
Parent 8876 got [b'Spam 003'] at 1268579221.75
Parent 8876 got [b'Spam 004'] at 1268579225.76
Parent 8876 got [b'Spam 000'] at 1268579225.76
Parent 8876 got [b'Spam 001'] at 1268579226.77
Parent 8876 got [b'Spam 002'] at 1268579228.79
...etc.: Ctrl-C or Task Manager to exit...

Pipes normally let data flow in only one direction—one side is input, one is output. What if you need your programs to talk back and forth, though? For example, one program might send another a request for information and then wait for that information to be sent back. A single pipe can’t generally handle such bidirectional conversations, but two pipes can. One pipe can be used to pass requests to a program and another can be used to ship replies back to the requestor.

This really does have real-world applications. For instance, I once added a GUI interface to a command-line debugger for a C-like programming language by connecting two processes with pipes this way. The GUI ran as a separate process that constructed and sent commands to the non-GUI debugger’s input stream pipe and parsed the results that showed up in the debugger’s output stream pipe. In effect, the GUI acted like a programmer typing commands at a keyboard and a client to the debugger server. More generally, by spawning command-line programs with streams attached by pipes, systems can add new interfaces to legacy programs. In fact, we’ll see a simple example of this sort of GUI program structure in Chapter 10.

The module in Example 5-22 demonstrates one way to apply this idea to link the input and output streams of two programs. Its spawn function forks a new child program and connects the input and output streams of the parent to the output and input streams of the child. That is:

The net effect is that the two independent programs communicate by speaking over their standard streams.

"""
spawn a child process/program, connect my stdin/stdout to child process's
stdout/stdin--my reads and writes map to output and input streams of the
spawned program; much like tying together streams with subprocess module;
"""

import os, sys

def spawn(prog, *args):                       # pass progname, cmdline args
    stdinFd  = sys.stdin.fileno()             # get descriptors for streams
    stdoutFd = sys.stdout.fileno()            # normally stdin=0, stdout=1

    parentStdin, childStdout  = os.pipe()     # make two IPC pipe channels
    childStdin,  parentStdout = os.pipe()     # pipe returns (inputfd, outoutfd)
    pid = os.fork()                           # make a copy of this process
    if pid:
        os.close(childStdout)                 # in parent process after fork:
        os.close(childStdin)                  # close child ends in parent
        os.dup2(parentStdin,  stdinFd)        # my sys.stdin copy  = pipe1[0]
        os.dup2(parentStdout, stdoutFd)       # my sys.stdout copy = pipe2[1]
    else:
        os.close(parentStdin)                 # in child process after fork:
        os.close(parentStdout)                # close parent ends in child
        os.dup2(childStdin,  stdinFd)         # my sys.stdin copy  = pipe2[0]
        os.dup2(childStdout, stdoutFd)        # my sys.stdout copy = pipe1[1]
        args = (prog,) + args
        os.execvp(prog, args)                 # new program in this process
        assert False, 'execvp failed!'        # os.exec call never returns here

if __name__ == '__main__':
    mypid = os.getpid()
    spawn('python', 'pipes-testchild.py', 'spam')     # fork child program

    print('Hello 1 from parent', mypid)               # to child's stdin
    sys.stdout.flush()                                # subvert stdio buffering
    reply = input()                                   # from child's stdout
    sys.stderr.write('Parent got: "%s"
' % reply)    # stderr not tied to pipe!

    print('Hello 2 from parent', mypid)
    sys.stdout.flush()
    reply = sys.stdin.readline()
    sys.stderr.write('Parent got: "%s"
' % reply[:-1])

The spawn function in this module does not work on standard Windows Python (remember that fork isn’t yet available there today). In fact, most of the calls in this module map straight to Unix system calls (and may be arbitrarily terrifying at first glance to non-Unix developers!). We’ve already met some of these (e.g., os.fork), but much of this code depends on Unix concepts we don’t have time to address well in this text. But in simple terms, here is a brief summary of the system calls demonstrated in this code:

In terms of connecting standard streams, os.dup2 is the real nitty-gritty here. For example, the call os.dup2(parentStdin,stdinFd) essentially assigns the parent process’s stdin file to the input end of one of the two pipes created; all stdin reads will henceforth come from the pipe. By connecting the other end of this pipe to the child process’s copy of the stdout stream file with os.dup2(childStdout,stdoutFd), text written by the child to its sdtdout winds up being routed through the pipe to the parent’s stdin stream. The effect is reminiscent of the way we tied together streams with the subprocess module in Chapter 3, but this script is more low-level and less portable.

To test this utility, the self-test code at the end of the file spawns the program shown in Example 5-23 in a child process and reads and writes standard streams to converse with it over two pipes.

import os, time, sys
mypid     = os.getpid()
parentpid = os.getppid()
sys.stderr.write('Child %d of %d got arg: "%s"
' %
                                (mypid, parentpid, sys.argv[1]))
for i in range(2):
    time.sleep(3)              # make parent process wait by sleeping here
    recv = input()             # stdin tied to pipe: comes from parent's stdout
    time.sleep(3)
    send = 'Child %d got: [%s]' % (mypid, recv)
    print(send)                # stdout tied to pipe: goes to parent's stdin
    sys.stdout.flush()         # make sure it's sent now or else process blocks

The following is our test in action on Cygwin (it’s similar other Unix-like platforms like Linux); its output is not incredibly impressive to read, but it represents two programs running independently and shipping data back and forth through a pipe device managed by the operating system. This is even more like a client/server model (if you imagine the child as the server, responding to requests sent from the parent). The text in square brackets in this output went from the parent process to the child and back to the parent again, all through pipes connected to standard streams:

[C:...PP4ESystemProcesses]$ python pipes.py
Child 9228 of 9096 got arg: "spam"
Parent got: "Child 9228 got: [Hello 1 from parent 9096]"
Parent got: "Child 9228 got: [Hello 2 from parent 9096]"

The two processes of the prior section’s example engage in a simple dialog, but it’s already enough to illustrate some of the dangers lurking in cross-program communications. First of all, notice that both programs need to write to stderr to display a message; their stdout streams are tied to the other program’s input stream. Because processes share file descriptors, stderr is the same in both parent and child, so status messages show up in the same place.

More subtly, note that both parent and child call sys.stdout.flush after they print text to the output stream. Input requests on pipes normally block the caller if no data is available, but it seems that this shouldn’t be a problem in our example because there are as many writes as there are reads on the other side of the pipe. By default, though, sys.stdout is buffered in this context, so the printed text may not actually be transmitted until some time in the future (when the output buffers fill up). In fact, if the flush calls are not made, both processes may get stuck on some platforms waiting for input from the other—input that is sitting in a buffer and is never flushed out over the pipe. They wind up in a deadlock state, both blocked on input calls waiting for events that never occur.

Technically, by default stdout is just line-buffered when connected to a terminal, but it is fully buffered when connected to other devices such as files, sockets, and the pipes used here. This is why you see a script’s printed text in a shell window immediately as it is produced, but not until the process exits or its buffer fills when its output stream is connected to something else.

This output buffering is really a function of the system libraries used to access pipes, not of the pipes themselves (pipes do queue up output data, but they never hide it from readers!). In fact, it appears to occur in this example only because we copy the pipe’s information over to sys.stdout, a built-in file object that uses stream buffering by default. However, such anomalies can also occur when using other cross-process tools.

In general terms, if your programs engage in a two-way dialog like this, there are a variety of ways to avoid buffering-related deadlock problems:

Thread can avoid blocking a main GUI, too, but really just delegate the problem (the spawned thread will still be deadlocked). Of the options listed, the first two—manual flushes and command-line arguments—are often the simplest solutions. In fact, because it is so useful, the second technique listed above merits a few more words. Try this: comment-out all the sys.stdout.flush calls in Examples 5-22 and 5-23 (the files pipes.py and pipes-testchild.py) and change the parent’s spawn call in pipes.py to this (i.e., add a -u command-line argument):

spawn('python', '-u', 'pipes-testchild.py', 'spam')

Then start the program with a command line like this: python -u pipes.py. It will work as it did with the manual stdout flush calls, because stdout will be operating in unbuffered mode in both parent and child.

We’ll revisit the effects of unbuffered output streams in Chapter 10, where we’ll code a simple GUI that displays the output of a non-GUI program by reading it over both a nonblocking socket and a pipe in a thread. We’ll explore the topic again in more depth in Chapter 12, where we will redirect standard streams to sockets in more general ways. Deadlock in general, though, is a bigger problem than we have space to address fully here. On the other hand, if you know enough that you want to do IPC in Python, you’re probably already a veteran of the deadlock wars.

Anonymous pipes allow related tasks to communicate but are not directly suited for independently launched programs. To allow the latter group to converse, we need to move on to the next section and explore devices that have broader visibility.

In Python, named pipe files are created with the os.mkfifo call, which is available today on Unix-like platforms, including Cygwin’s Python on Windows, but is not currently available in standard Windows Python. This call creates only the external file, though; to send and receive data through a fifo, it must be opened and processed as if it were a standard file.

To illustrate, Example 5-24 is a derivation of the pipe2.py script listed in Example 5-20, but rewritten here to use fifos rather than anonymous pipes. Much like pipe2.py, this script opens the fifo using os.open in the child for low-level byte string access, but with the open built-in in the parent to treat the pipe as text; in general, either end may use either technique to treat the pipe’s data as bytes or text.

"""
named pipes; os.mkfifo is not available on Windows (without Cygwin);
there is no reason to fork here, since fifo file pipes are external
to processes--shared fds in parent/child processes are irrelevent;
"""

import os, time, sys
fifoname = '/tmp/pipefifo'                       # must open same name

def child():
    pipeout = os.open(fifoname, os.O_WRONLY)     # open fifo pipe file as fd
    zzz = 0
    while True:
        time.sleep(zzz)
        msg = ('Spam %03d
' % zzz).encode()     # binary as opened here
        os.write(pipeout, msg)
        zzz = (zzz+1) % 5

def parent():
    pipein = open(fifoname, 'r')                 # open fifo as text file object
    while True:
        line = pipein.readline()[:-1]            # blocks until data sent
        print('Parent %d got "%s" at %s' % (os.getpid(), line, time.time()))

if __name__ == '__main__':
    if not os.path.exists(fifoname):
        os.mkfifo(fifoname)                      # create a named pipe file
    if len(sys.argv) == 1:
        parent()                                 # run as parent if no args
    else:                                        # else run as child process
        child()

Because the fifo exists independently of both parent and child, there’s no reason to fork here. The child may be started independently of the parent as long as it opens a fifo file by the same name. Here, for instance, on Cygwin the parent is started in one shell window and then the child is started in another. Messages start appearing in the parent window only after the child is started and begins writing messages onto the fifo file:

[C:...PP4ESystemProcesses] $ python pipefifo.py           # parent window
Parent 8324 got "Spam 000" at 1268003696.07
Parent 8324 got "Spam 001" at 1268003697.06
Parent 8324 got "Spam 002" at 1268003699.07
Parent 8324 got "Spam 003" at 1268003702.08
Parent 8324 got "Spam 004" at 1268003706.09
Parent 8324 got "Spam 000" at 1268003706.09
Parent 8324 got "Spam 001" at 1268003707.11
Parent 8324 got "Spam 002" at 1268003709.12
Parent 8324 got "Spam 003" at 1268003712.13
Parent 8324 got "Spam 004" at 1268003716.14
Parent 8324 got "Spam 000" at 1268003716.14
Parent 8324 got "Spam 001" at 1268003717.15
...etc: Ctrl-C to exit...

[C:...PP4ESystemProcesses]$ file /tmp/pipefifo            # child window
/tmp/pipefifo: fifo (named pipe)

[C:...PP4ESystemProcesses]$ python pipefifo.py -child
...Ctrl-C to exit...

By mapping communication points to a file system entity accessible to all programs run on a machine, fifos can address a broad range of IPC goals on platforms where they are supported. For instance, although this section’s example runs independent programs, named pipes can also be used as an IPC device by both in-process threads and directly forked related processes, much as we saw for anonymous pipes earlier.

By also supporting unrelated programs, though, fifo files are more widely applicable to general client/server models. For example, named pipes can make the GUI and command-line debugger integration I described earlier for anonymous pipes even more flexible—by using fifo files to connect the GUI to the non-GUI debugger’s streams, the GUI could be started independently when needed.

Sockets provide similar functionality but also buy us both inherent network awareness and broader portability to Windows—as the next section explains.

Although sockets are one of the most commonly used IPC tools, it’s impossible to fully grasp their API without also seeing its role in networking. Because of that, we’ll defer most of our socket coverage until we can explore their use in network scripting in Chapter 12. This section provides a brief introduction and preview, so you can compare with the prior section’s named pipes (a.k.a. fifos). In short:

In addition, sockets support networking roles that go beyond both IPC and this chapter’s scope. To illustrate the basics, though, Example 5-25 launches a server and 5 clients in threads running in parallel on the same machine, to communicate over a socket—because all threads connect to the same port, the server consumes the data added by each of the clients.

"""
sockets for cross-task communication: start threads to communicate over sockets;
independent programs can too, because sockets are system-wide, much like fifos;
see the GUI and Internet parts of the book for more realistic socket use cases;
some socket servers may also need to talk to clients in threads or processes;
sockets pass byte strings, but can be pickled objects or encoded Unicode text;
caveat: prints in threads may need to be synchronized if their output overlaps;
"""

from socket import socket, AF_INET, SOCK_STREAM     # portable socket api

port = 50008                 # port number identifies socket on machine
host = 'localhost'           # server and client run on same local machine here

def server():
    sock = socket(AF_INET, SOCK_STREAM)         # ip addresses tcp connection
    sock.bind(('', port))                       # bind to port on this machine
    sock.listen(5)                              # allow up to 5 pending clients
    while True:
        conn, addr = sock.accept()              # wait for client to connect
        data = conn.recv(1024)                  # read bytes data from this client
        reply = 'server got: [%s]' % data       # conn is a new connected socket
        conn.send(reply.encode())               # send bytes reply back to client

def client(name):
    sock = socket(AF_INET, SOCK_STREAM)
    sock.connect((host, port))                  # connect to a socket port
    sock.send(name.encode())                    # send bytes data to listener
    reply = sock.recv(1024)                     # receive bytes data from listener
    sock.close()                                # up to 1024 bytes in message
    print('client got: [%s]' % reply)

if __name__ == '__main__':
    from threading import Thread
    sthread = Thread(target=server)
    sthread.daemon = True                       # don't wait for server thread
    sthread.start()                             # do wait for children to exit
    for i in range(5):
         Thread(target=client, args=('client%s' % i,)).start()

Study this script’s code and comments to see how the socket objects’ methods are used to transfer data. In a nutshell, with this type of socket the server accepts a client connection, which by default blocks until a client requests service, and returns a new socket connected to the client. Once connected, the client and server transfer byte strings by using send and receive calls instead of writes and reads, though as we’ll see later in the book, sockets can be wrapped in file objects much as we did earlier for pipe descriptors. Also like pipe descriptors, unwrapped sockets deal in binary bytes strings, not text str; that’s why string formatting results are manually encoded again here.

Here is this script’s output on Windows:

C:...PP4ESystemProcesses> socket_preview.py
client got: [b"server got: [b'client1']"]
client got: [b"server got: [b'client3']"]
client got: [b"server got: [b'client4']"]
client got: [b"server got: [b'client2']"]
client got: [b"server got: [b'client0']"]

This output isn’t much to look at, but each line reflects data sent from client to server, and then back again: the server receives a bytes string from a connected client and echoes it back in a larger reply string. Because all threads run in parallel, the order in which the clients are served is random on this machine.

Although sockets work for threads, the shared memory model of threads often allows them to employ simpler communication devices such as shared names and objects and queues. Sockets tend to shine brighter when used for IPC by separate processes and independently launched programs. Example 5-26, for instance, reuses the server and client functions of the prior example, but runs them in both processes and threads of independently launched programs.

"""
same socket, but talk between independent programs too, not just threads;
server here runs in a process and serves both process and thread clients;
sockets are machine-global, much like fifos: don't require shared memory
"""

from socket_preview import server, client         # both use same port number
import sys, os
from threading import Thread

mode = int(sys.argv[1])
if mode == 1:                                     # run server in this process
    server()
elif mode == 2:                                   # run client in this process
    client('client:process=%s' % os.getpid())
else:                                             # run 5 client threads in process
    for i in range(5):
        Thread(target=client, args=('client:thread=%s' % i,)).start()

Let’s run this script on Windows, too (again, this portability is a major advantage of sockets). First, start the server in a process as an independently launched program in its own window; this process runs perpetually waiting for clients to request connections (and as for our prior pipe example you may need to use Task Manager or a window close to kill the server process eventually):

C:...PP4ESystemProcesses> socket-preview-progs.py 1

Now, in another window, run a few clients in both processes and thread, by launching them as independent programs—using 2 as the command-line argument runs a single client process, but 3 spawns five threads to converse with the server on parallel:

C:...PP4ESystemProcesses> socket-preview-progs.py 2
client got: [b"server got: [b'client:process=7384']"]

C:...PP4ESystemProcesses> socket-preview-progs.py 2
client got: [b"server got: [b'client:process=7604']"]

C:...PP4ESystemProcesses> socket-preview-progs.py 3
client got: [b"server got: [b'client:thread=1']"]
client got: [b"server got: [b'client:thread=2']"]
client got: [b"server got: [b'client:thread=0']"]
client got: [b"server got: [b'client:thread=3']"]
client got: [b"server got: [b'client:thread=4']"]

C:..PP4ESystemProcesses> socket-preview-progs.py 3
client got: [b"server got: [b'client:thread=3']"]
client got: [b"server got: [b'client:thread=1']"]
client got: [b"server got: [b'client:thread=2']"]
client got: [b"server got: [b'client:thread=4']"]
client got: [b"server got: [b'client:thread=0']"]

C:...PP4ESystemProcesses> socket-preview-progs.py 2
client got: [b"server got: [b'client:process=6428']"]

This section’s examples illustrate the basic IPC role of sockets, but this only hints at their full utility. Despite their seemingly limited byte string nature, higher-order use cases for sockets are not difficult to imagine. With a little extra work, for instance:

Plus any other context in which programs exchange data—sockets are a general, portable, and flexible tool. For instance, they would provide the same utility as fifos for the GUI/debugger example used earlier, but would also work in Python on Windows and would even allow the GUI to connect to a debugger running on a different computer altogether. As such, they are seen by many as a more powerful IPC tool.

Again, you should consider this section just a preview; because the grander socket story also entails networking concepts, we’ll defer a more in-depth look at the socket API until Chapter 12. We’ll also see sockets again briefly in Chapter 10 in the GUI stream redirection use case listed above, and we’ll explore a variety of additional socket use cases in the Internet part of this book. In Part IV, for instance, we’ll use sockets to transfer entire files and write more robust socket servers that spawn threads or processes to converse with clients to avoid denying connections. For the purposes of this chapter, let’s move on to one last traditional IPC tool—the signal.

Technically, to achieve its portability, this module currently works by selecting from platform-specific alternatives:

We met pickling briefly in Chapter 1, and we will study it further later in this book. The implementation is a bit more complex than this, and is prone to change over time, of course, but it’s really quite an amazing trick. While the portable API generally hides these details from your code, its basic structure can still have subtle impacts on the way you’re allowed to use it. For instance:

There are additional rules documented in the library manual. In general, though, if you stick to passing in shared objects to processes and using the synchronization and communication tools provided by this package, your code will usually be portable and correct. Let’s look next at a few of those tools in action.

To demonstrate the IPC tools listed above, the next three examples implement three flavors of communication between parent and child processes. Example 5-30 uses a simple shared pipe object to send and receive data between parent and child processes.

"""
Use multiprocess anonymous pipes to communicate. Returns 2 connection
object representing ends of the pipe: objects are sent on one end and
received on the other, though pipes are bidirectional by default
"""

import os
from multiprocessing import Process, Pipe

def sender(pipe):
    """
    send object to parent on anonymous pipe
    """
    pipe.send(['spam'] +  [42, 'eggs'])
    pipe.close()

def talker(pipe):
    """
    send and receive objects on a pipe
    """
    pipe.send(dict(name='Bob', spam=42))
    reply = pipe.recv()
    print('talker got:', reply)

if __name__ == '__main__':
    (parentEnd, childEnd) = Pipe()
    Process(target=sender, args=(childEnd,)).start()        # spawn child with pipe
    print('parent got:', parentEnd.recv())                  # receive from child
    parentEnd.close()                                       # or auto-closed on gc

    (parentEnd, childEnd) = Pipe()
    child = Process(target=talker, args=(childEnd,))
    child.start()
    print('parent got:', parentEnd.recv())                  # receieve from child
    parentEnd.send({x * 2 for x in 'spam'})                 # send to child
    child.join()                                            # wait for child exit
    print('parent exit')

When run on Windows, here’s this script’s output—one child passes an object to the parent, and the other both sends and receives on the same pipe:

C:...PP4ESystemProcesses> multi2.py
parent got: ['spam', 42, 'eggs']
parent got: {'name': 'Bob', 'spam': 42}
talker got: {'ss', 'aa', 'pp', 'mm'}
parent exit

This module’s pipe objects make communication between two processes portable (and nearly trivial).

Example 5-31 uses shared memory to serve as both inputs and outputs of spawned processes. To make this work portably, we must create objects defined by the package and pass them to Process constructors. The last test in this demo (“loop4”) probably represents the most common use case for shared memory—that of distributing computation work to multiple parallel processes.

"""
Use multiprocess shared memory objects to communicate.
Passed objects are shared, but globals are not on Windows.
Last test here reflects common use case: distributing work.
"""

import os
from multiprocessing import Process, Value, Array

procs = 3
count = 0    # per-process globals, not shared

def showdata(label, val, arr):
    """
    print data values in this process
    """
    msg = '%-12s: pid:%4s, global:%s, value:%s, array:%s'
    print(msg % (label, os.getpid(), count, val.value, list(arr)))

def updater(val, arr):
    """
    communicate via shared memory
    """
    global count
    count += 1                         # global count not shared
    val.value += 1                     # passed in objects are
    for i in range(3): arr[i] += 1

if __name__ == '__main__':
    scalar = Value('i', 0)             # shared memory: process/thread safe
    vector = Array('d', procs)         # type codes from ctypes: int, double

    # show start value in parent process
    showdata('parent start', scalar, vector)

    # spawn child, pass in shared memory
    p = Process(target=showdata, args=('child ', scalar, vector))
    p.start(); p.join()

    # pass in shared memory updated in parent, wait for each to finish
    # each child sees updates in parent so far for args (but not global)

    print('
loop1 (updates in parent, serial children)...')
    for i in range(procs):
        count += 1
        scalar.value += 1
        vector[i] += 1
        p = Process(target=showdata, args=(('process %s' % i), scalar, vector))
        p.start(); p.join()

    # same as prior, but allow children to run in parallel
    # all see the last iteration's result because all share objects

    print('
loop2 (updates in parent, parallel children)...')
    ps = []
    for i in range(procs):
        count += 1
        scalar.value += 1
        vector[i] += 1
        p = Process(target=showdata, args=(('process %s' % i), scalar, vector))
        p.start()
        ps.append(p)
    for p in ps: p.join()

    # shared memory updated in spawned children, wait for each

    print('
loop3 (updates in serial children)...')
    for i in range(procs):
        p = Process(target=updater, args=(scalar, vector))
        p.start()
        p.join()
    showdata('parent temp', scalar, vector)

    # same, but allow children to update in parallel

    ps = []
    print('
loop4 (updates in parallel children)...')
    for i in range(procs):
        p = Process(target=updater, args=(scalar, vector))
        p.start()
        ps.append(p)
    for p in ps: p.join()
                                           # global count=6 in parent only
    # show final results here              # scalar=12:  +6 parent, +6 in 6 children
    showdata('parent end', scalar, vector) # array[i]=8: +2 parent, +6 in 6 children

The following is this script’s output on Windows. Trace through this and the code to see how it runs; notice how the changed value of the global variable is not shared by the spawned processes on Windows, but passed-in Value and Array objects are. The final output line reflects changes made to shared memory in both the parent and spawned children—the array’s final values are all 8.0, because they were incremented twice in the parent, and once in each of six spawned children; the scalar value similarly reflects changes made by both parent and child; but unlike for threads, the global is per-process data on Windows:

C:...PP4ESystemProcesses> multi3.py
parent start: pid:6204, global:0, value:0, array:[0.0, 0.0, 0.0]
child       : pid:9660, global:0, value:0, array:[0.0, 0.0, 0.0]

loop1 (updates in parent, serial children)...
process 0   : pid:3900, global:0, value:1, array:[1.0, 0.0, 0.0]
process 1   : pid:5072, global:0, value:2, array:[1.0, 1.0, 0.0]
process 2   : pid:9472, global:0, value:3, array:[1.0, 1.0, 1.0]

loop2 (updates in parent, parallel children)...
process 1   : pid:9468, global:0, value:6, array:[2.0, 2.0, 2.0]
process 2   : pid:9036, global:0, value:6, array:[2.0, 2.0, 2.0]
process 0   : pid:9548, global:0, value:6, array:[2.0, 2.0, 2.0]

loop3 (updates in serial children)...
parent temp : pid:6204, global:6, value:9, array:[5.0, 5.0, 5.0]

loop4 (updates in parallel children)...
parent end  : pid:6204, global:6, value:12, array:[8.0, 8.0, 8.0]

If you imagine the last test here run with a much larger array and many more parallel children, you might begin to sense some of the power of this package for distributing work.

Finally, besides basic spawning and IPC tools, the multiprocessing module also:

Queues support a more flexible multiple client/server model. Example 5-32, for instance, spawns three producer threads to post to a shared queue and repeatedly polls for results to appear—in much the same fashion that a GUI might collect results in parallel with the display itself, though here the concurrency is achieved with processes instead of threads.

"""
Process class can also be subclassed just like threading.Thread;
Queue works like queue.Queue but for cross-process, not cross-thread
"""

import os, time, queue
from multiprocessing import Process, Queue           # process-safe shared queue
                                                     # queue is a pipe + locks/semas
class Counter(Process):
    label = '  @'
    def __init__(self, start, queue):                # retain state for use in run
        self.state = start
        self.post  = queue
        Process.__init__(self)

    def run(self):                                   # run in newprocess on start()
        for i in range(3):
            time.sleep(1)
            self.state += 1
            print(self.label ,self.pid, self.state)  # self.pid is this child's pid
            self.post.put([self.pid, self.state])    # stdout file is shared by all
        print(self.label, self.pid, '-')

if __name__ == '__main__':
    print('start', os.getpid())
    expected = 9

    post = Queue()
    p = Counter(0, post)                        # start 3 processes sharing queue
    q = Counter(100, post)                      # children are producers
    r = Counter(1000, post)
    p.start(); q.start(); r.start()

    while expected:                             # parent consumes data on queue
        time.sleep(0.5)                         # this is essentially like a GUI,
        try:                                    # though GUIs often use threads
            data = post.get(block=False)
        except queue.Empty:
            print('no data...')
        else:
            print('posted:', data)
            expected -= 1

    p.join(); q.join(); r.join()                # must get before join putter
    print('finish', os.getpid(), r.exitcode)    # exitcode is child exit status

Notice in this code how:

When run, the output of the main consumer process traces its queue fetches, and the (indented) output of spawned child producer processes gives process IDs and state.

C:...PP4ESystemProcesses> multi4.py
start 6296
no data...
no data...
  @ 8008 101
posted: [8008, 101]
  @ 6068 1
  @ 3760 1001
posted: [6068, 1]
  @ 8008 102
posted: [3760, 1001]
  @ 6068 2
  @ 3760 1002
posted: [8008, 102]
  @ 8008 103
  @ 8008 -
posted: [6068, 2]
  @ 6068 3
  @ 6068 -
  @ 3760 1003
  @ 3760 -
posted: [3760, 1002]
posted: [8008, 103]
posted: [6068, 3]
posted: [3760, 1003]
finish 6296 0

If you imagine the “@” lines here as results of long-running operations and the others as a main GUI thread, the wide relevance of this package may become more apparent.

To be fair, besides such additional features and tools, multiprocessing also comes with additional constraints beyond those we’ve already covered (pickleability, mutable state, and so on). For example, consider the following sort of code:

def action(arg1, arg2):
    print(arg1, arg2)

if __name__ == '__main__':
    Process(target=action, args=('spam', 'eggs')).start()    # shell waits for child

This works as expected, but if we change the last line to the following it fails on Windows because lambdas are not pickleable (really, not importable):

Process(target=(lambda: action('spam', 'eggs'))).start()  # fails!-not pickleable

This precludes a common coding pattern that uses lambda to add data to calls, which we’ll use often for callbacks in the GUI part of this book. Moreover, this differs from the threading module that is the model for this package—calls like the following which work for threads must be translated to a callable and arguments:

threading.Thread(target=(lambda: action(2, 4))).start()   # but lambdas work here

Conversely, some behavior of the threading module is mimicked by multiprocessing, whether you wish it did or not. Because programs using this package wait for child processes to end by default, we must mark processes as daemon if we don’t want to block the shell where the following sort of code is run (technically, parents attempt to terminate daemonic children on exit, which means that the program can exit when only daemonic children remain, much like threading):

def action(arg1, arg2):
   print(arg1, arg2)
   time.sleep(5)          # normally prevents the parent from exiting

if __name__ == '__main__':
    p = Process(target=action, args=('spam', 'eggs'))
    p.daemon = True                                        # don't wait for it
    p.start()

There’s more on some of these issues in the Python library manual; they are not show-stoppers by any stretch, but special cases and potential pitfalls to some. We’ll revisit the lambda and daemon issues in a more realistic context in Chapter 8, where we’ll use multiprocessing to launch GUI demos independently.

To understand why, first you need to know how the DOS start command works in general. Roughly, a DOS command line of the form start command works as if command were typed in the Windows Run dialog box available in the Start button menu. If command is a filename, it is opened exactly as if its name was double-clicked in the Windows Explorer file selector GUI.

For instance, the following three DOS commands automatically start Internet Explorer, my registered image viewer program, and my sound media player program on the files named in the commands. Windows simply opens the file with whatever program is associated to handle filenames of that form. Moreover, all three of these programs run independently of the DOS console box where the command is typed:

C:...PP4ESystemMedia> start lp4e-preface-preview.html
C:...PP4ESystemMedia> start ora-lp4e.jpg
C:...PP4ESystemMedia> start sousa.au

Because the start command can run any file and command line, there is no reason it cannot also be used to start an independently running Python program:

C:...PP4ESystemProcesses> start child.py 1

This works because Python is registered to open names ending in .py when it is installed. The script child.py is launched independently of the DOS console window even though we didn’t provide the name or path of the Python interpreter program. Because child.py simply prints a message and exits, though, the result isn’t exactly satisfying: a new DOS window pops up to serve as the script’s standard output, and it immediately goes away when the child exits. To do better, add an input call at the bottom of the program file to wait for a key press before exiting:

C:...PP4ESystemProcesses> type child-wait.py
import os, sys
print('Hello from child', os.getpid(), sys.argv[1])
input("Press <Enter>")       # don't flash on Windows

C:...PP4ESystemProcesses> start child-wait.py 2

Now the child’s DOS window pops up and stays up after the start command has returned. Pressing the Enter key in the pop-up DOS window makes it go away.

Since we know that Python’s os.system and os.popen can be called by a script to run any command line that can be typed at a DOS shell prompt, we can also start independently running programs from a Python script by simply running a DOS start command line. For instance:

C:...PP4ESystemMedia> python
>>> import os
>>> cmd = 'start lp4e-preface-preview.html'            # start IE browser
>>> os.system(cmd)                                     # runs independent
0

The Python os.system calls here start whatever web page browser is registered on your machine to open .html files (unless these programs are already running). The launched programs run completely independent of the Python session—when running a DOS start command, os.system does not wait for the spawned program to exit.

In fact, start is so useful that recent Python releases also include an os.startfile call, which is essentially the same as spawning a DOS start command with os.system and works as though the named file were double-clicked. The following calls, for instance, have a similar effect:

>>> os.startfile('lp-code-readme.txt')
>>> os.system('start lp-code-readme.txt')

Both pop up the text file in Notepad on my Windows computer. Unlike the second of these calls, though, os.startfile provides no option to wait for the application to close (the DOS start command’s /WAIT option does) and no way to retrieve the application’s exit status (returned from os.system).

On recent versions of Windows, the following has a similar effect, too, because the registry is used at the command line (though this form pauses until the file’s viewer is closed—like using start /WAIT):

>>> os.system('lp-code-readme.txt')       # 'start' is optional today

This is a convenient way to open arbitrary document and media files, but keep in mind that the os.startfile call works only on Windows, because it uses the Windows registry to know how to open a file. In fact, there are even more obscure and nonportable ways to launch programs, including Windows-specific options in the PyWin32 package, which we’ll finesse here. If you want to be more platform neutral, consider using one of the other many program launcher tools we’ve seen, such as os.popen or os.spawnv. Or better yet, write a module to hide the details—as the next and final section demonstrates.