In conceptual terms, the Internet can roughly be thought of as being composed of multiple functional layers:

This book covers the middle three tiers in this list—sockets, the Internet protocols that run on them, and the CGI model of web-based conversations. What we learn here will also apply to more specific toolkits in the last tier above, because they are all ultimately based upon the same Internet and web fundamentals.

More specifically, in this and the next chapter, our main focus is on programming the second and third layers: sockets and higher-level Internet protocols. We’ll start this chapter at the bottom, learning about the socket model of network programming. Sockets aren’t strictly tied to Internet scripting, as we saw in Chapter 5’s IPC examples, but they are presented in full here because this is one of their primary roles. As we’ll see, most of what happens on the Internet happens through sockets, whether you notice or not.

After introducing sockets, the next two chapters make their way up to Python’s client-side interfaces to higher-level protocols—things like email and FTP transfers, which run on top of sockets. It turns out that a lot can be done with Python on the client alone, and Chapters 13 and 14 will sample the flavor of Python client-side scripting. Finally, the last two chapters in this part of the book then move on to present server-side scripting—programs that run on a server computer and are usually invoked by a web browser.

Now that I’ve told you what we will cover in this book, I also want to be clear about what we won’t cover. Like tkinter, the Internet is a vast topic, and this part of the book is mostly an introduction to its core concepts and an exploration of representative tasks. Because there are so many Internet-related modules and extensions, this book does not attempt to serve as an exhaustive survey of the domain. Even in just Python’s own tool set, there are simply too many Internet modules to include each in this text in any sort of useful fashion.

Moreover, higher-level tools like Django, Jython, and App Engine are very large systems in their own right, and they are best dealt with in more focused documents. Because dedicated books on such topics are now available, we’ll merely scratch their surfaces here with a brief survey later in this chapter. This book also says almost nothing about lower-level networking layers such as TCP/IP. If you’re curious about what happens on the Internet at the bit-and-wire level, consult a good networking text for more details.

In other words, this part is not meant to be an exhaustive reference to Internet and web programming with Python—a topic which has evolved between prior editions of this book, and will undoubtedly continue to do so after this one is published. Instead, the goal of this part of the book is to serve as a tutorial introduction to the domain to help you get started, and to provide context and examples which will help you understand the documentation for tools you may wish to explore after mastering the fundamentals here.

Like the prior parts of the book, this one has other agendas, too. Along the way, this part will also put to work many of the operating-system and GUI interfaces we studied in Parts II and III (e.g., processes, threads, signals, and tkinter). We’ll also get to see the Python language applied in realistically scaled programs, and we’ll investigate some of the design choices and challenges that the Internet presents.

That last statement merits a few more words. Internet scripting, like GUIs, is one of the “sexier” application domains for Python. As in GUI work, there is an intangible but instant gratification in seeing a Python Internet program ship information all over the world. On the other hand, by its very nature, network programming can impose speed overheads and user interface limitations. Though it may not be a fashionable stance these days, some applications are still better off not being deployed on the Web.

A traditional “desktop” GUI like those of Part III, for example, can combine the feature-richness and responsiveness of client-side libraries with the power of network access. On the other hand, web-based applications offer compelling benefits in portability and administration. In this part of the book, we will take an honest look at the Net’s trade-offs as they arise and explore examples which illustrate the advantages of both web and nonweb architectures. In fact, the larger PyMailGUI and PyMailCGI examples we’ll explore are intended in part to serve this purpose.

The Internet is also considered by many to be something of an ultimate proof of concept for open source tools. Indeed, much of the Net runs on top of a large number of such tools, such as Python, Perl, the Apache web server, the sendmail program, MySQL, and Linux.^[42] Moreover, new tools and technologies for programming the Web sometimes seem to appear faster than developers can absorb them.

The good news is that Python’s integration focus makes it a natural in such a heterogeneous world. Today, Python programs can be installed as client-side and server-side tools; used as applets and servlets in Java applications; mixed into distributed object systems like CORBA, SOAP, and XML-RPC; integrated into AJAX-based applications; and much more. In more general terms, the rationale for using Python in the Internet domain is exactly the same as in any other—Python’s emphasis on quality, productivity, portability, and integration makes it ideal for writing Internet programs that are open, maintainable, and delivered according to the ever-shrinking schedules in this field.

Suppose for just a moment that you wish to have a telephone conversation with someone halfway across the world. In the real world, you would probably need either that person’s telephone number or a directory that you could use to look up the number from her name (e.g., a telephone book). The same is true on the Internet: before a script can have a conversation with another computer somewhere in cyberspace, it must first know that other computer’s number or name.

Luckily, the Internet defines standard ways to name both a remote machine and a service provided by that machine. Within a script, the computer program to be contacted through a socket is identified by supplying a pair of values—the machine name and a specific port number on that machine:

The combination of a machine name and a port number uniquely identifies every dialog on the Net. For instance, an ISP’s computer may provide many kinds of services for customers—web pages, Telnet, FTP transfers, email, and so on. Each service on the machine is assigned a unique port number to which requests may be sent. To get web pages from a web server, programs need to specify both the web server’s Internet Protocol (IP) or domain name and the port number on which the server listens for web page requests.

If this sounds a bit strange, it may help to think of it in old-fashioned terms. To have a telephone conversation with someone within a company, for example, you usually need to dial both the company’s phone number and the extension of the person you want to reach. If you don’t know the company’s number, you can probably find it by looking up the company’s name in a phone book. It’s almost the same on the Net—machine names identify a collection of services (like a company), port numbers identify an individual service within a particular machine (like an extension), and domain names are mapped to IP numbers by domain name servers (like a phone book).

When programs use sockets to communicate in specialized ways with another machine (or with other processes on the same machine), they need to avoid using a port number reserved by a standard protocol—numbers in the range of 0 to 1023—but we first need to discuss protocols to understand why.

Technically speaking, socket port numbers can be any 16-bit integer value between 0 and 65,535. However, to make it easier for programs to locate the standard protocols, port numbers in the range of 0 to 1023 are reserved and preassigned to the standard higher-level protocols. Table 12-1 lists the ports reserved for many of the standard protocols; each gets one or more preassigned numbers from the reserved range.

To socket programmers, the standard protocols mean that port numbers 0 to 1023 are off-limits to scripts, unless they really mean to use one of the higher-level protocols. This is both by standard and by common sense. A Telnet program, for instance, can start a dialog with any Telnet-capable machine by connecting to its port, 23; without preassigned port numbers, each server might install Telnet on a different port. Similarly, websites listen for page requests from browsers on port 80 by standard; if they did not, you might have to know and type the HTTP port number of every site you visit while surfing the Net.

By defining standard port numbers for services, the Net naturally gives rise to a client/server architecture. On one side of a conversation, machines that support standard protocols perpetually run a set of programs that listen for connection requests on the reserved ports. On the other end of a dialog, other machines contact those programs to use the services they export.

We usually call the perpetually running listener program a server and the connecting program a client. Let’s use the familiar web browsing model as an example. As shown in Table 12-1, the HTTP protocol used by the Web allows clients and servers to talk over sockets on port 80:

In general, many clients may connect to a server over sockets, whether it implements a standard protocol or something more specific to a given application. And in some applications, the notion of client and server is blurred—programs can also pass bytes between each other more as peers than as master and subordinate. An agent in a peer-to-peer file transfer system, for instance, may at various times be both client and server for parts of files transferred.

For the purposes of this book, though, we usually call programs that listen on sockets servers, and those that connect clients. We also sometimes call the machines that these programs run on server and client (e.g., a computer on which a web server program runs may be called a web server machine, too), but this has more to do with the physical than the functional.

Functionally, protocols may accomplish a familiar task, like reading email or posting a Usenet newsgroup message, but they ultimately consist of message bytes sent over sockets. The structure of those message bytes varies from protocol to protocol, is hidden by the Python library, and is mostly beyond the scope of this book, but a few general words may help demystify the protocol layer.

Some protocols may define the contents of messages sent over sockets; others may specify the sequence of control messages exchanged during conversations. By defining regular patterns of communication, protocols make communication more robust. They can also minimize deadlock conditions—machines waiting for messages that never arrive.

For example, the FTP protocol prevents deadlock by conversing over two sockets: one for control messages only and one to transfer file data. An FTP server listens for control messages (e.g., “send me a file”) on one port, and transfers file data over another. FTP clients open socket connections to the server machine’s control port, send requests, and send or receive file data over a socket connected to a data port on the server machine. FTP also defines standard message structures passed between client and server. The control message used to request a file, for instance, must follow a standard format.

Before we see these programs in action, let’s take a minute to explain how this client and server do their stuff. Both are fairly simple examples of socket scripts, but they illustrate the common call patterns of most socket-based programs. In fact, this is boilerplate code: most connected socket programs generally make the same socket calls that our two scripts do, so let’s step through the important points of these scripts line by line.

Programs such as Example 12-1 that provide services for other programs with sockets generally start out by following this sequence of calls:

At this point, the server is ready to accept connection requests from client programs running on remote machines (or the same machine) and falls into an infinite loop—while True (or the equivalent while 1 for older Pythons and ex-C programmers)—waiting for them to arrive:

Once we have a client connection, we fall into another loop to receive data from the client in blocks of up to 1,024 bytes at a time, and echo each block back to the client:

So far we’ve seen calls used to transfer data in a server, but what is it that is actually shipped through a socket? As we learned in Chapter 5, sockets by themselves always deal in binary byte strings, not text. To your scripts, this means you must send and will receive bytes strings, not str, though you can convert to and from text as needed with bytes.decode and str.encode methods. In our scripts, we use b'...' bytes literals to satisfy socket data requirements. In other contexts, tools such as the struct and pickle modules return the byte strings we need automatically, so no extra steps are needed.

For example, although the socket model is limited to transferring byte strings, you can send and receive nearly arbitrary Python objects with the standard library pickle object serialization module. Its dumps and loads calls convert Python objects to and from byte strings, ready for direct socket transfer:

>>> import pickle
>>> x = pickle.dumps([99, 100])        # on sending end... convert to byte strings

>>> x                                  # string passed to send, returned by recv
b'x80x03]qx00(KcKde.'

>>> pickle.loads(x)                    # on receiving end... convert back to object
[99, 100]

For simpler types that correspond to those in the C language, the struct module provides the byte-string conversion we need as well:

>>> import struct
>>> x = struct.pack('>ii', 99, 100)    # convert simpler types for transmission
>>> x
b'x00x00x00cx00x00x00d'
>>> struct.unpack('>ii', x)
(99, 100)

When converted this way, Python native objects become candidates for socket-based transfers. See Chapter 4 for more on struct. We previewed pickle and object serialization in Chapter 1, but we’ll learn more about it and its few pickleability constraints when we explore data persistence in Chapter 17.

In fact there are a variety of ways to extend the basic socket transfer model. For instance, much like os.fdopen and open for the file descriptors we studied in Chapter 4, the socket.makefile method allows you to wrap sockets in text-mode file objects that handle text encodings for you automatically. This call also allows you to specify nondefault Unicode encodings and end-line behaviors in text mode with extra arguments in 3.X just like the open built-in function. Because its result mimics file interfaces, the socket.makefile call additionally allows the pickle module’s file-based calls to transfer objects over sockets implicitly. We’ll see more on socket file wrappers later in this chapter.

For our simpler scripts here, hardcoded byte strings and direct socket calls do the job. After talking with a given connected client, the server in Example 12-1 goes back to its infinite loop and waits for the next client connection request. Let’s move on to see what happened on the other side of the fence.

The actual socket-related calls in client programs like the one shown in Example 12-2 are even simpler; in fact, half of that script is preparation logic. The main thing to keep in mind is that the client and server must specify the same port number when opening their sockets and the client must identify the machine on which the server is running; in our scripts, server and client agree to use port number 50007 for their conversation, outside the standard protocol range. Here are the client’s socket calls:

Once the client establishes a connection to the server, it falls into a loop, sending a message one line at a time and printing whatever the server sends back after each line is sent:

And that’s it. The server exchanges one or more lines of text with each client that connects. The operating system takes care of locating remote machines, routing bytes sent between programs and possibly across the Internet, and (with TCP) making sure that our messages arrive intact. That involves a lot of processing, too—our strings may ultimately travel around the world, crossing phone wires, satellite links, and more along the way. But we can be happily ignorant of what goes on beneath the socket call layer when programming in Python.

Before we move on, there are three practical usage details you should know. First, you can run the client and server like this on any two Internet-aware machines where Python is installed. Of course, to run the client and server on different computers, you need both a live Internet connection and access to another machine on which to run the server.

This need not be an expensive proposition, though; when sockets are opened, Python is happy to initiate and use whatever connectivity you have, be it a dedicated T1 line, wireless router, cable modem, or dial-up account. Moreover, if you don’t have a server account of your own like the one I’m using on learning-python.com, simply run client and server examples on the same machine, localhost, as shown earlier; all you need then is a computer that allows sockets, and most do.

Second, the socket module generally raises exceptions if you ask for something invalid. For instance, trying to connect to a nonexistent server (or unreachable servers, if you have no Internet link) fails:

C:...PP4EInternetSockets> python echo-client.py www.nonesuch.com hello
Traceback (most recent call last):
  File "echo-client.py", line 24, in <module>
    sockobj.connect((serverHost, serverPort))   # connect to server machine...
socket.error: [Errno 10060] A connection attempt failed because the connected
party did not properly respond after a period of time, or established connection
failed because connected host has failed to respond

Finally, also be sure to kill the server process before restarting it again, or else the port number will still be in use, and you’ll get another exception; on my remote server machine:

[...]$ ps -x
  PID TTY      STAT   TIME COMMAND
 5378 pts/0    S      0:00 python echo-server.py
22017 pts/0    Ss     0:00 -bash
26805 pts/0    R+     0:00 ps –x

[...]$ python echo-server.py
Traceback (most recent call last):
  File "echo-server.py", line 14, in <module>
    sockobj.bind((myHost, myPort))               # bind it to server port number
socket.error: [Errno 10048] Only one usage of each socket address (protocol/
network address/port) is normally permitted

A series of Ctrl-Cs will kill the server on Linux (be sure to type fg to bring it to the foreground first if started with an &):

[...]$ fg
python echo-server.py
Traceback (most recent call last):
  File "echo-server.py", line 18, in <module>
    connection, address = sockobj.accept()   # wait for next client connect
KeyboardInterrupt

As mentioned earlier, a Ctrl-C kill key combination won’t kill the server on my Windows 7 machine, however. To kill the perpetually running server process running locally on Windows, you may need to start Task Manager (e.g., using a Ctrl-Alt-Delete key combination), and then end the Python task by selecting it in the process listbox that appears. Closing the window in which the server is running will also suffice on Windows, but you’ll lose that window’s command history. You can also usually kill a server on Linux with a kill −9 pid shell command if it is running in another window or in the background, but Ctrl-C requires less typing.

The net effect is that our echo server converses with multiple clients, whether running locally or remotely. Keep in mind, however, that this works for our simple scripts only because the server doesn’t take a long time to respond to each client’s requests—it can get back to the top of the server script’s outer while loop in time to process the next incoming client. If it could not, we would probably need to change the server to handle each client in parallel, or some might be denied a connection.

Technically, client connections would fail after 5 clients are already waiting for the server’s attention, as specified in the server’s listen call. To prove this to yourself, add a time.sleep call somewhere inside the echo server’s main loop in Example 12-1 after a connection is accepted, to simulate a long-running task (this is from file echo-server-sleep.py in the examples package if you wish to experiment):

while True:                                  # listen until process killed
    connection, address = sockobj.accept()   # wait for next client connect
    while True:
        data = connection.recv(1024)         # read next line on client socket
        time.sleep(3)                        # take time to process request
        ...

If you then run this server and the testecho clients script, you’ll notice that not all 8 clients wind up receiving a connection, because the server is too busy to empty its pending-connections queue in time. Only 6 clients are served when I run this on Windows—one accepted initially, and 5 in the pending-requests listen queue. The other two clients are denied connections and fail.

The following shows the server and client messages produced when the server is stalled this way, including the error messages that the two denied clients receive. To see the clients’ messages on Windows, you can change testecho to use the StartArgs launcher with a /B switch at the front of the command line to route messages to the persistent console window (see file testecho-messages.py in the examples package):

C:...PP4EdevExamplesPP4EInternetSockets> echo-server-sleep.py
Server connected by ('127.0.0.1', 59625)
Server connected by ('127.0.0.1', 59626)
Server connected by ('127.0.0.1', 59627)
Server connected by ('127.0.0.1', 59628)
Server connected by ('127.0.0.1', 59629)
Server connected by ('127.0.0.1', 59630)

C:...PP4EdevExamplesPP4EInternetSockets> testecho-messages.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
/B echo-client.py
Client received: b'Echo=>Hello network world'

Traceback (most recent call last):
  File "C:...PP4EInternetSocketsecho-client.py", line 24, in <module>
    sockobj.connect((serverHost, serverPort))   # connect to server machine...
socket.error: [Errno 10061] No connection could be made because the target
machine actively refused it

Traceback (most recent call last):
  File "C:...PP4EInternetSocketsecho-client.py", line 24, in <module>
    sockobj.connect((serverHost, serverPort))   # connect to server machine...
socket.error: [Errno 10061] No connection could be made because the target
machine actively refused it

Client received: b'Echo=>Hello network world'
Client received: b'Echo=>Hello network world'
Client received: b'Echo=>Hello network world'
Client received: b'Echo=>Hello network world'
Client received: b'Echo=>Hello network world'

As you can see, with such a sleepy server, 8 clients are spawned, but only 6 receive service, and 2 fail with exceptions. Unless clients require very little of the server’s attention, to handle multiple requests overlapping in time we need to somehow service clients in parallel. We’ll see how servers can handle multiple clients more robustly in a moment; first, though, let’s experiment with some special ports.

Speaking of reserved ports, it’s all right to open client-side connections on reserved ports as in the prior section, but you can’t install your own server-side scripts for these ports unless you have special permission. On the server I use to host learning-python.com, for instance, the web server port 80 is off limits (presumably, unless I shell out for a virtual or dedicated hosting account):

[...]$ python
>>> from socket import *
>>> sock = socket(AF_INET, SOCK_STREAM)    # try to bind web port on general server
>>> sock.bind(('', 80))                    # learning-python.com is a shared machine
Traceback (most recent call last):
  File "<stdin>", line 1, in
  File "<string>", line 1, in bind
socket.error: (13, 'Permission denied')

Even if run by a user with the required permission, you’ll get the different exception we saw earlier if the port is already being used by a real web server. On computers being used as general servers, these ports really are reserved. This is one reason we’ll run a web server of our own locally for testing when we start writing server-side scripts later in this book—the above code works on a Windows PC, which allows us to experiment with websites locally, on a self-contained machine:

C:...PP4EInternetSockets> python
>>> from socket import *
>>> sock = socket(AF_INET, SOCK_STREAM)      # can bind port 80 on Windows
>>> sock.bind(('', 80))                      # allows running server on localhost
>>>

We’ll learn more about installing web servers later in Chapter 15. For the purposes of this chapter, we need to get realistic about how our socket servers handle their clients.

Parts of this script are a bit tricky, and most of its library calls work only on Unix-like platforms. Crucially, it runs on Cygwin Python on Windows, but not standard Windows Python. Before we get into too many forking details, though, let’s focus on how this server arranges to handle multiple client requests.

First, notice that to simulate a long-running operation (e.g., database updates, other network traffic), this server adds a five-second time.sleep delay in its client handler function, handleClient. After the delay, the original echo reply action is performed. That means that when we run a server and clients this time, clients won’t receive the echo reply until five seconds after they’ve sent their requests to the server.

To help keep track of requests and replies, the server prints its system time each time a client connect request is received, and adds its system time to the reply. Clients print the reply time sent back from the server, not their own—clocks on the server and client may differ radically, so to compare apples to apples, all times are server times. Because of the simulated delays, we also must usually start each client in its own console window on Windows (clients will hang in a blocked state while waiting for their reply).

But the grander story here is that this script runs one main parent process on the server machine, which does nothing but watch for connections (in dispatcher), plus one child process per active client connection, running in parallel with both the main parent process and the other client processes (in handleClient). In principle, the server can handle any number of clients without bogging down.

To test, let’s first start the server remotely in a SSH or Telnet window, and start three clients locally in three distinct console windows. As we’ll see in a moment, this server can also be run under Cygwin locally if you have Cygwin but don’t have a remote server account like the one on learning-python.com used here:

[server window (SSH or Telnet)]
[...]$ uname -p -o
i686 GNU/Linux
[...]$ python fork-server.py
Server connected by ('72.236.109.185', 58395) at Sat Apr 24 06:46:45 2010
Server connected by ('72.236.109.185', 58396) at Sat Apr 24 06:46:49 2010
Server connected by ('72.236.109.185', 58397) at Sat Apr 24 06:46:51 2010

[client window 1]
C:...PP4EInternetSockets> python echo-client.py learning-python.com
Client received: b"Echo=>b'Hello network world' at Sat Apr 24 06:46:50 2010"

[client window 2]
C:...PP4EInternetSockets> python echo-client.py learning-python.com Bruce
Client received: b"Echo=>b'Bruce' at Sat Apr 24 06:46:54 2010"

[client window 3]
C:...Sockets> python echo-client.py learning-python.com The Meaning of Life
Client received: b"Echo=>b'The' at Sat Apr 24 06:46:56 2010"
Client received: b"Echo=>b'Meaning' at Sat Apr 24 06:46:56 2010"
Client received: b"Echo=>b'of' at Sat Apr 24 06:46:56 2010"
Client received: b"Echo=>b'Life' at Sat Apr 24 06:46:57 2010"

Again, all times here are on the server machine. This may be a little confusing because four windows are involved. In plain English, the test proceeds as follows:

In other words, clients are serviced at the same time by forked processes, while the main parent process continues listening for new client requests. If clients were not handled in parallel like this, no client could connect until the currently connected client’s five-second delay expired.

In a more realistic application, that delay could be fatal if many clients were trying to connect at once—the server would be stuck in the action we’re simulating with time.sleep, and not get back to the main loop to accept new client requests. With process forks per request, clients can be serviced in parallel.

Notice that we’re using the same client script here (echo-client.py, from Example 12-2), just a different server; clients simply send and receive data to a machine and port and don’t care how their requests are handled on the server. The result displayed shows a byte string within a byte string, because the client sends one to the server and the server sends one back; because the server uses string formatting and manual encoding instead of byte string concatenation, the client’s message is shown as byte string explicitly here.

Also note that the server is running remotely on a Linux machine in the preceding section. As we learned in Chapter 5, the fork call is not supported on Windows in standard Python at the time this book was written. It does run on Cygwin Python, though, which allows us to start this server locally on localhost, on the same machine as its clients:

[Cygwin shell window]
[C:...PP4EInternetSocekts]$ python fork-server.py
Server connected by ('127.0.0.1', 58258) at Sat Apr 24 07:50:15 2010
Server connected by ('127.0.0.1', 58259) at Sat Apr 24 07:50:17 2010

[Windows console, same machine]
C:...PP4EInternetSockets> python echo-client.py localhost bright side of life
Client received: b"Echo=>b'bright' at Sat Apr 24 07:50:20 2010"
Client received: b"Echo=>b'side' at Sat Apr 24 07:50:20 2010"
Client received: b"Echo=>b'of' at Sat Apr 24 07:50:20 2010"
Client received: b"Echo=>b'life' at Sat Apr 24 07:50:20 2010"

[Windows console, same machine]
C:...PP4EInternetSockets> python echo-client.py
Client received: b"Echo=>b'Hello network world' at Sat Apr 24 07:50:22 2010"

We can also run this test on the remote Linux server entirely, with two SSH or Telnet windows. It works about the same as when clients are started locally, in a DOS console window, but here “local” actually means a remote machine you’re using locally. Just for fun, let’s also contact the remote server from a locally running client to show how the server is also available to the Internet at large—when servers are coded with sockets and forks this way, clients can connect from arbitrary machines, and can overlap arbitrarily in time:

[one SSH (or Telnet) window]
[...]$ python fork-server.py
Server connected by ('127.0.0.1', 55743) at Sat Apr 24 07:15:14 2010
Server connected by ('127.0.0.1', 55854) at Sat Apr 24 07:15:26 2010
Server connected by ('127.0.0.1', 55950) at Sat Apr 24 07:15:36 2010
Server connected by ('72.236.109.185', 58414) at Sat Apr 24 07:19:50 2010

[another SSH window, same machine]
[...]$ python echo-client.py
Client received: b"Echo=>b'Hello network world' at Sat Apr 24 07:15:19 2010"
[...]$ python echo-client.py localhost niNiNI!
Client received: b"Echo=>b'niNiNI!' at Sat Apr 24 07:15:31 2010"
[...]$ python echo-client.py localhost Say no more!
Client received: b"Echo=>b'Say' at Sat Apr 24 07:15:41 2010"
Client received: b"Echo=>b'no' at Sat Apr 24 07:15:41 2010"
Client received: b"Echo=>b'more!' at Sat Apr 24 07:15:41 2010"

[Windows console, local machine]
C:...InternetSockets> python echo-client.py learning-python.com Blue, no yellow!
Client received: b"Echo=>b'Blue,' at Sat Apr 24 07:19:55 2010"
Client received: b"Echo=>b'no' at Sat Apr 24 07:19:55 2010"
Client received: b"Echo=>b'yellow!' at Sat Apr 24 07:19:55 2010"

Now that we have a handle on the basic model, let’s move on to the tricky bits. This server script is fairly straightforward as forking code goes, but a few words about the library tools it employs are in order.

We met os.fork in Chapter 5, but recall that forked processes are essentially a copy of the process that forks them, and so they inherit file and socket descriptors from their parent process. As a result, the new child process that runs the handleClient function has access to the connection socket created in the parent process. Really, this is why the child process works at all—when conversing on the connected socket, it’s using the same socket that parent’s accept call returns. Programs know they are in a forked child process if the fork call returns 0; otherwise, the original parent process gets back the new child’s ID.

In earlier fork examples, child processes usually call one of the exec variants to start a new program in the child process. Here, instead, the child process simply calls a function in the same program and exits with os._exit. It’s imperative to call os._exit here—if we did not, each child would live on after handleClient returns, and compete for accepting new client requests.

In fact, without the exit call, we’d wind up with as many perpetual server processes as requests served—remove the exit call and do a ps shell command after running a few clients, and you’ll see what I mean. With the call, only the single parent process listens for new requests. os._exit is like sys.exit, but it exits the calling process immediately without cleanup actions. It’s normally used only in child processes, and sys.exit is used everywhere else.

Note, however, that it’s not quite enough to make sure that child processes exit and die. On systems like Linux, though not on Cygwin, parents must also be sure to issue a wait system call to remove the entries for dead child processes from the system’s process table. If we don’t do this, the child processes will no longer run, but they will consume an entry in the system process table. For long-running servers, these bogus entries may become problematic.

It’s common to call such dead-but-listed child processes zombies: they continue to use system resources even though they’ve already passed over to the great operating system beyond. To clean up after child processes are gone, this server keeps a list, activeChildren, of the process IDs of all child processes it spawns. Whenever a new incoming client request is received, the server runs its reapChildren to issue a wait for any dead children by issuing the standard Python os.waitpid(0,os.WNOHANG) call.

The os.waitpid call attempts to wait for a child process to exit and returns its process ID and exit status. With a 0 for its first argument, it waits for any child process. With the WNOHANG parameter for its second, it does nothing if no child process has exited (i.e., it does not block or pause the caller). The net effect is that this call simply asks the operating system for the process ID of any child that has exited. If any have, the process ID returned is removed both from the system process table and from this script’s activeChildren list.

To see why all this complexity is needed, comment out the reapChildren call in this script, run it on a platform where this is an issue, and then run a few clients. On my Linux server, a ps -f full process listing command shows that all the dead child processes stay in the system process table (show as <defunct>):

[...]$ ps –f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094   9990 30778  0 04:34 pts/0    00:00:00 python fork-server.py
5693094  10844  9990  0 04:35 pts/0    00:00:00 [python] <defunct>
5693094  10869  9990  0 04:35 pts/0    00:00:00 [python] <defunct>
5693094  11130  9990  0 04:36 pts/0    00:00:00 [python] <defunct>
5693094  11151  9990  0 04:36 pts/0    00:00:00 [python] <defunct>
5693094  11482 30778  0 04:36 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

When the reapChildren command is reactivated, dead child zombie entries are cleaned up each time the server gets a new client connection request, by calling the Python os.waitpid function. A few zombies may accumulate if the server is heavily loaded, but they will remain only until the next client connection is received (you get only as many zombies as processes served in parallel since the last accept):

[...]$ python fork-server.py &
[1] 20515
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  20515 30778  0 04:43 pts/0    00:00:00 python fork-server.py
5693094  20777 30778  0 04:43 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash
[...]$
Server connected by ('72.236.109.185', 58672) at Sun Apr 25 04:43:51 2010
Server connected by ('72.236.109.185', 58673) at Sun Apr 25 04:43:54 2010
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  20515 30778  0 04:43 pts/0    00:00:00 python fork-server.py
5693094  21339 20515  0 04:43 pts/0    00:00:00 [python] <defunct>
5693094  21398 20515  0 04:43 pts/0    00:00:00 [python] <defunct>
5693094  21573 30778  0 04:44 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash
[...]$
Server connected by ('72.236.109.185', 58674) at Sun Apr 25 04:44:07 2010
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  20515 30778  0 04:43 pts/0    00:00:00 python fork-server.py
5693094  21646 20515  0 04:44 pts/0    00:00:00 [python] <defunct>
5693094  21813 30778  0 04:44 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

In fact, if you type fast enough, you can actually see a child process morph from a real running program into a zombie. Here, for example, a child spawned to handle a new request changes to <defunct> on exit. Its connection cleans up lingering zombies, and its own process entry will be removed completely when the next request is received:

[...]$
Server connected by ('72.236.109.185', 58676) at Sun Apr 25 04:48:22 2010
[...] ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  20515 30778  0 04:43 pts/0    00:00:00 python fork-server.py
5693094  27120 20515  0 04:48 pts/0    00:00:00 python fork-server.py
5693094  27174 30778  0 04:48 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  20515 30778  0 04:43 pts/0    00:00:00 python fork-server.py
5693094  27120 20515  0 04:48 pts/0    00:00:00 [python] <defunct>
5693094  27234 30778  0 04:48 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

On some systems, it’s also possible to clean up zombie child processes by resetting the signal handler for the SIGCHLD signal delivered to a parent process by the operating system when a child process stops or exits. If a Python script assigns the SIG_IGN (ignore) action as the SIGCHLD signal handler, zombies will be removed automatically and immediately by the operating system as child processes exit; the parent need not issue wait calls to clean up after them. Because of that, this scheme is a simpler alternative to manually reaping zombies on platforms where it is supported.

If you’ve already read Chapter 5, you know that Python’s standard signal module lets scripts install handlers for signals—software-generated events. By way of review, here is a brief bit of background to show how this pans out for zombies. The program in Example 12-5 installs a Python-coded signal handler function to respond to whatever signal number you type on the command line.

"""
Demo Python's signal module; pass signal number as a command-line arg, and use
a "kill -N pid" shell command to send this process a signal; on my Linux machine,
SIGUSR1=10, SIGUSR2=12, SIGCHLD=17, and SIGCHLD handler stays in effect even if
not restored: all other handlers are restored by Python after caught, but SIGCHLD
behavior is left to the platform's implementation; signal works on Windows too,
but defines only a few signal types; signals are not very portable in general;
"""

import sys, signal, time

def now():
    return time.asctime()

def onSignal(signum, stackframe):                # Python signal handler
    print('Got signal', signum, 'at', now())     # most handlers stay in effect
    if signum == signal.SIGCHLD:                 # but sigchld handler is not
        print('sigchld caught')
        #signal.signal(signal.SIGCHLD, onSignal)

signum = int(sys.argv[1])
signal.signal(signum, onSignal)                  # install signal handler
while True: signal.pause()                       # sleep waiting for signals

To run this script, simply put it in the background and send it signals by typing the kill -signal-number process-id shell command line; this is the shell’s equivalent of Python’s os.kill function available on Unix-like platforms only. Process IDs are listed in the PID column of ps command results. Here is this script in action catching signal numbers 10 (reserved for general use) and 9 (the unavoidable terminate signal):

[...]$ python signal-demo.py 10 &
[1] 10141
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  10141 30778  0 05:00 pts/0    00:00:00 python signal-demo.py 10
5693094  10228 30778  0 05:00 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

[...]$ kill −10 10141
Got signal 10 at Sun Apr 25 05:00:31 2010

[...]$ kill −10 10141
Got signal 10 at Sun Apr 25 05:00:34 2010

[...]$ kill −9 10141
[1]+  Killed                  python signal-demo.py 10

And in the following the script catches signal 17, which happens to be SIGCHLD on my Linux server. Signal numbers vary from machine to machine, so you should normally use their names, not their numbers. SIGCHLD behavior may vary per platform as well. On my Cygwin install, for example, signal 10 can have different meaning, and signal 20 is SIGCHLD—on Cygwin, the script works as shown on Linux here for signal 10, but generates an exception if it tries to install on handler for signal 17 (and Cygwin doesn’t require reaping in any event). See the signal module’s library manual entry for more details:

[...]$ python signal-demo.py 17 &
[1] 11592
[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094  11592 30778  0 05:00 pts/0    00:00:00 python signal-demo.py 17
5693094  11728 30778  0 05:01 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

[...]$ kill −17 11592
Got signal 17 at Sun Apr 25 05:01:28 2010
sigchld caught

[...]$ kill −17 11592
Got signal 17 at Sun Apr 25 05:01:35 2010
sigchld caught

[...]$ kill −9 11592
[1]+  Killed                  python signal-demo.py 17

Now, to apply all of this signal knowledge to killing zombies, simply set the SIGCHLD signal handler to the SIG_IGN ignore handler action; on systems where this assignment is supported, child processes will be cleaned up when they exit. The forking server variant shown in Example 12-6 uses this trick to manage its children.

"""
Same as fork-server.py, but use the Python signal module to avoid keeping
child zombie processes after they terminate, instead of an explicit reaper
loop before each new connection; SIG_IGN means ignore, and may not work with
SIG_CHLD child exit signal on all platforms; see Linux documentation for more
about the restartability of a socket.accept call interrupted with a signal;
"""

import os, time, sys, signal, signal
from socket import *                      # get socket constructor and constants
myHost = ''                               # server machine, '' means local host
myPort = 50007                            # listen on a non-reserved port number

sockobj = socket(AF_INET, SOCK_STREAM)           # make a TCP socket object
sockobj.bind((myHost, myPort))                   # bind it to server port number
sockobj.listen(5)                                # up to 5 pending connects
signal.signal(signal.SIGCHLD, signal.SIG_IGN)    # avoid child zombie processes

def now():                                       # time on server machine
    return time.ctime(time.time())

def handleClient(connection):                    # child process replies, exits
    time.sleep(5)                                # simulate a blocking activity
    while True:                                  # read, write a client socket
        data = connection.recv(1024)
        if not data: break
        reply = 'Echo=>%s at %s' % (data, now())
        connection.send(reply.encode())
    connection.close()
    os._exit(0)

def dispatcher():                                # listen until process killed
    while True:                                  # wait for next connection,
        connection, address = sockobj.accept()   # pass to process for service
        print('Server connected by', address, end=' ')
        print('at', now())
        childPid = os.fork()                     # copy this process
        if childPid == 0:                        # if in child process: handle
            handleClient(connection)             # else: go accept next connect

dispatcher()

Where applicable, this technique is:

In fact, only one line is dedicated to handling zombies here: the signal.signal call near the top, to set the handler. Unfortunately, this version is also even less portable than using os.fork in the first place, because signals may work slightly differently from platform to platform, even among Unix variants. For instance, some Unix platforms may not allow SIG_IGN to be used as the SIGCHLD action at all. On Linux systems, though, this simpler forking server variant works like a charm:

[...]$ python fork-server-signal.py &
[1] 3837
Server connected by ('72.236.109.185', 58817) at Sun Apr 25 08:11:12 2010

[...] ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094   3837 30778  0 08:10 pts/0    00:00:00 python fork-server-signal.py
5693094   4378  3837  0 08:11 pts/0    00:00:00 python fork-server-signal.py
5693094   4413 30778  0 08:11 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094   3837 30778  0 08:10 pts/0    00:00:00 python fork-server-signal.py
5693094   4584 30778  0 08:11 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 –bash

Notice how in this version the child process’s entry goes away as soon as it exits, even before a new client request is received; no “defunct” zombie ever appears. More dramatically, if we now start up the script we wrote earlier that spawns eight clients in parallel (testecho.py) to talk to this server remotely, all appear on the server while running, but are removed immediately as they exit:

[client window]
C:...PP4EInternetSockets> testecho.py learning-python.com

[server window]
[...]$
Server connected by ('72.236.109.185', 58829) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58830) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58831) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58832) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58833) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58834) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58835) at Sun Apr 25 08:16:34 2010
Server connected by ('72.236.109.185', 58836) at Sun Apr 25 08:16:34 2010

[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094   3837 30778  0 08:10 pts/0    00:00:00 python fork-server-signal.py
5693094   9666  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9667  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9668  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9670  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9674  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9678  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9681  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9682  3837  0 08:16 pts/0    00:00:00 python fork-server-signal.py
5693094   9722 30778  0 08:16 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 -bash

[...]$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
5693094   3837 30778  0 08:10 pts/0    00:00:00 python fork-server-signal.py
5693094  10045 30778  0 08:16 pts/0    00:00:00 ps -f
5693094  30778 30772  0 04:23 pts/0    00:00:00 –bash

And now that I’ve shown you how to use signal handling to reap children automatically on Linux, I should underscore that this technique is not universally supported across all flavors of Unix. If you care about portability, manually reaping children as we did in Example 12-4 may still be desirable.

In Chapter 5, we learned about Python’s new multiprocessing module. As we saw, it provides a way to start function calls in new processes that is more portable than the os.fork call used in this section’s server code, and it runs processes instead of threads to work around the thread GIL in some scenarios. In particular, multiprocessing works on standard Windows Python too, unlike direct os.fork calls.

I experimented with a server variant based upon this module to see if its portability might help for socket servers. Its full source code is in the examples package in file multi-server.py, but here are its important bits that differ:

...rest unchanged from fork-server.py...
from multiprocessing import Process

def handleClient(connection):
    print('Child:', os.getpid())                 # child process: reply, exit
    time.sleep(5)                                # simulate a blocking activity
    while True:                                  # read, write a client socket
        data = connection.recv(1024)             # till eof when socket closed
        ...rest unchanged...

def dispatcher():                                # listen until process killed
    while True:                                  # wait for next connection,
        connection, address = sockobj.accept()   # pass to process for service
        print('Server connected by', address, end=' ')
        print('at', now())
        Process(target=handleClient, args=(connection,)).start()

if __name__ == '__main__':
    print('Parent:', os.getpid())
    sockobj = socket(AF_INET, SOCK_STREAM)           # make a TCP socket object
    sockobj.bind((myHost, myPort))                   # bind it to server port number
    sockobj.listen(5)                                # allow 5 pending connects
    dispatcher()

This server variant is noticeably simpler too. Like the forking server it’s derived from, this server works fine under Cygwin Python on Windows running as localhost, and would probably work on other Unix-like platforms as well, because multiprocessing forks a process on such systems, and file and socket descriptors are inherited by child processes as usual. Hence, the child process uses the same connected socket as the parent. Here’s the scene in a Cygwin server window and two Windows client windows:

[server window]
[C:...PP4EInternetSockets]$ python multi-server.py
Parent: 8388
Server connected by ('127.0.0.1', 58271) at Sat Apr 24 08:13:27 2010
Child: 8144
Server connected by ('127.0.0.1', 58272) at Sat Apr 24 08:13:29 2010
Child: 8036

[two client windows]
C:...PP4EInternetSockets> python echo-client.py
Client received: b"Echo=>b'Hello network world' at Sat Apr 24 08:13:33 2010"

C:...PP4EInternetSockets> python echo-client.py localhost Brave Sir Robin
Client received: b"Echo=>b'Brave' at Sat Apr 24 08:13:35 2010"
Client received: b"Echo=>b'Sir' at Sat Apr 24 08:13:35 2010"
Client received: b"Echo=>b'Robin' at Sat Apr 24 08:13:35 2010"

However, this server does not work on standard Windows Python—the whole point of trying to use multiprocessing in this context—because open sockets are not correctly pickled when passed as arguments into the new process. Here’s what occurs in the server windows on Windows 7 with Python 3.1:

C:...PP4EInternetSockets> python multi-server.py
Parent: 9140
Server connected by ('127.0.0.1', 58276) at Sat Apr 24 08:17:41 2010
Child: 9628
Process Process-1:
Traceback (most recent call last):
  File "C:Python31libmultiprocessingprocess.py", line 233, in _bootstrap
    self.run()
  File "C:Python31libmultiprocessingprocess.py", line 88, in run
    self._target(*self._args, **self._kwargs)
  File "C:...PP4EInternetSocketsmulti-server.py", line 38, in handleClient
    data = connection.recv(1024)             # till eof when socket closed
socket.error: [Errno 10038] An operation was attempted on something that is not
a socket

Recall from Chapter 5 that on Windows multiprocessing passes context to a new Python interpreter process by pickling it, and that Process arguments must all be pickleable for Windows. Sockets in Python 3.1 don’t trigger errors when pickled thanks to the class they are an instance of, but they are not really pickled correctly:

>>> from pickle import *
>>> from socket import *
>>> s = socket()
>>> x = dumps(s)
>>> s
<socket.socket object, fd=180, family=2, type=1, proto=0>
>>> loads(x)
<socket.socket object, fd=-1, family=0, type=0, proto=0>
>>> x
b'x80x03csocket
socket
qx00)x81qx01N}qx02(Xx08x00x00x00_io_refsqx03
Kx00Xx07x00x00x00_closedqx04x89ux86qx05b.'

As we saw in Chapter 5, multiprocessing has other IPC tools such as its own pipes and queues that might be used instead of sockets to work around this issue, but clients would then have to use them, too—the resulting server would not be as broadly accessible as one based upon general Internet sockets.

Even if multiprocessing did work on Windows, though, its need to start a new Python interpreter would likely make it much slower than the more traditional technique of spawning threads to talk to clients. Coincidentally, that brings us to our next topic.

Let’s see how all of this translates into code. The script in Example 12-9 implements another echo server, one that can handle multiple clients without ever starting new processes or threads.

"""
Server: handle multiple clients in parallel with select. use the select
module to manually multiplex among a set of sockets: main sockets which
accept new client connections, and input sockets connected to accepted
clients; select can take an optional 4th arg--0 to poll, n.m to wait n.m
seconds, or omitted to wait till any socket is ready for processing.
"""

import sys, time
from select import select
from socket import socket, AF_INET, SOCK_STREAM
def now(): return time.ctime(time.time())

myHost = ''                             # server machine, '' means local host
myPort = 50007                          # listen on a non-reserved port number
if len(sys.argv) == 3:                  # allow host/port as cmdline args too
    myHost, myPort = sys.argv[1:]
numPortSocks = 2                        # number of ports for client connects

# make main sockets for accepting new client requests
mainsocks, readsocks, writesocks = [], [], []
for i in range(numPortSocks):
    portsock = socket(AF_INET, SOCK_STREAM)   # make a TCP/IP socket object
    portsock.bind((myHost, myPort))           # bind it to server port number
    portsock.listen(5)                        # listen, allow 5 pending connects
    mainsocks.append(portsock)                # add to main list to identify
    readsocks.append(portsock)                # add to select inputs list
    myPort += 1                               # bind on consecutive ports

# event loop: listen and multiplex until server process killed
print('select-server loop starting')
while True:
    #print(readsocks)
    readables, writeables, exceptions = select(readsocks, writesocks, [])
    for sockobj in readables:
        if sockobj in mainsocks:                     # for ready input sockets
            # port socket: accept new client
            newsock, address = sockobj.accept()      # accept should not block
            print('Connect:', address, id(newsock))  # newsock is a new socket
            readsocks.append(newsock)                # add to select list, wait
        else:
            # client socket: read next line
            data = sockobj.recv(1024)                # recv should not block
            print('	got', data, 'on', id(sockobj))
            if not data:                             # if closed by the clients
                sockobj.close()                      # close here and remv from
                readsocks.remove(sockobj)            # del list else reselected
            else:
                # this may block: should really select for writes too
                reply = 'Echo=>%s at %s' % (data, now())
                sockobj.send(reply.encode())

The bulk of this script is its while event loop at the end that calls select to find out which sockets are ready for processing; these include both main port sockets on which clients can connect and open client connections. It then loops over all such ready sockets, accepting connections on main port sockets and reading and echoing input on any client sockets ready for input. Both the accept and recv calls in this code are guaranteed to not block the server process after select returns; as a result, this server can quickly get back to the top of the loop to process newly arrived client requests and already connected clients’ inputs. The net effect is that all new requests and clients are serviced in pseudoparallel fashion.

To make this process work, the server appends the connected socket for each client to the readables list passed to select, and simply waits for the socket to show up in the selected inputs list. For illustration purposes, this server also listens for new clients on more than one port—on ports 50007 and 50008, in our examples. Because these main port sockets are also interrogated with select, connection requests on either port can be accepted without blocking either already connected clients or new connection requests appearing on the other port. The select call returns whatever sockets in readables are ready for processing—both main port sockets and sockets connected to clients currently being processed.

Let’s run this script locally to see how it does its stuff (the client and server can also be run on different machines, as in prior socket examples). First, we’ll assume we’ve already started this server script on the local machine in one window, and run a few clients to talk to it. The following listing gives the interaction in two such client console windows running on Windows. The first client simply runs the echo-client script twice to contact the server, and the second also kicks off the testecho script to spawn eight echo-client programs running in parallel.

As before, the server simply echoes back whatever text that client sends, though without a sleep pause here (more on this in a moment). Notice how the second client window really runs a script called echo-client-50008 so as to connect to the second port socket in the server; it’s the same as echo-client, with a different hardcoded port number; alas, the original script wasn’t designed to input a port number:

[client window 1]
C:...PP4EInternetSockets> python echo-client.py
Client received: b"Echo=>b'Hello network world' at Sun Apr 25 14:51:21 2010"

C:...PP4EInternetSockets> python echo-client.py
Client received: b"Echo=>b'Hello network world' at Sun Apr 25 14:51:27 2010"

[client window 2]
C:...PP4EInternetSockets> python echo-client-5008.py localhost Sir Galahad
Client received: b"Echo=>b'Sir' at Sun Apr 25 14:51:22 2010"
Client received: b"Echo=>b'Galahad' at Sun Apr 25 14:51:22 2010"

C:...PP4EInternetSockets> python testecho.py

The next listing is the sort of output that show up in the window where the server has been started. The first three connections come from echo-client runs; the rest is the result of the eight programs spawned by testecho in the second client window. We can run this server on Windows, too, because select is available on this platform. Correlate this output with the server’s code to see how it runs.

Notice that for testecho, new client connections and client inputs are multiplexed together. If you study the output closely, you’ll see that they overlap in time, because all activity is dispatched by the single event loop in the server. In fact, the trace output on the server will probably look a bit different nearly every time it runs. Clients and new connections are interleaved almost at random due to timing differences on the host machines. This happens in the earlier forking and treading servers, too, but the operating system automatically switches between the execution paths of the dispatcher loop and client transactions.

Also note that the server gets an empty string when the client has closed its socket. We take care to close and delete these sockets at the server right away, or else they would be needlessly reselected again and again, each time through the main loop:

[server window]
C:...PP4EInternetSockets> python select-server.py
C:UsersmarkStuffBooks4EPP4EdevExamplesPP4EInternetSockets>python sele
ct-server.py
select-server loop starting
Connect: ('127.0.0.1', 59080) 21339352
        got b'Hello network world' on 21339352
        got b'' on 21339352
Connect: ('127.0.0.1', 59081) 21338128
        got b'Sir' on 21338128
        got b'Galahad' on 21338128
        got b'' on 21338128
Connect: ('127.0.0.1', 59082) 21339352
        got b'Hello network world' on 21339352
        got b'' on 21339352

[testecho results]
Connect: ('127.0.0.1', 59083) 21338128
        got b'Hello network world' on 21338128
        got b'' on 21338128
Connect: ('127.0.0.1', 59084) 21339352
        got b'Hello network world' on 21339352
        got b'' on 21339352
Connect: ('127.0.0.1', 59085) 21338128
        got b'Hello network world' on 21338128
        got b'' on 21338128
Connect: ('127.0.0.1', 59086) 21339352
        got b'Hello network world' on 21339352
        got b'' on 21339352
Connect: ('127.0.0.1', 59087) 21338128
        got b'Hello network world' on 21338128
        got b'' on 21338128
Connect: ('127.0.0.1', 59088) 21339352
Connect: ('127.0.0.1', 59089) 21338128
        got b'Hello network world' on 21339352
        got b'Hello network world' on 21338128
Connect: ('127.0.0.1', 59090) 21338056
        got b'' on 21339352
        got b'' on 21338128
        got b'Hello network world' on 21338056
        got b'' on 21338056

Besides this more verbose output, there’s another subtle but crucial difference to notice—a time.sleep call to simulate a long-running task doesn’t make sense in the server here. Because all clients are handled by the same single loop, sleeping would pause everything, and defeat the whole point of a multiplexing server. Again, manual multiplexing servers like this one work well when transactions are short, but also generally require them to either be so, or be handled specially.

Before we move on, here are a few additional notes and options:

Before we move on, there are two remarkably subtle aspects of the example’s code worth highlighting:

To make some of this more concrete, Example 12-12 illustrates how some of these complexities apply to redirected standard streams, by attempting to connect them to both text and binary mode files produced by open and accessing them with print and input built-ins much as redirected script might.

"""
test effect of connecting standard streams to text and binary mode files
same holds true for socket.makefile: print requires text mode, but text
mode precludes unbuffered mode -- use -u or sys.stdout.flush() calls
"""

import sys

def reader(F):
    tmp, sys.stdin = sys.stdin, F
    line = input()
    print(line)
    sys.stdin = tmp

reader( open('test-stream-modes.py') )         # works: input() returns text
reader( open('test-stream-modes.py', 'rb') )   # works: but input() returns bytes

def writer(F):
    tmp, sys.stdout = sys.stdout, F
    print(99, 'spam')
    sys.stdout = tmp

writer( open('temp', 'w') )             # works: print() passes text str to .write()
print(open('temp').read())

writer( open('temp', 'wb') )            # FAILS on print: binary mode requires bytes
writer( open('temp', 'w', 0) )          # FAILS on open: text must be buffered

When run, the last two lines in this script both fail—the second to last fails because print passes text strings to a binary-mode file (never allowed for files in general), and the last fails because we cannot open text-mode files in unbuffered mode in Python 3.X (text mode implies Unicode encodings). Here are the errors we get when this script is run: the first run uses the script as shown, and the second shows what happens if the second to last line is commented out (I edited the exception text slightly for presentation):

C:...PP4EInternetSockets> test-stream-modes.py
"""
b'"""
'
99 spam

Traceback (most recent call last):
  File "C:...PP4EInternetSockets	est-stream-modes.py", line 26, in <module>
    writer( open('temp', 'wb') )            # FAILS on print: binary mode...
  File "C:...PP4EInternetSockets	est-stream-modes.py", line 20, in writer
    print(99, 'spam')
TypeError: must be bytes or buffer, not str

C:...PP4EInternetSockets> test-streams-binary.py
"""
b'"""
'
99 spam

Traceback (most recent call last):
  File "C:...PP4EInternetSockets	est-stream-modes.py", line 27, in <module>
    writer( open('temp', 'w', 0) )          # FAILS on open: text must be...
ValueError: can't have unbuffered text I/O

The same rules apply to socket wrapper file objects created with a socket’s makefile method—they must be opened in text mode for print and should be opened in text mode for input if we wish to receive text strings, but text mode prevents us from using fully unbuffered file mode altogether:

>>> from socket import *
>>> s = socket()                       # defaults to tcp/ip (AF_INET, SOCK_STREAM)
>>> s.makefile('w', 0)                 # this used to work in Python 2.X
Traceback (most recent call last):
  File "C:Python31libsocket.py", line 151, in makefile
ValueError: unbuffered streams must be binary

Text-mode socket wrappers also accept a buffering-mode argument of 1 to specify line-buffering instead of the default full buffering:

>>> from socket import *
>>> s = socket()
>>> f = s.makefile('w', 1)    # same as buffering=1, but acts as fully buffered!

This appears to be no different than full buffering, and still requires the resulting file to be flushed manually to transfer lines as they are produced. Consider the simple socket server and client scripts in Examples 12-13 and 12-14. The server simply reads three messages using the raw socket interface.

from socket import *           # read three messages over a raw socket
sock = socket()
sock.bind(('', 60000))
sock.listen(5)
print('accepting...')
conn, id = sock.accept()       # blocks till client connect

for i in range(3):
    print('receiving...')
    msg = conn.recv(1024)      # blocks till data received
    print(msg)                 # gets all print lines at once unless flushed

The client in Example 12-14 sends three messages; the first two over a socket wrapper file, and the last using the raw socket; the manual flush calls in this are commented out but retained so you can experiment with turning them on, and sleep calls make the server wait for data.

import time                            # send three msgs over wrapped and raw socket
from socket import *
sock = socket()                        # default=AF_INET, SOCK_STREAM (tcp/ip)
sock.connect(('localhost', 60000))
file = sock.makefile('w', buffering=1) # default=full buff, 0=error, 1 not linebuff!

print('sending data1')
file.write('spam
')
time.sleep(5)               # must follow with flush() to truly send now
#file.flush()               # uncomment flush lines to see the difference

print('sending data2')
print('eggs', file=file)    # adding more file prints does not flush buffer either
time.sleep(5)
#file.flush()               # output appears at server recv only upon flush or exit

print('sending data3')
sock.send(b'ham
')         # low-level byte string interface sends immediately
time.sleep(5)               # received first if don't flush other two!

Run the server in one window first and the client in another (or run the server first in the background in Unix-like platforms). The output in the server window follows—the messages sent with the socket wrapper are deferred until program exit, but the raw socket call transfers data immediately:

C:...PP4EInternetSockets> socket-unbuff-server.py
accepting...
receiving...
b'ham
'
receiving...
b'spam
eggs
'
receiving...
b''

The client window simply displays “sending” lines 5 seconds apart; its third message appears at the server in 10 seconds, but the first and second messages it sends using the wrapper file are deferred until exit (for 15 seconds) because the socket wrapper is still fully buffered. If the manual flush calls in the client are uncommented, each of the three sent messages is delivered in serial, 5 seconds apart (the third appears immediately after the second):

C:...PP4EInternetSockets> socket-unbuff-server.py
accepting...
receiving...
b'spam
'
receiving...
b'eggs
'
receiving...
b'ham
'

In other words, even when line buffering is requested, socket wrapper file writes (and by association, prints) are buffered until the program exits, manual flushes are requested, or the buffer becomes full.

The short story here is this: to avoid delayed outputs or deadlock, scripts that might send data to waiting programs by printing to wrapped sockets (or for that matter, by using print or sys.stdout.write in general) should do one of the following:

The latter option may be more direct (and the redirection utility module also returns the raw socket in support of such usage), but it isn’t viable in all scenarios, especially for existing or multimode scripts. In many cases, it may be most straightforward to use manual flush calls in shell-oriented programs whose streams might be linked to other programs through sockets.

Also keep in mind that buffered streams and deadlock are general issues that go beyond socket wrapper files. We explored this topic in Chapter 5; as a quick review, the nonsocket Example 12-15 does not fully buffer its output when it is connected to a terminal (output is only line buffered when run from a shell command prompt), but does if connected to something else (including a socket or pipe).

# output line buffered (unbuffered) if stdout is a terminal, buffered by default for
# other devices: use -u or sys.stdout.flush() to avoid delayed output on pipe/socket

import time, sys
for i in range(5):
    print(time.asctime())                 # print transfers per stream buffering
    sys.stdout.write('spam
')            # ditto for direct stream file access
    time.sleep(2)                         # unles sys.stdout reset to other file

Although text-mode files are required for Python 3.X’s print in general, the -u flag still works in 3.X to suppress full output stream buffering. In Example 12-16, using this flag makes the spawned script’s printed output appear every 2 seconds, as it is produced. Not using this flag defers all output for 10 seconds, until the spawned script exits, unless the spawned script calls sys.stdout.flush on each iteration.

# no output for 10 seconds unless Python -u flag used or sys.stdout.flush()
# but writer's output appears here every 2 seconds when either option is used

import os
for line in os.popen('python -u pipe-unbuff-writer.py'):    # iterator reads lines
    print(line, end='')                                     # blocks without -u!

Following is the reader script’s output; unlike the socket examples, it spawns the writer automatically, so we don’t need separate windows to test. Recall from Chapter 5 that os.popen also accepts a buffering argument much like socket.makefile, but it does not apply to the spawned program’s stream, and so would not prevent output buffering in this case.

C:...PP4EInternetSockets> pipe-unbuff-reader.py
Wed Apr 07 09:32:28 2010
spam
Wed Apr 07 09:32:30 2010
spam
Wed Apr 07 09:32:32 2010
spam
Wed Apr 07 09:32:34 2010
spam
Wed Apr 07 09:32:36 2010
spam

The net effect is that -u still works around the stream buffering issue for connected programs in 3.X, as long as you don’t reset the streams to other objects in the spawned program as we did for socket redirection in Example 12-11. For socket redirections, manual flush calls or replacement socket wrappers may be required.

So why use sockets in this redirection role at all? In short, for server independence and networked use cases. Notice how for command pipes it’s not clear who should be called “server” and “client,” since neither script runs perpetually. In fact, this is one of the major downsides of using command pipes like this instead of sockets—because the programs require a direct spawning relationship, command pipes do not support longer-lived or remotely running servers the way that sockets do.

With sockets, we can start client and server independently, and the server may continue running perpetually to serve multiple clients (albeit with some changes to our utility module’s listener initialization code). Moreover, passing in remote machine names to our socket redirection tools would allow a client to connect to a server running on a completely different machine. As we learned in Chapter 5, named pipes (fifos) accessed with the open call support stronger independence of client and server, too, but unlike sockets, they are usually limited to the local machine, and are not supported on all platforms.

Experiment with this code on your own for more insight. Also try changing Example 12-11 to run the client function in a spawned process instead of or in addition to the server, with and without flush calls and time.sleep calls to defer exits; the spawning structure might have some impact on the soundness of a given socket dialog structure as well, which we’ll finesse here in the interest of space.

Despite the care that must be taken with text encodings and stream buffering, the utility provided by Example 12-10 is still arguably impressive—prints and input calls are routed over network or local-machine socket connections in a largely automatic fashion, and with minimal changes to the nonsocket code that uses the module. In many cases, the technique can extend a script’s applicability.

In the next section, we’ll use the makefile method again to wrap the socket in a file-like object, so that it can be read by lines using normal text-file method calls and techniques. This isn’t strictly required in the example—we could read lines as byte strings with the socket recv call, too. In general, though, the makefile method comes in handy any time you wish to treat sockets as though they were simple files. To see this at work, let’s move on.

To help make all of this more concrete, let’s very quickly explore a few simple scripts that add a tkinter frontend to the getfile client-side program. All of these examples assume that you are running a server instance of getfile; they merely add a GUI for the client side of the conversation, to fetch a file from the server. The first, in Example 12-18, uses form construction techniques we met in Chapters 8 and 9 to create a dialog for inputting server, port, and filename information, and simply constructs the corresponding getfile command line and runs it with the os.system call we studied in Part II.

"""
launch getfile script client from simple tkinter GUI;
could also use os.fork+exec, os.spawnv (see Launcher);
windows: replace 'python' with 'start' if not on path;
"""

import sys, os
from tkinter import *
from tkinter.messagebox import showinfo

def onReturnKey():
    cmdline = ('python getfile.py -mode client -file %s -port %s -host %s' %
                      (content['File'].get(),
                       content['Port'].get(),
                       content['Server'].get()))
    os.system(cmdline)
    showinfo('getfilegui-1', 'Download complete')

box = Tk()
labels = ['Server', 'Port', 'File']
content = {}
for label in labels:
    row = Frame(box)
    row.pack(fill=X)
    Label(row, text=label, width=6).pack(side=LEFT)
    entry = Entry(row)
    entry.pack(side=RIGHT, expand=YES, fill=X)
    content[label] = entry

box.title('getfilegui-1')
box.bind('<Return>', (lambda event: onReturnKey()))
mainloop()

When run, this script creates the input form shown in Figure 12-1. Pressing the Enter key (<Return>) runs a client-side instance of the getfile program; when the generated getfile command line is finished, we get the verification pop up displayed in Figure 12-2.

Figure 12-1. getfilegui-1 in action

Figure 12-2. getfilegui-1 verification pop up

The first user-interface script (Example 12-18) uses the pack geometry manager and row Frames with fixed-width labels to lay out the input form and runs the getfile client as a standalone program. As we learned in Chapter 9, it’s arguably just as easy to use the grid manager for layout and to import and call the client-side logic function instead of running a program. The script in Example 12-19 shows how.

"""
same, but with grids and import+call, not packs and cmdline;
direct function calls are usually faster than running files;
"""

import getfile
from tkinter import *
from tkinter.messagebox import showinfo

def onSubmit():
    getfile.client(content['Server'].get(),
                   int(content['Port'].get()),
                   content['File'].get())
    showinfo('getfilegui-2', 'Download complete')

box    = Tk()
labels = ['Server', 'Port', 'File']
rownum  = 0
content = {}
for label in labels:
    Label(box, text=label).grid(column=0, row=rownum)
    entry = Entry(box)
    entry.grid(column=1, row=rownum, sticky=E+W)
    content[label] = entry
    rownum += 1

box.columnconfigure(0, weight=0)   # make expandable
box.columnconfigure(1, weight=1)
Button(text='Submit', command=onSubmit).grid(row=rownum, column=0, columnspan=2)

box.title('getfilegui-2')
box.bind('<Return>', (lambda event: onSubmit()))
mainloop()

This version makes a similar window (Figure 12-3), but adds a button at the bottom that does the same thing as an Enter key press—it runs the getfile client procedure. Generally speaking, importing and calling functions (as done here) is faster than running command lines, especially if done more than once. The getfile script is set up to work either way—as program or function library.

Figure 12-3. getfilegui-2 in action

If you’re like me, though, writing all the GUI form layout code in those two scripts can seem a bit tedious, whether you use packing or grids. In fact, it became so tedious to me that I decided to write a general-purpose form-layout class, shown in Example 12-20, which handles most of the GUI layout grunt work.

"""
##################################################################
a reusable form class, used by getfilegui (and others)
##################################################################
"""

from tkinter import *
entrysize = 40

class Form:                                           # add non-modal form box
    def __init__(self, labels, parent=None):          # pass field labels list
        labelsize = max(len(x) for x in labels) + 2
        box = Frame(parent)                           # box has rows, buttons
        box.pack(expand=YES, fill=X)                  # rows has row frames
        rows = Frame(box, bd=2, relief=GROOVE)        # go=button or return key
        rows.pack(side=TOP, expand=YES, fill=X)       # runs onSubmit method
        self.content = {}
        for label in labels:
            row = Frame(rows)
            row.pack(fill=X)
            Label(row, text=label, width=labelsize).pack(side=LEFT)
            entry = Entry(row, width=entrysize)
            entry.pack(side=RIGHT, expand=YES, fill=X)
            self.content[label] = entry
        Button(box, text='Cancel', command=self.onCancel).pack(side=RIGHT)
        Button(box, text='Submit', command=self.onSubmit).pack(side=RIGHT)
        box.master.bind('<Return>', (lambda event: self.onSubmit()))

    def onSubmit(self):                                      # override this
        for key in self.content:                             # user inputs in
            print(key, '	=>	', self.content[key].get())    # self.content[k]

    def onCancel(self):                                      # override if need
        Tk().quit()                                          # default is exit

class DynamicForm(Form):
    def __init__(self, labels=None):
        labels = input('Enter field names: ').split()
        Form.__init__(self, labels)
    def onSubmit(self):
        print('Field values...')
        Form.onSubmit(self)
        self.onCancel()

if __name__ == '__main__':
    import sys
    if len(sys.argv) == 1:
        Form(['Name', 'Age', 'Job'])     # precoded fields, stay after submit
    else:
        DynamicForm()                    # input fields, go away after submit
    mainloop()

Compare the approach of this module with that of the form row builder function we wrote in Chapter 10’s Example 10-9. While that example much reduced the amount of code required, the module here is a noticeably more complete and automatic scheme—it builds the entire form given a set of label names, and provides a dictionary with every field’s entry widget ready to be fetched.

Running this module standalone triggers its self-test code at the bottom. Without arguments (and when double-clicked in a Windows file explorer), the self-test generates a form with canned input fields captured in Figure 12-4, and displays the fields’ values on Enter key presses or Submit button clicks:

C:...PP4EInternetSockets> python form.py
Age     =>       40
Name    =>       Bob
Job     =>       Educator, Entertainer

Figure 12-4. Form test, canned fields

With a command-line argument, the form class module’s self-test code prompts for an arbitrary set of field names for the form; fields can be constructed as dynamically as we like. Figure 12-5 shows the input form constructed in response to the following console interaction. Field names could be accepted on the command line, too, but the input built-in function works just as well for simple tests like this. In this mode, the GUI goes away after the first submit, because DynamicForm.onSubmit says so:

C:...PP4EInternetSockets> python form.py -
Enter field names: Name Email Web Locale
Field values...
Locale  =>       Florida
Web     =>       http://learning-python.com
Name    =>       Book
Email   =>       [email protected]

Figure 12-5. Form test, dynamic fields

And last but not least, Example 12-21 shows the getfile user interface again, this time constructed with the reusable form layout class. We need to fill in only the form labels list and provide an onSubmit callback method of our own. All of the work needed to construct the form comes “for free,” from the imported and widely reusable Form superclass.

"""
launch getfile client with a reusable GUI form class;
os.chdir to target local dir if input (getfile stores in cwd);
to do: use threads, show download status and getfile prints;
"""

from form import Form
from tkinter import Tk, mainloop
from tkinter.messagebox import showinfo
import getfile, os

class GetfileForm(Form):
    def __init__(self, oneshot=False):
        root = Tk()
        root.title('getfilegui')
        labels = ['Server Name', 'Port Number', 'File Name', 'Local Dir?']
        Form.__init__(self, labels, root)
        self.oneshot = oneshot

    def onSubmit(self):
        Form.onSubmit(self)
        localdir   = self.content['Local Dir?'].get()
        portnumber = self.content['Port Number'].get()
        servername = self.content['Server Name'].get()
        filename   = self.content['File Name'].get()
        if localdir:
            os.chdir(localdir)
        portnumber = int(portnumber)
        getfile.client(servername, portnumber, filename)
        showinfo('getfilegui', 'Download complete')
        if self.oneshot: Tk().quit()  # else stay in last localdir

if __name__ == '__main__':
    GetfileForm()
    mainloop()

The form layout class imported here can be used by any program that needs to input form-like data; when used in this script, we get a user interface like that shown in Figure 12-6 under Windows 7 (and similar on other versions and platforms).

Figure 12-6. getfilegui in action

Pressing this form’s Submit button or the Enter key makes the getfilegui script call the imported getfile.client client-side function as before. This time, though, we also first change to the local directory typed into the form so that the fetched file is stored there (getfile stores in the current working directory, whatever that may be when it is called). Here are the messages printed in the client’s console, along with a check on the file transfer; the server is still running above testdir, but the client stores the file elsewhere after it’s fetched on the socket:

C:...InternetSockets> getfilegui.py
Local Dir?      =>       C:usersMark	emp
File Name       =>       testdirora-lp4e.gif
Server Name     =>       localhost
Port Number     =>       50001
Client got testdirora-lp4e.gif at Sun Apr 25 17:22:39 2010

C:...InternetSockets> fc /B C:Usersmark	empora-lp4e.gif testdirora-lp4e.gif
FC: no differences encountered

As usual, we can use this interface to connect to servers running locally on the same machine (as done here), or remotely on a different computer. Use a different server name and file paths if you’re running the server on a remote machine; the magic of sockets make this all “just work” in either local or remote modes.

One caveat worth pointing out here: the GUI is essentially dead while the download is in progress (even screen redraws aren’t handled—try covering and uncovering the window and you’ll see what I mean). We could make this better by running the download in a thread, but since we’ll see how to do that in the next chapter when we explore the FTP protocol, you should consider this problem a preview.

In closing, a few final notes: first, I should point out that the scripts in this chapter use tkinter techniques we’ve seen before and won’t go into here in the interest of space; be sure to see the GUI chapters in this book for implementation hints.

Keep in mind, too, that these interfaces just add a GUI on top of the existing script to reuse its code; any command-line tool can be easily GUI-ified in this way to make it more appealing and user friendly. In Chapter 14, for example, we’ll meet a more useful client-side tkinter user interface for reading and sending email over sockets (PyMailGUI), which largely just adds a GUI to mail-processing tools. Generally speaking, GUIs can often be added as almost an afterthought to a program. Although the degree of user-interface and core logic separation varies per program, keeping the two distinct makes it easier to focus on each.

And finally, now that I’ve shown you how to build user interfaces on top of this chapter’s getfile, I should also say that they aren’t really as useful as they might seem. In particular, getfile clients can talk only to machines that are running a getfile server. In the next chapter, we’ll discover another way to download files—FTP—which also runs on sockets but provides a higher-level interface and is available as a standard service on many machines on the Net. We don’t generally need to start up a custom server to transfer files over FTP, the way we do with getfile. In fact, the user-interface scripts in this chapter could be easily changed to fetch the desired file with Python’s FTP tools, instead of the getfile module. But instead of spilling all the beans here, I’ll just say, “Read on.”