Socket Addresses

The different types of sockets use different address formats.

Unix socket addresses are strings naming a node in the filesystem (on Linux platforms, strings starting with b'' correspond to names in a kernel table).

IPv4 socket addresses are pairs (address, port). The first item is an IPv4 address, the second a port number in the range 1–65,535.

IPv6 socket addresses are four-item (address, port, flowinfo, scopeid) tuples. When providing an address as an argument, the flowinfo and scopeid items can be omitted, though this can sometimes cause problems.

Client-Server Computing

The pattern we discuss hereafter is usually referred to as client-server networking, where a server listens for traffic on a specific endpoint from clients who require the service. We do not cover peer-to-peer networking, which, lacking any central server, has to include the ability for peers to discover each other. This is often, ironically, achieved by contacting a central server, although discovery protocols such as SSDP allow complete independence from any central authority.

Most, though by no means all, network communication is performed using client-server techniques. The server listens for incoming traffic at a predetermined or advertised network endpoint. In the absence of such input it does nothing, simply sitting there waiting for input from clients. Communication is somewhat different between connectionless and connection-oriented endpoints.

In connectionless networking such as UDP, requests arrive at a server randomly and are dealt with immediately: a response is dispatched to the requester without delay. Each message is handled on its own, usually without reference to any communications that may previously have occurred between the two parties. Connectionless networking is thus well-suited to short-term, stateless interactions such as those required by DNS or network booting.

In connection-oriented networking, the client engages in an initial exchange with the server that effectively establishes a connection across a network pipeline between two processes (sometimes referred to as a virtual circuit), across which the processes can communicate until both have indicated their willingness to end the connection. Serving under these conditions requires the use of parallelism using a concurrency mechanism (such as threads or processes, covered in Chapter 14, or asynchronous programming, covered in Chapter 18) to handle each incoming connection asynchronously or simultaneously. Without such parallelism, the server would be unable to handle new incoming connections before earlier ones had terminated, since calls to socket methods normally block (meaning they pause the thread calling them until they terminate or time-out). Connections are the best way to handle lengthy interactions such as mail exchanges, command-line shell interactions, or the transmission of web content, and offer automatic error detection and correction when TCP is used.

The socket Module

Python’s socket module handles networking with the socket interface. There are minor differences between platforms, but the module hides most of them, making it relatively easy to write portable networking applications.

The module defines four exceptions: their base class socket.error, which in v2 is a subclass of exceptions.IOError; in v3 it is a (deprecated) alias for exceptions.OSError, and three exception subclasses as follows:

The module also defines a large set of constants. The most important of these are the address families (AF_*) and the socket types (SOCK_*) listed next. In v2 these constants are integers, while in v3 they are members of IntEnum collections. This difference can be disregarded, but, to debug your code, it’s more helpful to see, for example, <SocketKind.SOCK_STREAM: 1> than a simple 1. The module defines many other constants, used to set socket options, but the documentation does not define them fully: to use them you must be familiar with the low-level documentation for the C sockets libraries and system calls.

The module defines a number of functions to create sockets, manipulate address information, and assist with representing data in a standard way. We do not cover all possibilities in this book, as the socket module documentation is comprehensive; we deal with the ones that are most essential in writing networked applications.

Socket Objects

The socket object is the primary means of network communication in Python. A new socket is also created when a SOCK_STREAM socket accepts a connection, each such socket being used to communicate with the relevant client.

There are a number of ways you can create a socket, as detailed in the next section. The socket can operate in different modes, determined by its timeout value, established in one of three ways:

• By providing the timeout value on creation

• By calling the socket object’s settimeout method

• According to the socket module’s default timeout value as returned by the socket.getdefaulttimeout function

The timeout values to establish each mode are as follows:

A Connectionless Socket Client

Consider a simplistic packet-echo service. Text encoded in UTF-8 (the assumed encoding of most Python source files) is sent to a server, which sends the same information back to the client originating it. In a connectionless service, all the client has to do is send each chunk of data to the defined server endpoint.

The following client works equally well on either v2 or v3 (printing slightly different output to stdout in each case. In v2, Unicode text strings are marked as u'...' while bytestrings are not marked; in v3, Unicode text strings are not marked, while bytestrings are marked as b'...'):

# coding: utf-8
from __future__ import print_function
import socket

UDP_IP = '127.0.0.1'
UDP_PORT = 8883
MESSAGE = u"""
This is a bunch of lines, each
of which will be sent in a single
UDP datagram. No error detection
or correction will occur.
Crazy bananas! £€ should go through."""

sock = socket.socket(socket.AF_INET,  # Internet
                     socket.SOCK_DGRAM)  # UDP
server = UDP_IP, UDP_PORT
for line in MESSAGE.splitlines():
    data = line.encode('utf-8')
    sock.sendto(data, server)
    print('Sent', repr(data), 'to', server)
    response, address = sock.recvfrom(1024)  # buffer size: 1024
    print('Recv', repr(response.decode('utf-8')), 'from', address)

Note that the server is only expected to perform a byte-oriented echo function. The client therefore encodes its Unicode data into specific bytestrings, and decodes the bytestring responses received from the server back into Unicode text.

A Connectionless Socket Server

A server for this service is also quite simple. It binds to its endpoint, receives packets (datagrams) at that endpoint, and returns a packet to the client sending each datagram, with exactly the same data. The server treats all clients equally and does not need to use any kind of concurrency (though this handy characteristic might not hold for a service where request handling takes more time).

The following server works equally well on v2 or v3 (with slightly different output in each version, as explained previously for the client):

from __future__ import print_function
import socket

UDP_IP = '127.0.0.1'
UDP_PORT = 8883

sock = socket.socket(socket.AF_INET,  # Internet
                     socket.SOCK_DGRAM)  # UDP
sock.bind((UDP_IP, UDP_PORT))
print('Serving at', UDP_IP, UDP_PORT)

while True:
    data, addr = sock.recvfrom(1024)  # buffer size is 1024 bytes
    print('Recv', repr(data), 'from', addr)
    sock.sendto(data, addr)
    print('Sent', repr(data), 'to', addr)

This code offers no way to terminate the service other than by interrupting it (typically from the keyboard, with Ctrl-C or Ctrl-Break).

A Connection-Oriented Socket Client

Consider a simplistic connection-oriented “echo-like” protocol: a server lets clients connect to its listening socket, receives arbitrary bytes from them, and sends back to each client the same bytes that client sent to the server, until the client closes the connection. Here’s an example of an elementary test client (fully coded to run equally well in v2 and v3)¹:

# coding: utf-8
from __future__ import print_function
import socket

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect(('localhost', 8881))
    print('Connected to server')
    data = u"""A few lines of text
including non-ASCII characters: €£
to test the operation
of both server
and client."""
    for line in data.splitlines():
        sock.sendall(line.encode('utf-8'))
        print('Sent:', line)
        response = sock.recv(1024)
        print('Recv:', response.decode('utf-8'))
print('Disconnected from server')

Note that the data is text, so it must be encoded with a suitable representation, for which we chose the usual suspect—UTF-8. The server works in terms of bytes (since it is bytes, AKA octets, that travel on the network); the received bytes object gets decoded with UTF-8 back into Unicode text before printing. Any other suitable codec could be used: the key point is that text must be encoded before transmission and decoded after reception. The server, working in terms of bytes, does not even need to know which encoding is being used, except maybe for logging purposes.

A Connection-Oriented Socket Server

Here is a simplistic server corresponding to the testing client shown in “A Connection-Oriented Socket Client”, using multithreading via concurrent.futures, covered in “The concurrent.futures Module”, also fully coded to run equally well in v3 and v2 (as long as you’ve installed the concurrent backport—see “The concurrent.futures Module”):

from __future__ import print_function
from concurrent import futures as cf
import socket

def handle(new_sock, address):
    print('Connected from', address)
    while True:
        received = new_sock.recv(1024)
        if not received: break
        s = received.decode('utf-8', errors='replace')
        print('Recv:', s)
        new_sock.sendall(received)
        print('Echo:', s)
    new_sock.close()
    print('Disconnected from', address)

servsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
servsock.bind(('localhost', 8881))
servsock.listen(5)
print('Serving at', servsock.getsockname())

with cf.ThreadPoolExecutor(20) as e:
    try:
        while True:
            new_sock, address = servsock.accept()
            e.submit(handle, new_sock, address)
    except KeyboardInterrupt:
        pass
    finally:
        servsock.close()

This server has its limits. In particular, it runs only 20 threads, so it cannot be simultaneously serving more than 20 clients; any further client trying to connect while 20 are being served waits in servsock’s listening queue, and (should that queue fill up with 5 clients waiting to be accepted) further clients attempting connection get rejected outright. This server is intended just as an elementary example for demonstration purposes, not as a solid, scalable, or secure system.

SSLContext

The ssl module supplies an ssl.SSLContext class, whose instances hold information about TLS configuration (including certificates and private keys) and offer many methods to set, change, check, and use that information. If you know exactly what you’re doing, you can manually instantiate, set up, and use your own SSLContext instances for your own specialized purposes.

However, we recommend instead that you instantiate an SSLContext using the well-tuned function named ssl.create_default_context with a single argument: ssl.Purpose.CLIENT_AUTH if your code is a server (and thus may need to authenticate clients), or ssl.Purpose.SERVER_AUTH if your code is a client (and thus definitely needs to authenticate servers). If your code is both a client to some servers and a server to other clients (as, for example, some Internet proxies are), then you’ll need two instances of SSLContext, one for each purpose.

For most client-side uses, your SSLContext is ready. If you’re coding a server, or a client for one of the rare servers that require TLS authentication of the clients, you need to have a certificate file and a key file, and add them to the SSLContext instance (so that counter-parties can verify your identify) with code such as, for example:

ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ctx.load_cert_chain(certfile='mycert.pem', keyfile='mykey.key')

passing the paths to the certificate and key files to the load_cert_chain method. (See the online docs to learn how to obtain key and certificate files.)

Once your context instance ctx is ready, if you’re coding a client, just call ctx.wrap_socket to wrap any socket you’re about to connect to a server, and use the wrapped result (an instance of ssl.SSLSocket) instead of the socket you just wrapped. For example:

sock = socket.socket(socket.AF_INET)
sock = ctx.wrap_socket(sock, server_hostname='www.example.com')
sock.connect(('www.example.com', 443))
# just use `sock` normally from here onwards

Note that, in the client case, you should also pass wrap_socket a server_hostname argument corresponding to the server you’re about to connect to; this way, the connection can verify that the identity of the server you end up connecting to is indeed correct, one absolutely crucial step to any Internet security.

Server-side, don’t wrap the socket that you are binding to an address, listening on, or accepting connections on; just bind the new socket to accept returns. For example:

sock = socket.socket(socket.AF_INET)
sock.bind(('www.example.com', 443))
sock.listen(5)
while True:
    newsock, fromaddr = sock.accept()
    newsock = ctx.wrap_socket(newsock, server_side=True)
    # deal with `newsock` as usual, shutdown and close it when done

In this case, what you need to pass to wrap_socket is an argument server_side=True, so it knows that you’re on the server side of things.

Again, we recommend the online docs, and particularly the examples, for better understanding of even this simple subset of ssl operations.