4.1 Files

It has been a long time since “everything is a file” was an accurate mantra on Unix. Nevertheless, many things are files, and even more things are enough like files that manipulating them with file-based system calls works.

Python programs can go down one of two routes when dealing with a file’s contents. They can open a file as text or as binary. Although files are neither text nor binary, just a blob of bytes, the opening mode is important.

When opening a file as binary, the bytes are read and written as byte strings. This is useful with files that are non-textual, such as picture files.

When opening a file as text, encoding must be used. It can be specified explicitly, but in certain situations, defaults apply. All bytes read from the file are decoded, and the code receives a character string. All strings written to the file are encoded into bytes. This means the interface with the file is with strings, or sequences of characters.

The next four bytes are the width. This simple code gave you the high-level data, confirming that this is XCF. It showed the format it is, and you could see the dimensions of the image.

When opening files as text, the default encoding is UTF-8. One advantage of UTF-8 is that it is designed to quickly if something is not UTF-8. It is carefully designed to fail on ISO-8859-[1–9], which predates Unicode, as well as on most binary files. It is also backward compatible with ASCII, which means pure ASCII files are still valid UTF-8.

Usually, you do not call next directly but use for. Additionally, you use files as context managers to make sure they close at a well-understood point. However, there is a trade-off, especially in REPL scenarios; opening the file without a context manager allows you to explore reading bits and pieces.

Files on a Unix system are more than just blobs of data. They have various metadata attached, which can be queried, and sometimes changed.

The rename system call is wrapped in the os.rename Python function. Since rename is atomic, this can help implement operations that require a certain state.

Note that the os module tends to be a slight shim over operating system calls. The discussion here is relevant to Unix-like systems: Linux, BSD-based systems, and, for the most part, macOS. It is worth keeping in mind, but it is not worth pointing out each place where you are making Unix-specific assumptions.

This ensures that you do not accidentally misunderstand the sentence when reading an important file. If the code crashes in the middle, instead of believing that the horse raced past the barn, you get nothing from important. You only rename important.tmp to important at the end, after the last word has been written to the file.

This can come in useful, for example, when mirroring an existing file layout.

The os.path module has mostly string manipulation functions that assume strings are file names. The dirname function returns the directory name, so os.path.dirname("a/b/c") would return a/b. Similarly, the basename function returns the “file name,” so os.path.basename("a/b/c") would return c. The inverse of both is the os.path.join function, which joins paths; os.path.join("some", "long/and/winding", "path") would return some/long/and/finding/path.

Another set of functions in the os.path module has a slightly higher-level abstraction for getting file metadata. It is important to note that these functions are often light wrappers around operating system functionality and do not try to hide operating system quirks. This means that operating system quirks can “leak” through the abstraction.

The biggest metadata is os.path.exists. Does the file exist? This comes in handy sometimes, though often-case it is better to write code in a way that is agnostic of file existence: file existence can have races. Subtler are the os.path.is... functions (isdir, isfile, islink, etc.), which can decide if a file name points to what you expect.

The os.path.get... functions get non-boolean metadata: access time, modification time, c-time (sometimes shortened to “creation time” but misleadingly is not the actual time of creation in a set of subtle circumstances, more accurately referred to as “i-node modification time”). getsize gets the size of the file.

The shutil module (“shell utilities”) contains some higher-level operations. shutil.copy copies a file’s contents as well as metadata. shutil.copyfile copies the contents only. shutil.rmtree is the equivalent of rm -r, while shutil.copytree is the equivalent of cp -r.

Finally, temporary files are often useful. Python’s tempfile module produces temporary files which are secure and resistant to leaks. The most useful functionality is NamedTemporaryFile, which can be used as a context.

Note that the fp.flush there is important. The file object caches write until closed. However, NamedTemporaryFile vanishes when closed. Explicitly flushing it is important before calling a function that reopens the file for reading.

4.2 Processes

The main module to deal with running subprocesses in Python is subprocess. It contains a high-level abstraction that matches the intuitive model most have when they think of “running commands,” rather than the low-level model implemented in Unix, using exec and fork.

It is also a powerful alternative to calling the os.system function, which is problematic in several ways. For one, os.system spawns an extra process, the shell. This means that it depends on the shell, which on some weirder installations can differ from a more “exotic” system shell, like ash or fish. Finally, the shell parses the string, which means the string must be properly serialized. This is a hard task to do since the formal specification for the shell parser is long. Unfortunately, it is not hard to write something that works fine most of the time, so most bugs are subtle and break at the worst possible time. This sometimes even manifests as a security flaw.

While the subprocess module is not completely flexible, it is perfectly adequate for most needs. It is divided into high-level functions and a lower-level implementation level. The high-level functions, which should be used in most circumstances, are check_call and check_output. Among other benefits, they behave like running a shell with -e or set err. They immediately raise an exception if a command returns with a non-zero value.

The slightly lower level is Popen, which creates processes and allows fine-grained configuration of their inputs and outputs. Both check_call and check_output are implemented on top of Popen. Because of that, they share some semantics and arguments. The most important argument is shell=True, and it is most important in that it is almost always a bad idea to use it. When the argument is given, a string is expected and is passed to the shell to parse it.

Shell parsing rules are subtle and full of corner cases. If it is a constant command, there is no benefit there. You can translate the command to separate arguments in code. If it includes some input, it is almost impossible to reliably escape it in a way that makes it impossible to introduce an injection problem. On the other hand, without this, creating commands on the fly is reliable, even in the face of potentially hostile inputs.

Using check_call means that if the command fails for some reason, such as the user not existing, this automatically raises an exception. This avoids a common failure mode, where scripts do not report accurate status.

Again, this automatically raises an exception if the command fails. If it succeeds, you get the output as a string. There is no need to manually read and determine end conditions.

This gets the current git hash of the project, assuming the project is a git directory. If it is not, git rev-parse HEAD returns non-zero, and causes an exception to be raised.

Note that you had to decode the output since subprocess.check_output, like most functions in a subprocess, returns a byte string, not a Unicode string. In this case, rev-parse HEAD always returns a hexadecimal string, so you used the ascii codec. This fails on any non-ASCII characters.

There are some circumstances under which using high-level abstractions is impossible; for example, having to send standard input or read the output in chunks is not possible with them.

Popen runs a subprocess and allows fine-grained control of the inputs and outputs. While all things are possible, most things are not easy to do correctly. The shell pattern of writing long pipelines is unpleasant to implement, even more unpleasant to make sure there are no lingering deadlock conditions, and unnecessary.

As noted, it is a best practice in Python to avoid true command parallelism. In all the cases, you tried to finish one stage before reading from the next. In Python, subprocess is generally only used for calling out to external commands. You usually use Python’s built-in processing abilities for pre-processing of inputs and post-processing of outputs. In the preceding case, you would use sorted, slices, and string manipulation to simulate the logic.

The commands for text and number processing are seldom useful in Python, which has a good in-memory model for doing such processing. The general case for calling commands in scripts is for things that manipulate data in a way that is either only documented as accessible by commands; for example, querying processes via ps -ef, or where the alternative to the command is a subtle library, sometimes requiring binary binding, such as in the case of docker or git.

This is one place where translating shell scripts into Python must be done with care and thought. Where the original had a long pipeline that depended on ad hoc string manipulation via awk or sed, Python code can be less parallel and more obvious. It is important to note that there is something lost in translation in those cases. The original low-memory requirements and transparent parallelism. However, in return, you get more maintainable and debuggable code.

4.3 Networking

Python has plenty of networking support. It has it from the lowest level—support of the socket-based system calls to high-level protocol supports. Some of the best approaches to problems are built-in libraries. For other problems, the best solution is a third-party library.

The most straightforward translation of low-level networking APIs is in the socket module. This module exposes the socket object.

The line s = socket.socket() creates a new socket object. There are various things that you can do with socket objects. One of them is to connect them to an endpoint; in this case, to the httpbin.org server on port 80. The default socket type is a stream, Internet type, which is the way Unix refers to TCP sockets.

After the socket is connected, you can send bytes to it. With sockets, only byte strings can be sent. You read back the result, do some ad hoc HTTP response parsing, and parse the actual content as JSON.

Generally, it is better to use a real HTTP client, but this showcases how to write low-level socket code. This can be useful, for example, if you want to diagnose a problem by replaying exact messages.

The socket API is subtle, and the example has a few incorrect assumptions. In most cases, this code works but fails in strange ways in the face of corner cases.

The send method is allowed to not send all the data if not all of it can fit into the internal kernel-level send buffer. This means that it can do a “partial send.” It returned 40, which was the entire length of the byte string. Correct code checks for the return value and sends the remaining chunks until nothing is left. Luckily, Python already has a method to do it: sendall.

However, a more subtle problem occurs with recv. It returns as much as the kernel-level buffer has because it does not know how much the other side intended to send. Again, much of the time, especially for short messages, this works fine. For protocols like HTTP 1.0, the correct behavior is to read until the connection is closed.

This is a common problem in networking code and can happen using higher-level abstractions. Things can appear to work in simple cases while failing in more extreme circumstances, such as high load or network congestion.

There are ways to test for these things. One of them is using proxies that exhibit extreme behaviors. Writing, or customizing proxies, requires low-level network coding using socket.

Python also has higher-level abstractions for networking. While the urllib and urllib2 modules are part of the standard library, best practices on the web evolve fast, and in general, for higher-level abstractions, third-party libraries are usually better.

Instead of crafting your own HTTP requests out of raw bytes, all you needed to do was to give a URL similar to a URL you might type into a browser. Requests parsed it to find the host to connect to (httpbin.org), the port (80, the default for HTTP), and the path (/get). Once the response came in, it automatically parsed it into headers and content and allowed you to access the content directly as JSON.

As easy as requests are to use, it is almost better to put in more effort and use the Session object. Otherwise, the default session is used. This leads to code with non-local side effects: one sublibrary that calls requests changes a session state, which leads to another sublibrary’s calls to act differently. For example, HTTP cookies are shared across a session.

In this example, the request is simple, and the session state does not matter. However, this is a good habit; even in the interactive interpreter, avoid using the get, put, and other functions directly and use only the session interface.

It is natural to use an interactive environment to prototype code that would later make it into a production program. By keeping good habits like this, you ease the transition.

4.4 Summary

Python is a powerful tool for automating operating system operations. This comes from having libraries that are thin wrappers around native operating system calls and powerful third-party libraries.

This allows you to get close to the operating systems without any intervening abstractions and write high-level code that does not care about the details when these do not matter.

This combination often makes Python a superior alternative for writing scripts instead of the Unix shell. It does require a different way of thinking. Python is not as suitable for the long pipeline of text transformers approach, but in practice, those long pipelines of text transformers turn out to be an artifact of shell limitations.

With a modern memory-managed language, it is often easier to read the entire text stream into memory and then manipulate it without being limited to only those transformations specified as pipes.