When the output gets large, it becomes harder to spot errors.
When the programmer tires, it is easy to miss subtly incorrect output.
When the implemented feature becomes larger, one tends to miss regressions in parts that were “tested” earlier.
Because the informal test scripts are generally only useful for the programmer who wrote them, their utility is lost to other developers.
Thus, unit testing was invented, in which one writes sample calls to pieces of code and compares the return value to the expected value.
This comparison is typically done in a way that produces little or no output when the test passes and very obvious output otherwise. A test harness can be used to run tests from several test scripts and only report the errors and a statistical summary of the passed tests.
2.1 Digression: Virtualenvs
To run the unit tests we are going to write, we require some additional tools that are available as Python packages. To install them, you should use a tool called a virtualenv . This is a Python directory that contains a Python interpreter, package management programs such as pip, as well as symbolic links to the base Python packages, thus giving you a pristine Python environment on which to build a customized, isolated virtual environment containing exactly the libraries you need. A virtualenv enables you to install any Python package you want; you don’t need root privileges in order to install a dependency for your application. You can activate one virtualenv within a given shell session and simply delete the directory when you don’t need it anymore.
When you are done, disable it with the command deactivate.
2.2 Getting Started with Unit Tests
To illustrate unit testing, let’s start with a single function and how to test it. The function I want to implement here is a binary search . Given a sorted list of numbers (we call it the haystack), search for another number (the needle) in it. If it’s present, return the index at which it was found. If not, raise an exception of type ValueError. You can find the code and tests for this example at https://github.com/python-ci-cd/binary-search .
We start by looking at the middle element of the haystack. If it happens to be equal to the needle, we are done. If it is smaller than the needle, we can repeat the search in the left half of the haystack. If it’s larger, we can continue the search in the right half of the haystack.
To keep track of the area inside the haystack that we need to search, we keep two indices, left and right, and in each iteration, move one of them closer to the other, cutting the space to be searched in half in each step.
The First Test
The test run prints various pieces of information: These include details about the platform and version of the software involved, the working directory, and what pytest configuration file was used (none in this example).
The line collected 1 item then shows that pytest found one test function. The dot behind the file name in the next line shows the progress, with one dot for each test that has been executed.
This shows the test function that fails, both as source code and with values substituted in on both sides of the == operator in the assert call, showing exactly what went wrong. In a terminal with color support, the failed test and the status line at the bottom are shown in red, to make failed tests obvious.
Writing More Tests
The test for finding the first element passes, but the test for the last element hangs, that is, it runs indefinitely without terminating. You can abort the Python process by pressing the Ctrl and C keys simultaneously.
If function search can find the first but not the last element, there must be some kind of asymmetry in it. Indeed there is: determining the middle element uses the integer division operator //, which rounds positive numbers toward zero. For example, 1 // 2 == 0. This explains why the loop can get stuck: when right is equal to left + 1, the code sets middle to the value of left. If the branch left = middle is executed, the area of the haystack in which the function searches does not decrease in size, and the loop gets stuck.
With this fix in place, all three tests pass.
Testing the Unhappy Path
The tests so far focused on the “happy path,” the path in which an element was found and no error encountered. Because exceptions are not the exception (excuse the pun) in normal control flow, they should be tested too.
Here, we test three scenarios: that a value was smaller than the first element in the haystack, larger than the last, and, finally, that it is between the first and the last element in size but simply not inside the haystack.
The pytest.raises routine returns a context manager . Context managers are, among other things, a neat way to wrap code (inside the with ... block) in some other code. In this case, the context manager catches the exception from the with block, and the test passes if it is of the right type. Conversely, the test fails if either no exception was raised or one of a wrong type, such as a KeyError, was.
2.3 Dealing with Dependencies
Not all code is as simple to test, as with the search function from the previous sections. Some functions call external libraries or interact with databases, APIs, or the Internet.
The actions might have unwanted side effects, such as sending e-mails to customers or colleagues and confusing them or even causing harm.
You typically do not have control over external services, which means you do not have control over consistent responses, which makes writing reliable tests much harder.
Performing external actions, such as writing or deleting files, leaves the environment in a different state, which potentially leads to test results that cannot be reproduced.
Performance suffers, which negatively impacts the development feedback cycle.
Often, external services, such as databases or APIs, require credentials, which are a hassle to manage and pose a serious barrier to setting up a development environment and running tests.
How do you, then, avoid these external dependencies in your unit tests? Let’s explore some options.
Separating Logic from External Dependencies
Many applications get data from somewhere, often different sources, then do some logic with it, and maybe print out the result in the end.
At the time of writing, this code prints Python as the answer, though this might change in future, at the discretion of the python.org maintainers.
You can find the sample code and tests at https://github.com/python-ci-cd/python-webcount .
This code uses the requests library to fetch the contents of a web page and accesses the resulting text (which is really HTML). The function then iterates over the search words, counts how often each occurs in the text (using the string.count method), and constructs a dictionary with these counts. It then sorts the lists of words by their frequency and returns the most commonly occurring one, which is the last element of the sorted list.
most_common_word does not actually look for word boundaries, so it will happily count the “word” b three times in the string abbbcc.
The function will raise an exception when called with an empty list of keywords, but we haven’t bothered to specify what kind of error.1
We haven’t specified which value to return if two or more words have the same occurrence count, hence the last test using in with a list of two valid answers.
Depending on your situation, you might want to leave such tests as documentation of known edge cases or refine both the specification and the implementation.
Returning to the topic of testing functions with external dependencies, we have reached partial success. The interesting logic is now a separate, pure function and can be tested easily. The original function, most_common_word_in_web_page, is now simpler but still untested.
We have, implicitly, established the principle that it is acceptable to change code to make it easier to test, but it is worth mentioning explicitly. We will use it more in the future.
Dependency Injection for Testing
Instead of using requests directly, the function now accepts an optional argument, user_agent, which defaults to requests. Inside the function, the sole use of requests has been replaced by user_agent.
The technique demonstrated in this section is a simple form of dependency injection .2 The caller has the option to inject an object or class on which a function depends.
Dependency injection is useful not just for testing but also for making software more pluggable. For example, you might want your software to be able to use different storage engines in different contexts, or different XML parsers, or any number of other pieces of software infrastructure for which multiple implementations exist.
Mock Objects
Writing test double classes can become tedious pretty quickly, because you often require one class per method called in the test, and all of these classes must be set up to correctly chain their responses. If you write multiple test scenarios, you either have to make the test doubles generic enough to cover several scenarios or repeat nearly the same code all over again.
This installs an attribute, get, in object mock_requests, which, when it is called, returns another mock object. The attribute text on that second mock object has an attribute text, which holds the string 'aa bb c'.
Let’s start with some simpler examples. If you have a Mock object m, then m.a = 1 installs an attribute a with value 1. On the other hand, m.b.return_value = 2 configures m, so that m.b() returns 2.
You can continue to chain, so m.c.return_value.d.e.return_value = 3 makes m.c().d.e() return 3. In essence, each return_value in the assignment corresponds to a pair of parentheses in the call chain.
In addition to setting up these prepared return values, mock objects also record calls. The previous example checked the call_count of a mock object, which simply records how often that mock has been called as a function.
The call_args property contains a tuple of arguments passed to its last call. The first element of this tuple is a list of positional arguments, the second a dict of named arguments.
If you want to check multiple invocations of a mock object, call_args_list contains a list of such tuples.
The Mock class has more useful methods. Please refer to the official documentation3 for a comprehensive list.
Patching
The call to the patch function (imported from unittest.mock, a standard library shipped with Python) specifies both the symbol to be patched (temporarily replaced) and the test double by which it is replaced. The patch function returns a context manager. So, after execution leaves the with block that the call occurs in, the temporary replacement is undone automatically.
When patching an imported symbol , it is important to patch the symbol in the namespace that it was imported in, not in the source library. In our example, we patched webcount.functions.requests, not requests.get.
Patching removes interactions with other code, typically libraries. This is good for testing code in isolation, but it also means that patched tests cannot detect misuse of libraries that have been patched out. Thus, it is important to write broader scoped tests, such as integration tests or acceptance tests, to cover correct usage of such libraries.
2.4 Separating Code and Tests
So far, we’ve put code and tests into the same file, just for the sake of convenience. However, code and tests serve different purposes, so as they grow in size, it is common to split them into different files and, typically, even into different directories. Our test code now also loads a module on its own (pytest) , a burden that you don’t want to put on production code. Finally, some testing tools assume different files for test and code, so we will follow this convention.
When developing a Python application, you typically have a package name for the project and a top-level directory of the same name. Tests go into a second top-level directory called tests. For example, the Django web framework has the directories django and test, as well as a README.rst as the entry point for beginners, and setup.py for installing the project.
Each directory that serves as a Python module must contain a file called __init__.py, which can be empty or contain some code. Typically, this code just imports other symbols, so that users of the module can import them from the top-level module name.
We can put test functions into any file in the test/ directory. In this instance, we put them into the file test/test_functions.py, to mirror the location of the implementation. The test_ prefix tells pytest that this is a file that contains tests.
Tweaking the Python Path
Python cannot find the module under test, webcount, because it is not located in Python’s default module loading path.
You can fix this by adding the absolute path of your project’s root directory to a file with the extension .pth into your virtualenv’s site-packages directory. For example, if you use Python 3.5, and your virtualenv is in the directory venv/, you could put the absolute path into the file venv/lib/python3.5/site-packages/webcount.pth. Other methods of manipulating the “Python path” are discussed in the official Python documentation.4
A pytest-specific approach is to add an empty file, conftest.py, to your project’s root directory. Pytest looks for files of that name and, on detecting their presence, marks the containing directory as a project to be tested and adds the directory to the Python path during the test run.
You don’t have to specify the test file when invoking pytest. If you leave it out, pytest searches for all test files and runs them. The pytest documentation on integration practices5 has more information on how this search works.
2.5 More on Unit Testing and Pytest
There are many more topics that you might encounter while trying to write tests for your code. For example, you might have to manage fixtures, pieces of data that serve as the baseline of your tests. Or you may have to patch functions from code that is loaded at runtime or do a number of other things that nobody prepared you for.
For such cases, the pytest documentation6 is a good starting point. If you want a more thorough introduction, the book Python Testing with pytest by Brian Okken (The Pragmatic Bookshelf, 2017) is worth reading.
2.6 Running Unit Tests in Fresh Environments
Developers typically have a development environment in which they implement their changes, run the automatic and sometimes manual tests, commit their changes, and push them to a central repository. Such a development environment tends to accumulate Python packages that are not explicit dependencies of the software under development, and they tend to use just one Python version. These two factors also tend to make test suites not quite reproducible, which can lead to a “works on my machine” mentality.
To avoid that, you need a mechanism to easily execute a test suite in a reproducible manner and on several Python versions. The tox automation project7 provides a solution: you supply it with a short configuration file, tox.ini, that lists Python versions and a standard setup.py file for installing the module. Then, you can just run the tox command .
This uses Python’s library setuptools to make the code under development installable. Usually, you’d include more metadata, such as author, e-mail address, a more verbose description, etc.
The envlist in this example contains just py35 for Python 3.5. If you also want to run the tests on Python 3.6, you could write envlist = py35,py36. The key pypy35 would refer to the alternative pypy implementation of Python in version 3.5.
2.7 Another Sample Project: matheval
Many projects these days are implemented as web services, so they can be used through HTTP—either as an API or through an actual web site. Let’s consider a tiny web service that evaluates mathematical expressions that are encoded as trees in a JSON data structure. (You can find the full source code for this project at https://github.com/python-ci-cd/python-matheval/ .) As an example, the expression 5 * (4 - 2) would be encoded as the JSON tree ["*", 5, ["+", 4, 2]] and evaluate to 10.
Application Logic
File matheval/evaluator.py: Evaluation Logic
File matheval/frontend.py: Web Service Binding
File test/test_evaluator.py: Unit Tests for Evaluating Expression Trees
The index route is not complicated enough to warrant a unit test on its own, but in a later chapter, we will write a smoke test that exercises it once the application is installed.
File setup.py for matheval
2.8 Summary
Unit tests exercise a piece of code in isolation, by calling it with sample inputs and verifying that it returns the expected result or throws the expected exception. With pytest, a test is a function whose name starts with test_ and contains assert statements that verify return values. You run these test files with pytest path/to/file.py, and it finds and runs the tests for you. It makes test failures very obvious and tries to provide as much context as possible to debug them.
Mock objects provide a quick way to create test doubles, and the patching mechanism provides a convenient way to inject them into the tested code.
The tox command and project create isolated test environments that make test suites reproducible and more convenient to test on multiple Python versions and implementations.