Even though the overall format of a doctest is identical to the interpreter sessions shown throughout this book, there are some specific details that are important to identify. Each line of code to execute begins with three right angle brackets (>>>) and a single space, followed by the code itself.

>>> a = 2

Just like the interactive interpreter, any code that extends beyond one line is indicated by new lines beginning with three periods (...) rather than brackets. You can include as many of these as necessary to complete multi-line structures, such as lists and dictionaries, as well as function and class definitions.

>>> b = ('example',
... 'value')
>>> def test():
...     return b * a

All the lines that start with periods like this are combined with the last line that started with angle brackets, and they're all evaluated together. That means you can leave extra lines if necessary, anywhere in the structure or even after it. This is useful for mimicking the output of an actual interpreter session, which requires a blank line to indicate when indented structures, such as functions or classes, are completed.

>>> b = ('example',
...
... 'value')
>>> def test():
...     return b * a
...

With the code in place, we just need to verify that its output matches what's expected. In keeping with the interpreter format, output is presented beneath one or more lines of input code. The exact formatting of the output will depend on the code being executed, but it's the same as you'd see when typing the code into the interpreter directly.

>>> a
2
>>> b

('example', 'value')
>>> test()
('example', 'value', 'example', 'value')

In these examples, the output string is equivalent to passing the return value from the expression into the built-in repr() function. Therefore, strings will always be quoted, and many specific types will have a different format than if you print them directly. Testing the output of str() can be achieved simply by calling str() in the line of code. Alternatively, the print() function is also supported and works just as you'd expect.

>>> for value in test():
...     print(value)
example
value
example
value

In examples like this, all lines of the output are checked against what was actually returned or printed by the code provided. This provides a very readable way to deal with sequences, as shown here. For longer sequences, as well as situations where output is allowed to change from one run to another, output may also include three periods as ellipses, indicating a place where additional content should be ignored.

>>> for value in test():
...     print(value)
example
...
value

This form is particularly useful when testing exceptions, because the interpreter output includes file paths, which will nearly always change from one system to another, and aren't relevant to most tests. In these cases, what's important to test is that the exception is raised, that it's the correct type and that its value, if any, is correct.

>>> for value in test:
...     print(value)
Traceback (most recent call last):
  ...
TypeError: 'function' object is not iterable

As the output format here suggests, the doctest will verify the first and last lines of the exception output, while ignoring the entire traceback in between. Since the traceback details are typically irrelevant to the documentation as well, this format is also much more readable.

Since the tests are meant to be built into documentation, there needs to be a way to make sure that only the tests get executed. In order to distinguish between the two without interrupting the flow of documentation, tests are set aside by nothing more than an extra newline. You'd always have to use one newline to avoid them all running together on a single line, so adding an extra simply leaves one blank line between the two.

"""
This is an example of placing documentation alongside tests in a single string.

>>> print 'Hello, world!'
'Hello, world!'

Additional documentation can be placed between snippets of code, and it won't
disturb the behavior or validity of the tests.
"""

The actual execution of doctests is provided by the doctest module. In the simplest form, you can run a single function to test an entire module. This is useful when writing a set of tests for a file that was already written because you can easily test the file individually after writing new tests. Simply import doctest and run its testmod() function to test the module. Here's an example module that contains a couple types of doctests.

def times2(value):
    """
    Multiplies the provided value by two. Because input objects can override
    the behavior of multiplication, the result can be different depending on
    the type of object passed in.

    >>> times2(5)
    10
    >>> times2('test')
    'testtest'
    >>> times2(('a', 1))
    ('a', 1, 'a', 1)
    """
    return value * 2

if __name__ == '__main__':
    import doctest
    doctest.testmod()

The docstring in times2() function includes tests and, because it's available as a module-level function, the testmod() can see it and execute the tests. This simple construct allows you to call the module directly from the command line and see the results of all doctests in the module. For example, if this module was called times2.py, you could invoke it from the command line as follows.

$ python times2.py
$

By default, the output only contains errors and failures, so if all the tests pass, there won't be any output at all. Failures are reported on individual tests, with each input/output combination being considered a unique test. This provides fine-grained details about the nature of the tests that were attempted and how they failed. If the final line in the example doctest were to read just ('a', 1) instead, here's what would happen.

$ python times2.py
**********************************************************************
File "...", line 11, in __main__.times2
Failed example:
    times2((a, '1'))
Expected:
    (a, '1')
Got:
    (a, '1', a, '1')
**********************************************************************
1 items had failures:
   1 of   3 in __main__.times2
***Test Failed*** 1 failures.
$

When working with more complicated applications and frameworks, though, the simple input/output paradigm of doctests breaks down fairly quickly. In those situations, Python provides a more powerful alternative, the unittest module.

The starting point for most test cases is the setUp() method, which you can define to perform some tasks at the start of all the tests that will be defined on the class. Common setup tasks include defining static values that will be compared later, opening connections to databases, opening files and loading data to analyze.

This method takes no arguments and doesn't return anything. If you need to control its behavior with any parameters, you'll need to define those in a way that setUp() can access without them being passed in as arguments. A common technique is to check os.environ for specific values that affect the behavior of the tests. Another option is to have a customizable settings modules that can be imported in setUp(), which can then modify the test behavior.

Likewise, any values that setUp() defines for later use can't be returned using the standard value. Instead, they can be stored on the TestCase object itself, which will be instantiated prior to running setUp(). The next section will show that individual tests are defined as methods on that same object, so any attributes stored during setup will be available for use by the tests when they execute.

import unittest

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

With the setup in place, you can write some tests to verify whatever behavior you're working with. Like setUp(), these are implemented as custom methods on your test case class. Unlike setUp(), though, there's no single specific method that must implement all the tests. Instead, the test framework will look at your test case class for any methods whose names begin with the word test.

For each method that it finds, the test framework executes setUp() before executing the test method. This helps ensure that each method can rely on a consistent environment, regardless of how many methods there are, what they each do or in what order they're executed. Completely ensuring consistency requires one other step, but that will be covered in the next section.

When writing the body of a test method, the TestCase class offers some utility methods to describe how your code is supposed to work. These are designed in such a way that each represents a condition that must be true in order to continue. There are several of these methods, with each covering a specific type of assertion. If the given assertion passes, the test will continue to the next line of code; otherwise, the test halts immediately and a failure message will be generated. Each method provides a default message to use in case of a failure but also accepts an argument to customize that message.

These functions alone provide a basic palette for the rest of your tests. To start converting the earlier doctest to a unit test, we can start by providing a testNumber() method to simulate the first test that was performed previously. Like doctests, the unittest module also provides a simple function to run all the tests found in the given module; this time, it's called main().

import unittest
import times2

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

    def testNumber(self):
        self.assertTrue(times2.times2(5) == 10)

if __name__ == '__main__':
    unittest.main()

Tests are typically stored in a module called tests.py. After saving this file, we can execute it just like the doctest example shown previously.

$ python tests.py
.
----------------------------------------------------------------------
Ran 1 test in 0.001s

Unlike doctests, unit testing does show some statistics by default. Each period represents a single test that was run, so complex applications with dozens, hundreds or even thousands of tests can easily fill several screens with results. Failures and errors are also represented here, using E for errors and F for failures. In addition, each failure will produce a block of text to describe what went wrong. Look what happens when we change the test expression.

import unittest
import times2

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

    def testNumber(self):
        self.assertTrue(times2.times2(5) == 42

if __name__ == '__main__':
    unittest.main()
$ python tests.py
F
======================================================================
FAIL: testNumber (__main__.MultiplicationTests)
----------------------------------------------------------------------
Traceback (most recent call last):

File "tests.py", line 9, in testNumber
    self.assertTrue(times2(5) == 42)
AssertionError: False is not True
----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

As you can see, it shows exactly which test method generated the failure, with a traceback to help track down the code flow that led to the failure. In addition, the failure itself is shown as an AssertionError, with the assertion shown plainly.

In this case, though, the failure message isn't as useful as it could be. All it reports is that False is not True. That's a correct report, of course, but it doesn't really tell the whole story. In order to better track down what went wrong, it'd be useful to know what the function actually returned.

To provide more information about the values involved, we'll need to use a test method that can identify the different values individually. If they're not equal, the test fails just like the standard assertion, but the failure message can now include the two distinct values so you can see how they're different. That can be a valuable tool in determining how and where the code went wrong—which is, after all, the whole point of testing.

With assertEqual() available, we can change testNumber() to produce a more useful message in the event that the assertion fails.

import unittest
import times2

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

    def testNumber(self):
        self.assertEqual(times2.times2(5), 42)

if __name__ == '__main__':
    unittest.main()

$ python tests.py
F
======================================================================
FAIL: testNumber (__main__.MultiplicationTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests.py", line 9, in testNumber
    self.assertEqual(times2(5), 42)
AssertionError: 10 != 42
----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

Behind the scenes, assertEqual() does a couple interesting things to be as flexible and powerful as possible. First, by using the == operator, it can compare the two objects using whatever more efficient method the objects themselves may define. Second, the formatting of the output can be configured by supplying a custom comparison method. Several of these customized methods are provided in the unittest module.

In addition to these methods provided out of the box, you can add your own to the test framework, so that assertEqual() can more effectively work with your own types. By passing a type and a comparison function into the addTypeEqualityFunc() method, you can register it for use with assertEqual() later on.

Using addTypeEqualityFunc() effectively can be tricky, because it's valid for the entire test case class, no matter how many tests there may be inside it. It may be tempting to add the equality function in the setUp() method, but remember that setUp() gets called once for each test method that was found on the TestCase class. If the equality function will be registered for all tests on that class, there's no point registering it before each one.

A better solution would be to add the addTypeEqualityFunc() call to the __init__() method of the test case class. This also has the additional benefit that you can subclass your own test case class to provide a more suitable base for other tests to work with. That process is explained in more detail later in this chapter.

Beyond simple equality, unittest.TestCase includes a few other methods that can be used to compare two values. Aimed primarily at numbers, these address the question of whether a tested value is less than or greater than what was expected.

Sequences present an interesting challenge because they're made up of multiple individual values. Any value in a sequence could determine the success or failure of a given test, so it's necessary to have tools to work with them specifically. First, there are two methods designed for strings, where simple equality may not always be sufficient.

More generally, tests for sequences need to make sure that certain items are present in the sequence in order to pass. The equality methods shown previously will only work if the entire sequence must be equal. In the event that some items in the sequence are important but the rest can be different, we'll need to use some other methods to verify that.

So far, all the test methods have taken a positive approach, where the test verifies that a successful outcome really is successful. It's just as important to verify unsuccessful outcomes, though, because they still need to be reliable. Many functions are expected to raise exceptions in certain situations, and unit testing is just as useful in verifying that behavior.

In our times2 example, there are many types of values that can't be multiplied by an integer. Those situations can be part of the explicit behavior of the function, as long as they're handled consistently. The typical response would be to raise a TypeError, as Python does by default. Using the assertRaises() method, we can test for this as well.

import unittest
import times2

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

    def testNumber(self):
        self.assertEqual(times2.times2(5), 42)

    def testInvalidType(self):
        self.assertRaises(TypeError, times2.times2, {})

Some situations are a bit more complicated, which can cause difficulties with testing. One common example is an object that overrides one of the standard operators. You could call the overridden method by name, but it would be more readable to simply use the operator itself. Unfortunately, the normal form of assertRaises() requires a callable, rather than just an expression.

To address this, both of these methods can act as context managers using a with block. In this form, you don't supply a callable or arguments, but rather just pass in the exception type and, if using assertRaisesRegexp(), a regular expression. Then, in the body of the with block, you can add the code that must raise the given exception. This can also be more readable than the standard version, even for situations that wouldn't otherwise require it.

import unittest
import times2

class MultiplicationTestCase(unittest.TestCase):
    def setUp(self):
        self.factor = 2

    def testNumber(self):
        self.assertEqual(times2.times2(5), 42)

    def testInvalidType(self):
        with self.assertRaises(TypeError):
            times2.times2({})

The last group contains methods for testing the identity of objects. Rather than just checking to see if their values are equivalent, these methods check to see if two objects are in fact the same. One common scenario for this test is when your code caches values for use later. By testing for identity, you can verify that a value returned from cache is the same value that was placed in the cache to begin with, rather than simply an equivalent copy.

Just as setUp() gets called before each individual test is carried out, the TestCase object also calls a tearDown() method to clean up any initialized values after testing is carried out. This is used quite often in tests that need to create and store information outside of Python during testing. Examples of such information are database rows and temporary files. Once the tests are complete, that information is no longer necessary, so it makes good sense to clean up after they've completed.

Typically, a set of tests that works with files will have to create temporary files along the way, to verify that they get accessed and modified properly. These files can be created in setUp() and deleted in tearDown(), ensuring that each test has a fresh copy when it runs. The same can be done with databases or other data structures.

Another powerful technique available when creating a testing class is the ability to change how the tests themselves are performed. The most obvious way to do this is to override the existing assertion methods, which can change how those tests are performed. There are a few other ways to alter the standard behavior, without overriding the assertion methods.

These additional overrides can be managed in the __init__() method of your custom class because, unlike setUp(), the __init__() method will only be called once per TestCase object. That makes it good for those customizations that need to affect all tests but won't be affected by any of the tests as they run. One such example, mentioned previously in this chapter, is the ability to add custom equality comparison methods, which are registered with the addTypeEqualityFunc() method.

Another modification you can make to the test class is to define what type of exception is used to identify failures. Normally, all test failures raise an AssertionError behind the scenes—the same exception used when an assert statement fails. If you need to change that for any reason, such as to better integrate with a higher-level testing framework, you can assign a new exception type to the failureException class attribute.

As a side-effect of using the failureException attribute to generate failures, you can raise it explicitly using self.failureException to generate a test failure. This is essentially the same as simply calling self.fail(), but it can be more readable in some cases to raise an exception rather than call a method.