When you write a try statement, a variety of clauses can appear after the try header. Table 33-1 summarizes all the possible forms—you must use at least one. We’ve already met some of these: as you know, except clauses catch exceptions, finally clauses run on the way out, and else clauses run if no exceptions are encountered.

Syntactically, there may be any number of except clauses, but you can code else only if there is at least one except, and there can be only one else and one finally. Through Python 2.4, the finally clause must appear alone (without else or except); the try/finally is really a different statement. As of Python 2.5, however, a finally can appear in the same statement as except and else (more on the ordering rules later in this chapter when we meet the unified try statement).

We’ll explore the entries with the extra as value part when we meet the raise statement. They provide access to the objects that are raised as exceptions.

The first and fourth entries in Table 33-1 are new here:

Because Python looks for a match within a given try by inspecting the except clauses from top to bottom, the parenthesized version has the same effect as listing each exception in its own except clause, but you have to code the statement body only once. Here’s an example of multiple except clauses at work, which demonstrates just how specific your handlers can be:

try:
    action()
except NameError:
    ...
except IndexError:
    ...
except KeyError:
    ...
except (AttributeError, TypeError, SyntaxError):
    ...
else:
    ...

In this example, if an exception is raised while the call to the action function is running, Python returns to the try and searches for the first except that names the exception raised. It inspects the except clauses from top to bottom and left to right, and runs the statements under the first one that matches. If none match, the exception is propagated past this try. Note that the else runs only when no exception occurs in action—it does not run when an exception without a matching except is raised.

If you really want a general “catch-all” clause, an empty except does the trick:

try:
    action()
except NameError:
    ...                   # Handle NameError
except IndexError:
    ...                   # Handle IndexError
except:
    ...                   # Handle all other exceptions
else:
    ...                   # Handle the no-exception case

The empty except clause is a sort of wildcard feature—because it catches everything, it allows your handlers to be as general or specific as you like. In some scenarios, this form may be more convenient than listing all possible exceptions in a try. For example, the following catches everything without listing anything:

try:
    action()
except:
    ...                   # Catch all possible exceptions

Empty excepts also raise some design issues, though. Although convenient, they may catch unexpected system exceptions unrelated to your code, and they may inadvertently intercept exceptions meant for another handler. For example, even system exit calls in Python trigger exceptions, and you usually want these to pass. That said, this structure may also catch genuine programming mistakes for you which you probably want to see an error message. We’ll revisit this as a gotcha at the end of this part of the book. For now, I’ll just say “use with care.”

Python 3.0 introduced an alternative that solves one of these problems—catching an exception named Exception has almost the same effect as an empty except, but ignores exceptions related to system exits:

try:
    action()
except Exception:
    ...                   # Catch all possible exceptions, except exits

This has most of the same convenience of the empty except, but also most of the same dangers. We’ll explore how this form works its voodoo in the next chapter, when we study exception classes.

The purpose of the else clause is not always immediately obvious to Python newcomers. Without it, though, there is no way to tell (without setting and checking Boolean flags) whether the flow of control has proceeded past a try statement because no exception was raised, or because an exception occurred and was handled:

try:
    ...run code...
except IndexError:
    ...handle exception...
# Did we get here because the try failed or not?

Much like the way else clauses in loops make the exit cause more apparent, the else clause provides syntax in a try that makes what has happened obvious and unambiguous:

try:
    ...run code...
except IndexError:
    ...handle exception...
else:
    ...no exception occurred...

You can almost emulate an else clause by moving its code into the try block:

try:
    ...run code...
    ...no exception occurred...
except IndexError:
    ...handle exception...

This can lead to incorrect exception classifications, though. If the “no exception occurred” action triggers an IndexError, it will register as a failure of the try block and erroneously trigger the exception handler below the try (subtle, but true!). By using an explicit else clause instead, you make the logic more obvious and guarantee that except handlers will run only for real failures in the code you’re wrapping in a try, not for failures in the else case’s action.

Because the control flow through a program is easier to capture in Python than in English, let’s run some examples that further illustrate exception basics. I’ve mentioned that exceptions not caught by try statements percolate up to the top level of the Python process and run Python’s default exception-handling logic (i.e., Python terminates the running program and prints a standard error message). Let’s look at an example. Running the following module file, bad.py, generates a divide-by-zero exception:

def gobad(x, y):
    return x / y

def gosouth(x):
    print(gobad(x, 0))

gosouth(1)

Because the program ignores the exception it triggers, Python kills the program and prints a message:

% python bad.py
Traceback (most recent call last):
  File "bad.py", line 7, in <module>
    gosouth(1)
  File "bad.py", line 5, in gosouth
    print(gobad(x, 0))
  File "bad.py", line 2, in gobad
    return x / y
ZeroDivisionError: int division or modulo by zero

I ran this in a shell window with Python 3.0. The message consists of a stack trace (“Traceback”) and the name of and details about the exception that was raised. The stack trace lists all lines active when the exception occurred, from oldest to newest. Note that because we’re not working at the interactive prompt, in this case the file and line number information is more useful. For example, here we can see that the bad divide happens at the last entry in the trace—line 2 of the file bad.py, a return statement.^[74]

Because Python detects and reports all errors at runtime by raising exceptions, exceptions are intimately bound up with the ideas of error handling and debugging in general. If you’ve worked through this book’s examples, you’ve undoubtedly seen an exception or two along the way—even typos usually generate a SyntaxError or other exception when a file is imported or executed (that’s when the compiler is run). By default, you get a useful error display like the one just shown, which helps you track down the problem.

Often, this standard error message is all you need to resolve problems in your code. For more heavy-duty debugging jobs, you can catch exceptions with try statements, or use one of the debugging tools that I introduced in Chapter 3 and will summarize again in Chapter 35 (such as the pdb standard library module).

Python’s default exception handling is often exactly what you want—especially for code in a top-level script file, an error generally should terminate your program immediately. For many programs, there is no need to be more specific about errors in your code.

Sometimes, though, you’ll want to catch errors and recover from them instead. If you don’t want your program terminated when Python raises an exception, simply catch it by wrapping the program logic in a try. This is an important capability for programs such as network servers, which must keep running persistently. For example, the following code catches and recovers from the TypeError Python raises immediately when you try to concatenate a list and a string (the + operator expects the same sequence type on both sides):

def kaboom(x, y):
    print(x + y)               # Trigger TypeError

try:
    kaboom([0,1,2], "spam")
except TypeError:              # Catch and recover here
    print('Hello world!')
print('resuming here')         # Continue here if exception or not

When the exception occurs in the function kaboom, control jumps to the try statement’s except clause, which prints a message. Since an exception is “dead” after it’s been caught like this, the program continues executing below the try rather than being terminated by Python. In effect, the code processes and clears the error, and your script recovers:

% python kaboom.py
Hello world!
resuming here

Notice that once you’ve caught an error, control resumes at the place where you caught it (i.e., after the try); there is no direct way to go back to the place where the exception occurred (here, in the function kaboom). In a sense, this makes exceptions more like simple jumps than function calls—there is no way to return to the code that triggered the error.

We saw some simple try/finally examples in the prior chapter. Here’s a more realistic example that illustrates a typical role for this statement:

class MyError(Exception): pass

def stuff(file):
    raise MyError()

file = open('data', 'w')     # Open an output file
try:
    stuff(file)              # Raises exception
finally:
    file.close()             # Always close file to flush output buffers
print('not reached')         # Continue here only if no exception

In this code, we’ve wrapped a call to a file-processing function in a try with a finally clause to make sure that the file is always closed, and thus finalized, whether the function triggers an exception or not. This way, later code can be sure that the file’s output buffer’s content has been flushed from memory to disk. A similar code structure can guarantee that server connections are closed, and so on.

As we learned in Chapter 9, file objects are automatically closed on garbage collection; this is especially useful for temporary files that we don’t assign to variables. However, it’s not always easy to predict when garbage collection will occur, especially in larger programs. The try statement makes file closes more explicit and predictable and pertains to a specific block of code. It ensures that the file will be closed on block exit, regardless of whether an exception occurs or not.

This particular example’s function isn’t all that useful (it just raises an exception), but wrapping calls in try/finally statements is a good way to ensure that your closing-time (i.e., termination) activities always run. Again, Python always runs the code in your finally blocks, regardless of whether an exception happens in the try block.^[75]

When the function here raises its exception, the control flow jumps back and runs the finally block to close the file. The exception is then propagated on to either another try or the default top-level handler, which prints the standard error message and shuts down the program; the statement after this try is never reached. If the function here did not raise an exception, the program would still execute the finally block to close the file, but it would then continue below the entire try statement.

Notice that the user-defined exception here is again defined with a class—as we’ll see in the next chapter, exceptions today must all be class instances in both 2.6 and 3.0.

When combined like this, the try statement must have either an except or a finally, and the order of its parts must be like this:

try -> except -> else -> finally

where the else and finally are optional, and there may be zero or more except, but there must be at least one except if an else appears. Really, the try statement consists of two parts: excepts with an optional else, and/or the finally.

In fact, it’s more accurate to describe the merged statement’s syntactic form this way (square brackets mean optional and star means zero-or-more here):

try:                               # Format 1
    statements
except [type [as value]]:          # [type [, value]] in Python 2
    statements
[except [type [as value]]:
    statements]*
[else:
    statements]
[finally:
    statements]

try:                               # Format 2
    statements
finally:
    statements

Because of these rules, the else can appear only if there is at least one except, and it’s always possible to mix except and finally, regardless of whether an else appears or not. It’s also possible to mix finally and else, but only if an except appears too (though the except can omit an exception name to catch everything and run a raise statement, described later, to reraise the current exception). If you violate any of these ordering rules, Python will raise a syntax error exception before your code runs.

Prior to Python 2.5, it is actually possible to combine finally and except clauses in a try by syntactically nesting a try/except in the try block of a try/finally statement (we’ll explore this technique more fully in Chapter 35). In fact, the following has the same effect as the new merged form shown at the start of this section:

try:                               # Nested equivalent to merged form
    try:
        main-action
    except Exception1:
        handler1
    except Exception2:
        handler2
    ...
    else:
        no-error
finally:
    cleanup

Again, the finally block is always run on the way out, regardless of what happened in the main action and regardless of any exception handlers run in the nested try (trace through the four cases listed previously to see how this works the same). Since an else always requires an except, this nested form even sports the same mixing constraints of the unified statement form outlined in the preceding section.

However, this nested equivalent is more obscure and requires more code than the new merged form (one four-character line, at least). Mixing finally into the same statement makes your code easier to write and read, so this is the generally preferred technique today.

Here’s a demonstration of the merged try statement form at work. The following file, mergedexc.py, codes four common scenarios, with print statements that describe the meaning of each:

sep = '-' * 32 + '
'
print(sep + 'EXCEPTION RAISED AND CAUGHT')
try:
    x = 'spam'[99]
except IndexError:
    print('except run')
finally:
    print('finally run')
print('after run')


print(sep + 'NO EXCEPTION RAISED')
try:
    x = 'spam'[3]
except IndexError:
    print('except run')
finally:
    print('finally run')
print('after run')


print(sep + 'NO EXCEPTION RAISED, WITH ELSE')
try:
    x = 'spam'[3]
except IndexError:
    print('except run')
else:
    print('else run')
finally:
    print('finally run')
print('after run')


print(sep + 'EXCEPTION RAISED BUT NOT CAUGHT')
try:
    x = 1 / 0
except IndexError:
    print('except run')
finally:
    print('finally run')
print('after run')

When this code is run, the following output is produced in Python 3.0 (actually, its behavior and output are the same in 2.6, because the print calls each print a single item). Trace through the code to see how exception handling produces the output of each of the four tests here:

c:misc> C:Python30python mergedexc.py
--------------------------------
EXCEPTION RAISED AND CAUGHT
except run
finally run
after run
--------------------------------
NO EXCEPTION RAISED
finally run
after run
--------------------------------
NO EXCEPTION RAISED, WITH ELSE
else run
finally run
after run
--------------------------------
EXCEPTION RAISED BUT NOT CAUGHT
finally run
Traceback (most recent call last):
  File "mergedexc.py", line 36, in <module>
    x = 1 / 0
ZeroDivisionError: int division or modulo by zero

This example uses built-in operations in the main action to trigger exceptions (or not), and it relies on the fact that Python always checks for errors as code is running. The next section shows how to raise exceptions manually instead.

A raise statement that does not include an exception name or extra data value simply reraises the current exception. This form is typically used if you need to catch and handle an exception but don’t want the exception to die in your code:

>>> try:
...     raise IndexError('spam')         # Exceptions remember arguments
... except IndexError:
...     print('propagating')
...     raise                            # Reraise most recent exception
...
propagating
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
IndexError: spam

Running a raise this way reraises the exception and propagates it to a higher handler (or the default handler at the top, which stops the program with a standard error message). Notice how the argument we passed to the exception class shows up in the error messages; you’ll learn why this happens in the next chapter.

Python 3.0 (but not 2.6) also allows raise statements to have an optional from clause:

raise exception from otherexception

When the from is used, the second expression specifies another exception class or instance to attach to the raised exception’s __cause__ attribute. If the raised exception is not caught, Python prints both exceptions as part of the standard error message:

>>> try:
...    1 / 0
... except Exception as E:
...    raise TypeError('Bad!') from E
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
ZeroDivisionError: int division or modulo by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
TypeError: Bad!

When an exception is raised inside an exception handler, a similar procedure is followed implicitly: the previous exception is attached to the new exception’s __context__ attribute and is again displayed in the standard error message if the exception goes uncaught. This is an advanced and still somewhat obscure extension, so see Python’s manuals for more details.

Assertions are typically used to verify program conditions during development. When displayed, their error message text automatically includes source code line information and the value listed in the assert statement. Consider the file asserter.py:

def f(x):
    assert x < 0, 'x must be negative'
    return x ** 2

% python
>>> import asserter
>>> asserter.f(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "asserter.py", line 2, in f
    assert x < 0, 'x must be negative'
AssertionError: x must be negative

It’s important to keep in mind that assert is mostly intended for trapping user-defined constraints, not for catching genuine programming errors. Because Python traps programming errors itself, there is usually no need to code asserts to catch things like out-of-bounds indexes, type mismatches, and zero divides:

def reciprocal(x):
    assert x != 0              # A useless assert!
    return 1 / x               # Python checks for zero automatically

Such asserts are generally superfluous—because Python raises exceptions on errors automatically, you might as well let it do the job for you.^[76] For another example of common assert usage, see the abstract superclass example in Chapter 28; there, we used assert to make calls to undefined methods fail with a message.

The basic format of the with statement looks like this:

with expression [as variable]:
    with-block

The expression here is assumed to return an object that supports the context management protocol (more on this protocol in a moment). This object may also return a value that will be assigned to the name variable if the optional as clause is present.

Note that the variable is not necessarily assigned the result of the expression; the result of the expression is the object that supports the context protocol, and the variable may be assigned something else intended to be used inside the statement. The object returned by the expression may then run startup code before the with-block is started, as well as termination code after the block is done, regardless of whether the block raised an exception or not.

Some built-in Python objects have been augmented to support the context management protocol, and so can be used with the with statement. For example, file objects (covered in Chapter 9) have a context manager that automatically closes the file after the with block regardless of whether an exception is raised:

with open(r'C:miscdata') as myfile:
    for line in myfile:
        print(line)
        ...more code here...

Here, the call to open returns a simple file object that is assigned to the name myfile. We can use myfile with the usual file tools—in this case, the file iterator reads line by line in the for loop.

However, this object also supports the context management protocol used by the with statement. After this with statement has run, the context management machinery guarantees that the file object referenced by myfile is automatically closed, even if the for loop raised an exception while processing the file.

Although file objects are automatically closed on garbage collection, it’s not always straightforward to know when that will occur. The with statement in this role is an alternative that allows us to be sure that the close will occur after execution of a specific block of code. As we saw earlier, we can achieve a similar effect with the more general and explicit try/finally statement, but it requires four lines of administrative code instead of one in this case:

myfile = open(r'C:miscdata')
try:
    for line in myfile:
        print(line)
        ...more code here...
finally:
    myfile.close()

We won’t cover Python’s multithreading modules in this book (for more on that topic, see follow-up application-level texts such as Programming Python), but the lock and condition synchronization objects they define may also be used with the with statement, because they support the context management protocol:

lock = threading.Lock()
with lock:
    # critical section of code
    ...access shared resources...

Here, the context management machinery guarantees that the lock is automatically acquired before the block is executed and released once the block is complete, regardless of exception outcomes.

As introduced in Chapter 5, the decimal module also uses context managers to simplify saving and restoring the current decimal context, which specifies the precision and rounding characteristics for calculations:

with decimal.localcontext() as ctx:
    ctx.prec = 2
    x = decimal.Decimal('1.00') / decimal.Decimal('3.00')

After this statement runs, the current thread’s context manager state is automatically restored to what it was before the statement began. To do the same with a try/finally, we would need to save the context before and restore it manually.

Although some built-in types come with context managers, we can also write new ones of our own. To implement context managers, classes use special methods that fall into the operator overloading category to tap into the with statement. The interface expected of objects used in with statements is somewhat complex, and most programmers only need to know how to use existing context managers. For tool builders who might want to write new application-specific context managers, though, let’s take a quick look at what’s involved.

Here’s how the with statement actually works:

Let’s look at a quick demo of the protocol in action. The following defines a context manager object that traces the entry and exit of the with block in any with statement it is used for:

class TraceBlock:
    def message(self, arg):
        print('running', arg)
    def __enter__(self):
        print('starting with block')
        return self
    def __exit__(self, exc_type, exc_value, exc_tb):
        if exc_type is None:
            print('exited normally
')
        else:
            print('raise an exception!', exc_type)
            return False                                  # Propagate

with TraceBlock() as action:
    action.message('test 1')
    print('reached')

with TraceBlock() as action:
    action.message('test 2')
    raise TypeError
    print('not reached')

Notice that this class’s __exit__ method returns False to propagate the exception; deleting the return statement would have the same effect, as the default None return value of functions is False by definition. Also notice that the __enter__ method returns self as the object to assign to the as variable; in other use cases, this might return a completely different object instead.

When run, the context manager traces the entry and exit of the with statement block with its __enter__ and __exit__ methods. Here’s the script in action being run under Python 3.0 (it runs in 2.6, too, but prints some extra tuple parentheses):

% python withas.py
starting with block
running test 1
reached
exited normally

starting with block
running test 2
raise an exception! <class 'TypeError'>
Traceback (most recent call last):
  File "withas.py", line 20, in <module>
    raise TypeError
TypeError

Context managers are somewhat advanced devices for tool builders, so we’ll skip additional details here (see Python’s standard manuals for the full story—for example, there’s a new contextlib standard module that provides additional tools for coding context managers). For simpler purposes, the try/finally statement provides sufficient support for termination-time activities.

with open('data') as fin, open('res', 'w') as fout:
    for line in fin:
        if 'some key' in line:
            fout.write(line)

with A() as a, B() as b:
    ...statements...

with A() as a:
    with B() as b:
        ...statements...