Planning for Flexibility

Planning parameter names, order, and default values is particularly important for functions intended to be called by someone who didn’t write them, such as those in distributed applications. If you don’t know the exact needs of the users who will eventually be using your code, it’s best to move any assumptions you may have into arguments that can be overridden later.

As an extremely simple example, consider a function that appends a prefix to a string:

The 'pro_' prefix here may make sense for the application it was written for, but what happens when anything else wants to use it? Right now, the prefix is hard-coded into the body of the function itself, so there’s no available alternative. Moving that assumption into an argument makes for an easy way to customize the function later:

The function call without the prefix argument doesn’t need to change, so existing code works just fine. The section on preloading arguments later in this chapter shows how even the prefix can be changed and still be used by code that doesn’t know about it.

Of course, this example is far too simple to provide much real-world value, but the functions illustrated throughout the rest of this book will take advantage of plenty of optional arguments, showing their value in each situation.

Variable Positional Arguments

Most functions are designed to work on a specific set of arguments, but some can handle any number of arguments, acting on each in turn. These may be passed into a single argument as a tuple, list, or other iterable.

Take a typical shopping cart, for example. Adding items to the cart could be done one at a time or in batches. Using a definition of a class, with a function inside, here’s how it could be done, using a standard argument:

That would certainly do the trick, but now consider what that means for all the code that has to call it. The common case would be to add just a single item, but as the function always accepts a list, it would end up looking something like this:

So we’ve basically sabotaged the majority case in order to support the minority. Worse yet, if add_to_cart() originally supported just one item and was changed to support multiples, this syntax would break any existing calls, requiring you to rewrite them just to avoid a TypeError.

Ideally, the method should support the standard syntax for single arguments, while still supporting multiple arguments. By adding an asterisk before an argument name, you can specify that that all remaining positional arguments are collected into one tuple bound to the argument prefixed with an asterisk that didn’t get assigned to anything before it. In this case there are no other arguments, so variable positional arguments can make up the entire argument list:

Now, the method can be called with any number of positional arguments rather than having to group those arguments first into a tuple or list. The extra arguments are bundled in a tuple automatically before the function starts executing. This cleans up the common case, while still enabling more arguments as needs require. Here are a few examples of how the method could be called:

There is still one more way to call this function that allows the calling code to support any number of items as well, but it’s not specific to functions that are designed to accept variable arguments. See the section on invoking functions with variable arguments for all the details.

Variable Keyword Arguments

Functions may need to take extra configuration options, particularly if passing those options to some other library further down the line. The obvious approach would be to accept a dictionary, which can map configuration names to their values:

Unfortunately, that ends up with a problem similar to the one we encountered with positional arguments described in the previous section. The simple case in which you only override one or two values gets fairly complicated. Here are two ways the function call could look, depending on preference:

Of course, this approach doesn’t scale any prettier than the list provided in the positional argument problem from the previous section. Also, like the previous problem, this can be problematic. If the function you’re working with were previously set up to accept some explicit keyword arguments, the new dictionary argument would break compatibility.

Combining Different Kinds of Arguments

This same ordering can be used when calling functions, but it has one shortcoming. In this example, you’d have to supply a value for editable as a positional argument in order to pass in any children at all. It’d be better to be able to supply them right after the name, avoiding the optional editable argument entirely most of the time.

To support this, Python also allows variable positional arguments to be placed among standard arguments. Both required and optional arguments can be positioned after the variable argument, but now they must be passed by keyword. All the arguments are still available, but the less common ones become more optional when not required and more explicit when they do make sense.

An added feature of this behavior is that explicit arguments placed after variable positional arguments can still be required. The only real difference between the two types of placement is the requirement of using keyword arguments; whether the argument requires a value still depends on whether you define a default argument:

In fact, remember that the ordering requirements of required and optional arguments are solely intended for the case of positional arguments. With the ability to define arguments as being keyword-only, you’re now free to define them as required and optional in any order, without any complaints from Python. Ordering isn’t important when calling the function, so it’s also not important when defining the function. Consider rewriting the previous example to require the prefix as a keyword argument, while also making the delimiter optional:

In all cases, however, variable keyword arguments must be positioned at the end of the list, after all other types of arguments.

Invoking Functions with Variable Arguments

In addition to being able to define arguments that can accept any number of values, the same syntax can be used to pass values into a function call. The big advantage to this is that it’s not restricted to arguments that were defined to be variable in nature. Instead, you can pass variable arguments into any function, regardless of how it was defined. The * unpacks an iterable and passes its contents as separate arguments.

The same asterisk ( * ) notation is used to specify variable arguments, which are then expanded into a function call as if all the arguments were specified directly. A single asterisk specifies positional arguments, while two asterisks specify keyword arguments. This is especially useful when passing in the return value of a function call directly as an argument, without assigning it to individual variables first:

This example seems obvious on its own, because it’s a variable argument being passed in to a variable argument, but the same process works just fine on other types of functions as well. Because the arguments get expanded before getting passed to the function, it can be used with any function, regardless of how its arguments were specified. It can even be used with built-in functions and those defined by extensions written in C.

Passing Arguments

When you start adding a number of arguments to function calls, many of which are optional, it becomes fairly common to know some of the argument values that will need to be passed, even if it’s still long before the function will actually be called. Rather than having to pass in all the arguments at the time the call is made, it can be quite useful to apply some of those arguments in advance, so fewer can be applied later.

This concept is officially called partial application of the function, but the function doesn’t get called at all yet, so it’s really more a matter of preloading some of the arguments in advance. When the preloaded function is called later, any arguments passed along are added to those that were provided earlier.

This behavior is provided as part of the built-in functools module, by way of its partial() function. By passing in a callable and any number of positional and keyword arguments, it will return a new callable that can be used later to apply those arguments:

This is commonly used to customize a more flexible function into something simpler, so it can be passed into an API that doesn’t know how to access that flexibility. By preloading the custom arguments beforehand, the code behind the API can call your function with the arguments it knows how to use, but all the arguments will still come into play.

Introspection

Python is very transparent, allowing code to inspect many aspects of objects at runtime. Because functions are objects like any others, there are several things that your code can glean from them, including the function signature, which specifies parameters. Obtaining a function’s arguments directly requires going through a fairly complicated set of attributes that describe Python’s bytecode structures, but thankfully Python also provides some functions to make it easier.

To better illustrate what values are present in each part of the tuple, here’s how it maps out to a basic function declaration:

Example: Identifying Argument Values

Sometimes it can be useful to log what arguments a function will receive, regardless of which function it is or what its arguments look like. This behavior often comes into play in systems that generate argument lists based on something other than a Python function call. Some examples include instructions from a template language and regular expressions that parse text input.

Unfortunately, positional arguments present a bit of a problem because their values don’t include the name of the argument they’ll be sent to. Default values also pose a problem because the function call doesn’t need to include any values at all. Because the log should include all the values that will be given to the function, both of these problems will need to be addressed.

First, the easy part. Any argument values passed by keyword don’t need to be matched up with anything manually, as the argument names are provided right with the values. Rather than concerning ourselves with logging at the outset, let’s start with a function to get all the arguments in a dictionary that can be logged. The function accepts a function, a tuple of positional arguments, and a dictionary of keyword arguments:

That really was easy. The function makes a copy of the keyword arguments instead of just returning it directly because we’ll be adding entries to that dictionary soon enough. Next, we have to deal with positional arguments. The trick is to identify which argument names map to the positional argument values, so that those values can be added to the dictionary with the appropriate names. This is where inspect.getfullargspec() comes into play, using zip() to do the heavy lifting:

Now that the positional arguments have been dealt with, let’s move on to figuring out default values. If there are any default values that weren’t overridden by the arguments provided, the defaults should be added to the argument dictionary, as they will be sent to the function:

Because optional arguments must come after required arguments, this addition uses the size of the defaults tuple to determine the names of the optional argument. Looping over them, it then assigns only those values that weren’t already provided. Unfortunately, this is only half of the default value situation. Because keyword-only arguments can take default values as well, getfullargspec() returns a separate dictionary for those values:

Because default values for keyword-only arguments also come in dictionary form, it’s much easier to apply those because the argument names are known in advance. With that in place, get_arguments() can produce a more complete dictionary of arguments that will be passed to the function. Unfortunately, because this returns a dictionary and variable positional arguments have no names, there’s no way to add them to the dictionary. This limits its usefulness a bit, but it’s still valid for a great many function definitions.

Example: A More Concise Version

The previous example is certainly functional, but it’s a bit more code than is really necessary. In particular, it takes a fair amount of work supplying default values when explicit values aren’t provided. That’s not very intuitive, however, because we usually think about default values the other way around: they’re provided first, then overridden by explicit arguments.

The get_arguments() function can be rewritten with that in mind by bringing the default values out of the function declaration first, before replacing them with any values passed in as actual arguments. This avoids a lot of the checks that have to be made to make sure nothing gets overwritten accidentally.

The first step is to get the default values out. Because the defaults and kwonlydefaults attributes of the argument specification are set to None if no default values were specified, we actually have to start by setting up an empty dictionary to update. Then the default values for positional arguments can be added in.

Because this only needs to update a dictionary this time, without regard for what might be in it already, it’s a bit easier to use a different technique to get the positional defaults. Rather than using a complex slice that’s fairly difficult to read, we can use a similar zip() to what was used to get the explicit argument values. By first reversing the argument list and the default values, they still match up starting at the end:

Adding default values for keyword arguments is much easier because the argument specification already supplies them as a dictionary. We can just pass that straight into an update() of the argument dictionary and move on:

Now all that’s left is to add the explicit argument values that were passed in. The same techniques used in the earlier version of this function will work here, with the only exception being that keyword arguments are passed in an update() function instead of being copied to form the argument dictionary in the first place:

With that, we have a much more concise function that works the way we normally think of default argument values. This type of refactoring is fairly common after you get more familiar with the advanced techniques available to you. It’s always useful to look over old code to see if there’s an easier, more straightforward way to go about the task at hand. This will often make your code faster as well as more readable and maintainable going forward. Now we'll extend our solution to also validate arguments.

Example: Validating Arguments

Unfortunately, that doesn’t mean that the arguments returned by get_arguments() are capable of being passed into the function without errors. As it stands, get_arguments() assumes that any keyword arguments supplied are in fact valid arguments for the function, but that isn’t always the case. In addition, any required arguments that didn’t get a value will cause an error when the function is called. Ideally, we should be able to validate the arguments as well.

We can start with get_arguments() , so we have a dictionary of all the values that will be passed to the function, then we have two validation tasks: make sure all arguments have values and make sure no arguments were provided that the function doesn’t know about. The function itself may impose additional requirements on the argument values, but as a generic utility, we can’t make any assumptions about the content of any of the provided values.

Let’s start off with making sure all the necessary values were provided. We don’t have to worry as much about required or optional arguments this time around, since get_arguments() already makes sure optional arguments have their default values. Any argument left without a value is therefore required:

With the basics in place to validate that all required arguments have values, the next step is to make sure the function knows how to deal with all the arguments that were provided. Any arguments passed in that aren’t defined in the function should be considered an error:

Of course, because this relies on get_arguments() , it inherits the same limitation of variable positional arguments. This means validate_arguments() may sometimes return an incomplete dictionary of errors. Variable positional arguments present an additional challenge that can’t be addressed with this function. A more comprehensive solution is provided in the section on function annotations.

Closures

Despite their usefulness, closures can seem to be an intimidating topic. Most explanations assume prior knowledge of things such as lexical scope, free variables, upvalues, and variable extent. Also, because so much can be done without ever learning about closures, the topic often seems mysterious and magical, as if it’s the domain of experts, unsuitable for the rest of us. Thankfully, closures really aren’t as difficult to understand as the terminology may suggest.

In a nutshell, a closure is a function that’s defined inside another function but is then passed outside that function where it can be used by other code. There are some other details to learn as well, but it’s still fairly abstract at this point, so here’s a simple example of a closure:

As you can see, when you call multiply_by() with a value to use as a multiplication factor, the inner multiply() gets returned to be used later on. Here’s how it would actually be used, which may help explain why this is useful. If you key in the previous code line by line from a Python prompt, the following would give you an idea about how this works:

This behavior looks a bit like the argument preloading feature of functools.partial(), but you don’t need to have a function that takes both arguments at once. The interesting part of about how this works, however, is that the inner function doesn’t need to accept a factor argument of its own; it essentially inherits that argument from the outer function.

The fact that an inner function can reference the values of an outer function often seems perfectly normal when looking at the code, but there are a couple of rules about how it works that might not be as obvious. First, the inner function must be defined within the outer function; simply passing in a function as an argument won’t work:

On the surface, this looks mostly equivalent to the working example shown previously, but with the added benefit of being able to provide a callable at runtime. After all, the inner function gets placed inside the outer function and gets returned for use by other code. The problem is that closures only work if the inner function is actually defined inside the outer function, not just anything that gets passed in:

This almost contradicts the functionality of functools.partial(), which works much like the custom_operator() function described here, but remember that partial() accepts all of the arguments at the same time as it accepts the callable to be bundled with them. It doesn’t try to pull in any arguments from anywhere else.

Wrappers

Closures come into play heavily in the construction of wrappers , the most common use of decorators. Wrappers are functions designed to contain another function, adding some extra behavior before or after the wrapped function executes. In the context of the closure discussion, a wrapper is the inner function, while the wrapped function is passed in as an argument to the outer function. Here’s the code behind the suppress_errors() decorator shown in the previous section:

A few things are going on here, but most of them have already been covered. The decorator takes a function as its only argument, which isn’t executed until the inner wrapper function executes. By returning the wrapper instead of the original function, we form a closure, which allows the same function name to be used even after suppress_errors() is done.

Because the wrapper has to be called as if it were the original function, regardless of how that function was defined, it must accept all possible argument combinations. This is achieved by using variable positional and keyword arguments together and passing them straight into the original function internally. This is a very common practice with wrappers because it allows maximum flexibility, without caring what type of function it’s applied to.

The actual work in the wrapper is quite simple: just execute the original function inside a try/except block to catch any exceptions that are raised. In the event of any errors it just continues merrily along, implicitly returning None instead of doing anything interesting. It also makes sure to return any value returned by the original function, so that everything meaningful about the wrapped function is maintained.

In this case the wrapper function is fairly simple, but the basic idea works for many more complex situations as well. There could be several lines of code both before and after the original function is called, perhaps with some decisions about whether it is called at all. Authorization wrappers, for instance, will typically return or raise an exception without ever calling the wrapped function, if the authorization failed for any reason.

Unfortunately, wrapping a function means some potentially useful information is lost. Chapter 5 shows how Python has access to certain attributes of a function, such as its name, docstring, and argument list. By replacing the original function with a wrapper, we’ve actually replaced all of that other information as well. In order to bring some of it back, we turn to a decorator in the functools module called wraps.

It may seem odd to use a decorator inside a decorator, but it really just solves the same problem as anything else: there’s a common need that shouldn’t require duplicate code everywhere it takes place. The functools.wraps() decorator copies the name, docstring, and some other information over to the wrapped function, so at least some of it gets retained. It does not copy over the argument list, but it’s better than nothing:

What may seem most odd about this construction is that functools.wraps() takes an argument in addition to the function to which it’s applied. In this case, that argument is the function to copy attributes from, which is specified on the line with the decorator itself. This is often useful for customizing decorators for specific tasks, so next we’ll examine how to take advantage of custom arguments in your own decorators.

Decorators with Arguments

Not all of that will happen for every decorator, but that’s the general approach of the most complex scenarios. Anything more complicated is simply an expansion of one of those four steps. As you’ll notice, three of the four have already been covered, so the extra layer imposed by decorator arguments is really the only thing left to discuss.

This new outermost function accepts all the arguments for the decorator, optionally validates them, and returns a new function as a closure over the argument variables. That new function must take a single argument, functioning as the decorator. Here’s how the suppress_errors() decorator might look if it instead accepted a logger function to report the errors to, rather than completely silencing them:

This layering allows suppress_errors() to accept arguments prior to being used as a decorator, but it removes the ability to call it without any arguments. Because that was the previous behavior, we’ve now introduced a backward incompatibility. The closest we can get to the original syntax is to actually call suppress_errors() first, but without any arguments.

Here’s an example function that processes updates files in a given directory. This is a task that’s often performed on an automated schedule, so that if something goes wrong, it can just stop running and try again at the next appointed time:

Unfortunately, this is still a strange situation to end up with, and it really doesn’t look like anything that Python programmers are used to. Clearly, we need a better solution.

Decorators with—or without—Arguments

Ideally, a decorator with optional arguments would be able to be called without parentheses if no arguments are provided, while still being able to provide the arguments when necessary. This means supporting two different flows in a single decorator, which can get tricky if you’re not careful. The main problem is that the outermost function must be able to accept arbitrary arguments or a single function, and it must be able to tell the difference between the two and behave accordingly.

That brings us to the first task: determining which flow to use when the outer function is called. One option would be to inspect the first positional argument and check to see if it’s a function, since decorators always receive the function as a positional argument.

Interestingly, a pretty good distinction can be made based on something mentioned briefly in the previous paragraph. Decorators always receive the decorated function as a positional argument, so we can use that as its distinguishing factor. For all other arguments we can instead rely on keyword arguments, which are generally more explicit anyway, thus making it more readable as well.

We could do this by way of using *args and **kwargs, but because we know the positional argument list is just a fixed single argument, it’s easier to just make that the first argument and make it optional. Then, any additional keyword arguments can be placed after it. They’ll all need default values, of course, but the whole point here is that all arguments are optional, so that’s not a problem.

With the argument distinction squared away, all that’s left is to branch into a different code block if arguments are provided, rather than a function to be decorated. By having an optional first positional argument, we can simply test for its presence to determine which branch to go through:

This now allows suppress_errors() to be called with or without arguments, but it’s still important to remember that arguments must be passed with keywords. This is an example in which an argument looks identical to the function being decorated. There’s no way to tell the difference by examining them, even if we tried.

If a logger function is provided as a positional argument, the decorator will assume it’s the function to be decorated, so it’ll actually execute the logger immediately, with the function to be decorated as its argument. In essence, you’ll end up logging the function you wanted to decorate. Worse yet, the value you’re left with after decorating the function is actually the return value from the logger, not the decorator. Because most loggers don’t return anything, it’ll probably be None—that’s right, your function has vanished. Given that you keyed in the aforementioned functions, you can try the following from a prompt:

This is a side effect of the way the decorator works, and there’s little to be done other than documenting it and making sure you always specify keywords when applying arguments.

Example: Memoization

To demonstrate how decorators can copy out common behavior into any function you like, consider what could be done to improve the efficiency of deterministic functions. Deterministic functions always return the same result given the same set of arguments, no matter how many times they’re called. Given such a function, it should be possible to cache the results of a given function call so if it’s called with the same arguments again, the result can be looked up without having to call the function again.

Using a cache, a decorator can store the result of a function using the argument list as its key. Dictionaries can’t be used as keys in a dictionary, so only positional arguments can be taken into account when populating the cache. Thankfully, most functions that would take advantage of memoization are simple mathematical operations, which are typically called with positional arguments anyway:

Now, whenever you define a deterministic function, you can use the memoize() decorator to automatically cache its result for future use. Here’s how it would work for some simple mathematical operations. Again, given you keyed in the aforelisted stub, try the following:

Example: A Decorator to Create Decorators

Astute readers will have noticed something of a contradiction in the descriptions of the more complex decorator constructs. The purpose of decorators is to avoid a lot of boilerplate code and simplify functions, but the decorators themselves end up getting quite complicated just to support features such as optional arguments. Ideally, we could put that boilerplate into a decorator as well, simplifying the process for new decorators.

Because decorators are Python functions, just like those they decorate, this is quite possible. As with the other situations, however, there’s something that needs to be taken into account. In this case, the function you define as a decorator will need to distinguish between the arguments meant for the decorator and those meant for the function it decorates:

With these arguments in mind, here’s how you might define the suppress_errors() decorator described previously in this chapter:

Example: Type Safety

To illustrate how annotations can be used by a library, consider a basic implementation of a type safety library that can understand and utilize the function described previously. It would expect argument annotations to specify a valid type for any incoming arguments, while the return annotation would be able to validate the value returned by the function.

Because type safety involves verifying values before and after the function is executed, a decorator is the most suitable option for the implementation. Also, because all of the type-hinting information is provided in the function declaration, we don’t need to worry about any additional arguments, so a simple decorator will suffice. The first task, however, is to validate the annotations themselves, as they must be valid Python types in order for the rest of the decorator to work properly:

So far this doesn’t do anything to the function, but it does check to see that each annotation provided is a valid type, which can then be used to verify the type of the arguments referenced by the annotations. This uses isinstance() , which compares an object to the type it’s expected to be. More information on isinstance() and on types and classes in general can be found in Chapter 4.

Now that we can be sure all the annotations are valid, it’s time to start validating some arguments. Given how many types of arguments there are, let’s take them one at a time. Keyword arguments are the easiest to start out with, since they already come with their name and value tied together, so that’s one less thing to worry about. With a name, we can get the associated annotation and validate the value against that. This would also be a good time to start factoring out some things, as we’ll end up having to use some of the same things over and over again. Here’s how the wrapper would look to begin with:

By now, this should be fairly self-explanatory. Any keyword arguments provided will be checked to see if there’s an associated annotation. If there is, the provided value is checked to make sure it’s an instance of the type found in the annotation. The error message is factored out because it’ll get reused a few more times before we’re done.

Next up is dealing with positional arguments. Once again, we can rely on zip() to line up the positional argument names with the values that were provided. Because the result of zip() is compatible with the items() method of dictionaries, we can actually use chain() from the itertools module to link them together into the same loop:

Although that takes care of both positional and keyword arguments, it’s not everything. Because variable arguments can also accept annotations, we have to account for argument values that don’t line up as nicely with defined argument names. Unfortunately, there’s something else that must be dealt with before we can do much of anything on that front.

If you’re paying really close attention, you might notice a very subtle bug in the code as it stands. In order to make the code a bit easier to follow and to account for any arguments that are passed by keywords, the wrapper iterates over the kwargs dictionary in its entirely, checking for associated annotations. Unfortunately, that leaves us with the possibility of an unintentional name conflict.

To illustrate how the bug could be triggered, first consider what would be expected when dealing with variable arguments. Because we can only apply a single annotation to the variable argument name itself, that annotation must be assumed to apply to all arguments that fall under that variable argument, whether passed positionally or by keyword. Without explicit support for that behavior yet, variable arguments should just be ignored, but here’s what happens with the code as it stands:

Interestingly, everything works fine unless the function call includes a keyword argument with the same name as the variable positional argument. Although it may not seem obvious at first, the problem is actually in the set of values to iterate over in the wrapper’s only loop. It assumes that the names of all the keyword arguments line up nicely with annotations.

Basically, the problem is that keyword arguments that are meant for the variable argument end up getting matched up with annotations from other arguments. For the most part, this is acceptable because two of the three types of arguments won’t ever cause problems. Matching it with an explicit argument name simply duplicates what Python already does, so using the associated annotation is fine, and matching the variable keyword argument name ends up using the same annotation that we were planning on using anyway.

So the problem only crops up when a keyword argument matches the variable positional argument name, because that association never makes sense. Sometimes if the annotation is the same as that of the variable keyword argument, the problem might never show up, but it’s still there, regardless. Because the code for the wrapper function is still fairly minimal, it’s not too difficult to see where the problem is occurring.

In the main loop, the second part of the iteration chain is the list of items in the kwargs dictionary. That means everything passed in by keyword is checked against named annotations, which clearly isn’t always what we want. Instead, we only want to loop over the explicit arguments at this point, while still supporting both positions and keywords. That means we’ll have to construct a new dictionary based on the function definition, rather than taking the easy way out and relying on kwargs, as we are now. The outer typesafe() function has been removed from the listing here to make the code easier to digest in print:

With that bug out of the way, we can focus on properly supporting variable arguments. Because keyword arguments have names but positional arguments don’t, we can’t manage both types in one pass like we could with the explicit arguments. The processes are fairly similar to the explicit arguments, but the values to iterate over are different in each case. The biggest difference, however, is that the annotations aren’t referenced by the name of the arguments.

In order to loop over just the truly variable positional arguments, we can simply use the number of explicit arguments as the beginning of a slice on the positional arguments tuple. This gets us all positional arguments provided after the explicit arguments or an empty list if only explicit arguments were provided.

For keyword arguments, we have to be a bit more creative. Because the function already loops over all the explicitly declared arguments at the beginning, we can use that same loop to exclude any matching items from a copy of the kwargs dictionary. Then we can iterate over what’s left over to account for all the variable keyword arguments:

This covers all explicit arguments as well as variable arguments passed in by position and keyword. The only thing left is to validate the value returned by the target function. Thus far the wrapper just calls the original function directly without regard for what it returns, but by now, it should be easy to see what needs to be done:

With that, we have a fully functional type safety decorator, which can validate all arguments to a function as well as its return value. There’s one additional safeguard we can include to find errors even more quickly, however. In the same way as the outer typesafe() function already validates that the annotations are types, that part of the function is also capable of validating the default values for all provided arguments. Because variable arguments can’t have default values, this is much simpler than dealing with the function call itself:

Factoring Out the Boilerplate

Looking over the code as it stands, you’ll notice a lot of repetition. Each form of annotation ends up doing the same things: checking to see if the value is appropriate and raising an exception if it’s not. Ideally, we’d be able to factor that into a separate function that can focus solely on the actual task of validation. The rest of the code is really just boilerplate, managing the details of finding the different types of annotations.

Because the common code will be going into a new function, the obvious way to tie it into the rest of the code is to create a new decorator. This new decorator will be placed on a function that will process the annotation for each value, so we’ll call it annotation_processor. The function passed into annotation_processor will then be used for each of the annotation types throughout the existing code:

Now we can apply this new decorator to a much simpler function to provide a new typesafe() decorator , which functions just like the one in the previous section:

The benefit of doing this is that it’s much easier to modify the behavior of the decorator in the future. In addition, you can now use annotation_processor() to create new types of decorators that use annotation for different purposes, such as type coercion.

Example: Type Coercion

Rather than strictly requiring that the arguments all be the types specified when they’re passed into the function, another approach is to coerce them to the required types inside the function itself. Many of the same types that are used to validate values can also be used to coerce them directly into the types themselves. In addition, if a value can’t be coerced, the type it’s passed into raises an exception, usually a TypeError, just like our validation function.

The decorator presented in the previous section provides a good starting point for adding this behavior to a new decorator, and we can use it to modify the incoming value according to the annotation that was provided along with it. Because we’re relying on a type constructor to do all the necessary type checking and raise exceptions appropriately, this new decorator can be much simpler. In fact, it can be expressed in just one actual instruction:

This is so simple that it doesn’t even require the annotation be a type at all. Any function or class that returns an object will work just fine, and the value returned will be passed into the function decorated by coerce_arguments() . Or will it? If you look back at the annotation_decorator() function as it stands, there’s a minor problem that prevents it from working the way this new decorator would need it to.

The problem is that in the lines that call the process() function that was passed into the outer decorator, the return value is thrown away. If you try to use coerce_arguments() with the existing decorator, all you’ll get is the exception-raising aspect of the code, not the value coercion aspect. So, in order to work properly, we’ll need to go back and add that feature to annotation_processor().

There are a few things that need to be done overall, however. Because the annotation processor will be modifying the arguments that will be eventually sent to the decorated function, we’ll need to set up a new list for positional arguments and a new dictionary for keyword arguments. Then we have to split up the explicit argument handling, so that we can distinguish between positional and keyword arguments. Without that, the function wouldn’t be able to apply variable positional arguments correctly:

With those changes in place, the new coerce_arguments() decorator will be able to replace the arguments on the fly, passing the replacements into the original function. Unfortunately, if you’re still using typesafe() from before, this new behavior causes problems because typesafe() doesn’t return a value. Fixing that is a simple matter of returning the original value, unchanged, if the type check was satisfactory:

Annotating with Decorators

The natural question to ask is: what happens if you want to use two libraries together? One might expect you to supply valid types, whereas the other expects a string to use for documentation. They’re completely incompatible with each other, which forces you to use one or the other, rather than both. Furthermore, any attempt to merge the two, using a dictionary or some other combined data type, would have to be agreed on by both libraries, as each would need to know how to get at the information it cares about.

Once you consider how many other frameworks and libraries might take advantage of these annotations, you can see how quickly the official function annotations fall apart. It’s still too early to see which applications will actually use it or how they’ll work together, but it’s certainly worth considering other options that can bypass the problems completely.

Because decorators can take arguments of their own, it’s possible to use them to provide annotations for the arguments of the functions they decorate. This way, the annotations are separate from the function itself and provided directly to the code that makes sense of them. And because multiple decorators can be stacked together on a single function, it’s already got a built-in way of managing multiple frameworks.

Example: Type Safety as a Decorator

To illustrate the decorator-based approach to function annotations, let’s consider the type safety example from earlier. It already relied on a decorator, so we can extend that to take arguments, using the same types that the annotations provided previously. Essentially, it’ll look something like this:

It works almost exactly like the true annotated version, except that the annotations are supplied to the decorator directly. In order to accept arguments, we’re going to just change the first portion of the code a bit so that we can get the annotations from the arguments instead of inspecting the function itself.

Because annotations come in through arguments to the decorator, we have a new outer wrapper for receiving them. When the next layer receives the function to be decorated it can match up the annotations with the function’s signature, providing names for any annotations passed positionally. Once all the available annotations have been given the right names, they can be used by the rest of the inner decorator without any further modifications:

That handles most of the situation, but it doesn’t handle return values yet. If you try to supply a return value using the right name, return, you’ll get a syntax error because it’s a reserved Python keyword. Trying to provide it alongside the other annotations would require each call to pass annotations using an actual dictionary, where you can provide the return annotation without upsetting Python’s syntax.

Instead, you’ll need to provide the return value annotation in a separate function call, where it can be the sole argument without any reserved name issues. When working with most types of decorators, this would be easy to do: just create a new decorator that checks the return value and be done with it. Unfortunately, as the eventual decorator you’re working with is created outside the control of our code, it’s not so easy.

If you completely detached the return value processing from the argument processing, the programmer who’s actually writing something like the typesafe() decorator would have to write it twice; once to create the argument-processing decorator and again to create the return-value–processing decorator. Because that’s a clear violation of DRY, let’s reuse as much of their work as possible.

Unfortunately this isn’t an option, for reasons that can be demonstrated without even adding the necessary code to support it. The trouble is, this formation isn’t allowed as Python syntax. If typesafe() hadn’t taken any arguments it would work, but there’s no support for calling two separate functions as part of a single decorator. Instead of supplying the return value annotation in the decorator itself, let’s look somewhere else.

Another option is to use the generated typesafe() decorator to add a function as an attribute to the wrapper around the add() function. This places the return value annotation at the end of the function definition, closer to where the return value is specified. In addition, it helps clarify the fact that you can use typesafe() to supply argument decorators without bothering to check the return value, if you want to. Here’s how it would look:

It’s still very clear and perhaps even more explicit than the syntax that doesn’t work anyway. As an added bonus, the code to support it is very simple, requiring just a few lines be added to the end of the inner decorator() function:

Because this new returns() function will be called before the final typesafe() function ever will, it can simply add a new annotation to the existing dictionary. Then, when typesafe() does get called later, the internal wrapper can just continue working like it always did. This just changes the way the return value annotation is supplied, which is all that was necessary.

Because all of this behavior was refactored into a separate decorator, you can apply this decorator to coerce_arguments() or any other similarly purposed function. The resulting function will work the same way as typesafe(), only swapping out the argument handling with whatever the new decorator needs to do.

Identifying Object Types

Python’s dynamic nature can sometimes make it seem difficult to ensure you’re getting the right type of value or to even know what type of value it is. Python does provide some options for accessing that information, but it’s necessary to realize those are two separate tasks, so Python uses two different approaches.

The most obvious requirement is to identify what type of object your code was given. For this Python supplies its built-in type() function , which accepts an object to identify. The return value is the Python class that was used to create the given object, even if that creation was done implicitly, by way of a literal value:

Chapter 4 explains in detail what you can do with that class object once you have it, but the more common case is to compare an object against a particular type you expect to receive. This is a different situation because it doesn’t really matter exactly what type the object is. As long as the value is an instance of the right type, you can make correct assumptions about how it behaves.

There are a number of different utility functions available for this purpose, most of which are covered in Chapter 4. This section and the next chapter will make use of one of them fairly frequently, so it merits some explanation here. The isinstance() function accepts two arguments: the object to check and the type you’re expecting it to be. The result is a simple True or False, making it suitable for if blocks:

Modules and Packages

Functions and classes that are defined in Python are placed inside of modules, which in turn are often part of a package structure. Accessing this structure when importing code is easy enough, using documentation or even just peeking at the source files on disk. Given a piece of code, however, it’s often useful to identify where it was defined in the source code.

For this reason, all functions and classes have a __module__ attribute, which contains the import location of the module where the code was defined. Rather than just supplying the name of the module, the math.sin._module__ also includes the full path to where the module resides. Essentially, it’s enough information for you to pass it straight into any of the dynamic importing features shown in Chapter 2.

Working with the interactive interpreter is something of a special case because there’s no named source file to work with. Any functions or classes defined there will have the special name '__main__' returned from the __module__ attribute:

Docstrings

Because you can document your functions with docstrings included right alongside the code, Python also stores those strings as part of the function object. By accessing the __doc__ attribute of a function, you can read a docstring into code, which can be useful for generating a library’s documentation on the fly. Consider the following example, showing simple docstring access on a simple function:

This is a more complex example, with more comprehensive documentation.

As you can see, simple docstrings are easy to handle just by reading in __doc__ and using it however you need to. Unfortunately, more complex docstrings will retain all whitespace, including newlines, making them more challenging to work with. Worse yet, your code can’t know which type of docstring you’re looking at without scanning it for certain characters. Even if you’re just printing it out to the interactive prompt, you still have an extra line before and after the real documentation, as well as the same indentation as was present in the file.

To handle complex docstrings more gracefully, like the one shown in the example, the inspect module mentioned previously also has a getdoc() function , designed to retrieve and format docstrings. It strips out whitespace both before and after the documentation, as well as any indentation that was used to line up the docstring with the code around it. Here’s that same docstring again, but formatted with inspect.getdoc():

We still have to use print() at the interactive prompt because the newline character is still retained in the result string. All inspect.getdoc() strips out is the whitespace that was used to make the docstring look right alongside the code for the function. In addition to trimming the space at the beginning and end of the docstring, getdoc() uses a simple technique to identify and remove whitespace used for indentation.

Essentially, getdoc() counts the number of spaces at the beginning of each line of code, even if the answer is 0. Then it determines the lowest value of those counts and removes that many characters from each line that remains after the leading and trailing whitespace has been removed. This allows you to keep other indentation in the docstring intact, as long as it’s greater than what you need to align the text with the surrounding code. Here’s an example of an even more complex docstring, so you can see how inspect.getdoc() handles it:

Notice how the descriptions of each argument are still indented four spaces, just like they appeared to be in the function definition. The shortest lines had just four total spaces at the beginning, while those had eight, so Python stripped out the first four, leaving the rest intact. Likewise, the example interpreter session was indented by two extra spaces, so the resulting string maintains a two-space indentation.

Oh, and don’t worry too much about the copy function just yet. Chapter 6 describes in detail how to make and manage copies of objects when necessary.

Install Pandas and Matplotlib

Make a Text File of Data

Use Pandas to Display Data

Your output should be similar to the following:

Output from reading students.csv data file using Pandas.

Running Some Data Analysis

Average student age output for various majors.

Plotting with Matplotlib

Types of Charts

There are many others such as bar, hist (histogram), box, density, area, scatter, and XKCD-style charts (comic web site with Pythonish humor). The format is similar to pie.

Combine Matplotlib with Pandas

The Pandas and Matplotlib documentation and main web site of course will describe other functions available, but this will get you using Pandas features so that you can easily integrate other features you might need into your applications as needed.