In the advanced class topics chapter of this book (Chapter 31), we met static and class methods and
took a quick look at the @
decorator
syntax Python offers for declaring them. We also met function decorators
briefly in the prior chapter (Chapter 37),
while exploring the property
built-in’s ability to serve as a decorator, and in Chapter 28 while studying the notion of abstract
superclasses.
This chapter picks up where the previous decorator coverage left off. Here, we’ll dig deeper into the inner workings of decorators and study more advanced ways to code new decorators ourselves. As we’ll see, many of the concepts we studied in earlier chapters, such as state retention, show up regularly in decorators.
This is a somewhat advanced topic, and decorator construction tends to be of more interest to tool builders than to application programmers. Still, given that decorators are becoming increasingly common in popular Python frameworks, a basic understanding can help demystify their role, even if you’re just a decorator user.
Besides covering decorator construction details, this chapter serves as a more realistic case study of Python in action. Because its examples are somewhat larger than most of the others we’ve seen in this book, they better illustrate how code comes together into more complete systems and tools. As an extra perk, much of the code we’ll write here may be used as general-purpose tools in your day-to-day programs.
Decoration is a way to specify management code for functions and classes. Decorators themselves take the form of callable objects (e.g., functions) that process other callable objects. As we saw earlier in this book, Python decorators come in two related flavors:
Function decorators do name rebinding at function definition time, providing a layer of logic that can manage functions and methods, or later calls to them.
Class decorators do name rebinding at class definition time, providing a layer of logic that can manage classes, or the instances created by calling them later.
In short, decorators provide a way to insert
automatically run code at the end of function and
class definition statements—at the end of a def
for function decorators, and at the end
of a class
for class decorators.
Such code can play a variety of roles, as described in the following
sections.
For example, in typical use, this automatically run code may be used to augment calls to functions and classes. It arranges this by installing wrapper objects to be invoked later:
Decorators achieve these effects by automatically rebinding
function and class names to other callables, at the end of def
and class
statements. When later invoked,
these callables can perform tasks such as tracing and timing
function calls, managing access to class instance attributes, and so
on.
Although most examples in this chapter deal with using wrappers to intercept later calls to functions and classes, this is not the only way decorators can be used:
Function decorators can also be used to manage function objects, instead of later calls to them—to register a function to an API, for instance. Our primary focus here, though, will be on their more commonly used call wrapper application.
Class decorators can also be used to manage class objects directly, instead of instance creation calls—to augment a class with new methods, for example. Because this role intersects strongly with that of metaclasses (indeed, both run at the end of the class creation process), we’ll see additional use cases in the next chapter.
In other words, function decorators can be used to manage both function calls and function objects, and class decorators can be used to manage both class instances and classes themselves. By returning the decorated object itself instead of a wrapper, decorators become a simple post-creation step for functions and classes.
Regardless of the role they play, decorators provide a convenient and explicit way to code tools useful both during program development and in live production systems.
Depending on your job description, you might encounter decorators as a user or a provider. As we’ve seen, Python itself comes with built-in decorators that have specialized roles—static method declaration, property creation, and more. In addition, many popular Python toolkits include decorators to perform tasks such as managing database or user-interface logic. In such cases, we can get by without knowing how the decorators are coded.
For more general tasks, programmers can code arbitrary decorators of their own. For example, function decorators may be used to augment functions with code that adds call tracing, performs argument validity testing during debugging, automatically acquires and releases thread locks, times calls made to function for optimization, and so on. Any behavior you can imagine adding to a function call is a candidate for custom function decorators.
On the other hand, function decorators are designed to augment only a specific function or method call, not an entire object interface. Class decorators fill the latter role better—because they can intercept instance creation calls, they can be used to implement arbitrary object interface augmentation or management tasks. For example, custom class decorators can trace or validate every attribute reference made for an object. They can also be used to implement proxy objects, singleton classes, and other common coding patterns. In fact, we’ll find that many class decorators bear a strong resemblance to the delegation coding pattern we met in Chapter 30.
Like many advanced Python tools, decorators are never strictly required from a purely technical perspective: their functionality can often be implemented instead using simple helper function calls or other techniques (and at a base level, we can always manually code the name rebinding that decorators perform automatically).
That said, decorators provide an explicit syntax for such tasks, which makes intent clearer, can minimize augmentation code redundancy, and may help ensure correct API usage:
Decorators have a very explicit syntax, which makes them easier to spot than helper function calls that may be arbitrarily far-removed from the subject functions or classes.
Decorators are applied once, when the subject function or class is defined; it’s not necessary to add extra code (which may have to be changed in the future) at every call to the class or function.
Because of both of the prior points, decorators make it less likely that a user of an API will forget to augment a function or class according to API requirements.
In other words, beyond their technical model, decorators offer some advantages in terms of code maintenance and aesthetics. Moreover, as structuring tools, decorators naturally foster encapsulation of code, which reduces redundancy and makes future changes easier.
Decorators do have some potential drawbacks, too—when they insert wrapper logic, they can alter the types of the decorated objects, and they may incur extra calls. On the other hand, the same considerations apply to any technique that adds wrapping logic to objects.
We’ll explore these tradeoffs in the context of real code later in this chapter. Although the choice to use decorators is still somewhat subjective, their advantages are compelling enough that they are quickly becoming best practice in the Python world. To help you decide for yourself, let’s turn to the details.
Let’s get started with a first-pass look at decoration behavior from a symbolic perspective. We’ll write real code soon, but since most of the magic of decorators boils down to an automatic rebinding operation, it’s important to understand this mapping first.
Function decorators have been available in Python since version 2.5. As we
saw earlier in this book, they are largely just syntactic sugar that
runs one function through another at the end of a def
statement, and rebinds the original
function name to the result.
A function decorator is a kind of runtime
declaration about the function whose definition
follows. The decorator is coded on a line just before the def
statement that defines a function or
method, and it consists of the @
symbol followed by a reference to a
metafunction—a function (or other callable
object) that manages another function.
In terms of code, function decorators automatically map the following syntax:
@decorator # Decorate function def F(arg): ... F(99) # Call function
into this equivalent form, where decorator
is a one-argument callable
object that returns a callable object with the same number of
arguments as F
:
def F(arg): ... F = decorator(F) # Rebind function name to decorator result F(99) # Essentially calls decorator(F)(99)
This automatic name rebinding
works on any def
statement,
whether it’s for a simple function or a method within a class.
When the function F
is later
called, it’s actually calling the object
returned by the decorator, which may be
either another object that implements required wrapping logic, or
the original function itself.
In other words, decoration essentially maps the first of the following into the second (though the decorator is really run only once, at decoration time):
func(6, 7) decorator(func)(6, 7)
This automatic name rebinding accounts for the static method and property decoration syntax we met earlier in the book:
class C: @staticmethod def meth(...): ... # meth = staticmethod(meth) class C: @property def name(self): ... # name = property(name)
In both cases, the method name is rebound to the result of a
built-in function decorator, at the end of the def
statement. Calling the original name
later invokes whatever object the decorator returns.
A decorator itself is a callable that returns a callable. That is, it returns the object to be called later when the decorated function is invoked through its original name—either a wrapper object to intercept later calls, or the original function augmented in some way. In fact, decorators can be any type of callable and return any type of callable: any combination of functions and classes may be used, though some are better suited to certain contexts.
For example, to tap into the decoration protocol in order to manage a function just after it is created, we might code a decorator of this form:
def decorator(F): # Process function F return F @decorator def func(): ... # func = decorator(func)
Because the original decorated function is assigned back to its name, this simply adds a post-creation step to function definition. Such a structure might be used to register a function to an API, assign function attributes, and so on.
In more typical use, to insert logic that intercepts later calls to a function, we might code a decorator to return a different object than the original function:
def decorator(F): # Save or use function F # Return a different callable: nested def, class with __call__, etc. @decorator def func(): ... # func = decorator(func)
This decorator is invoked at decoration time, and the callable it returns is invoked when the original function name is later called. The decorator itself receives the decorated function; the callable returned receives whatever arguments are later passed to the decorated function’s name. This works the same for class methods: the implied instance object simply shows up in the first argument of the returned callable.
In skeleton terms, here’s one common coding pattern that captures this idea—the decorator returns a wrapper that retains the original function in an enclosing scope:
def decorator(F): # On @ decoration def wrapper(*args): # On wrapped function call # Use F and args # F(*args) calls original function return wrapper @decorator # func = decorator(func) def func(x, y): # func is passed to decorator's F ... func(6, 7) # 6, 7 are passed to wrapper's *args
When the name func
is later called, it really invokes
the wrapper
function returned
by decorator
; the wrapper
function can then run the
original func
because it is
still available in an enclosing scope. When
coded this way, each decorated function produces a new scope to
retain state.
To do the same with classes, we can overload the call operation and use instance attributes instead of enclosing scopes:
class decorator: def __init__(self, func): # On @ decoration self.func = func def __call__(self, *args): # On wrapped function call # Use self.func and args # self.func(*args) calls original function @decorator def func(x, y): # func = decorator(func) ... # func is passed to __init__ func(6, 7) # 6, 7 are passed to __call__'s *args
When the name func
is
later called now, it really invokes the __call__
operator overloading method of
the instance created by decorator
; the __call__
method can then run the
original func
because it is
still available in an instance attribute.
When coded this way, each decorated function produces a new
instance to retain state.
One subtle point about the prior class-based coding is that while it works to intercept simple function calls, it does not quite work when applied to class method functions:
class decorator: def __init__(self, func): # func is method without instance self.func = func def __call__(self, *args): # self is decorator instance # self.func(*args) fails! # C instance not in args! class C: @decorator def method(self, x, y): # method = decorator(method) ... # Rebound to decorator instance
When coded this way, the decorated method is rebound to an instance of the decorator class, instead of a simple function.
The problem with this is that the self
in the decorator’s __call__
receives the decorator
class
instance when the method is later run, and the instance of class
C
is never included in *args
. This makes it impossible to
dispatch the call to the original method—the decorator object retains
the original method function, but it has no instance to pass to
it.
To support both functions and methods, the nested function alternative works better:
def decorator(F): # F is func or method without instance def wrapper(*args): # class instance in args[0] for method # F(*args) runs func or method return wrapper @decorator def func(x, y): # func = decorator(func) ... func(6, 7) # Really calls wrapper(6, 7) class C: @decorator def method(self, x, y): # method = decorator(method) ... # Rebound to simple function X = C() X.method(6, 7) # Really calls wrapper(X, 6, 7)
When coded this way wrapper
receives the C
class instance in its first argument,
so it can dispatch to the original method and access state
information.
Technically, this nested-function version works because
Python creates a bound method object and thus passes the subject
class instance to the self
argument only when a method attribute references a simple
function; when it references an instance of a callable class
instead, the callable class’s instance is passed to self
to give the callable class access
to its own state information. We’ll see how this subtle difference
can matter in more realistic examples later in this
chapter.
Also note that nested functions are perhaps the most straightforward way to support decoration of both functions and methods, but not necessarily the only way. The prior chapter’s descriptors, for example, receive both the descriptor and subject class instance when called. Though more complex, later in this chapter we’ll see how this tool can be leveraged in this context as well.
Function decorators proved so useful that the model was extended to allow class decoration in Python 2.6 and 3.0. Class decorators are strongly related to function decorators; in fact, they use the same syntax and very similar coding patterns. Rather than wrapping individual functions or methods, though, class decorators are a way to manage classes, or wrap up instance construction calls with extra logic that manages or augments instances created from a class.
Syntactically, class decorators appear just before class
statements (just as function
decorators appear just before function definitions). In symbolic
terms, assuming that decorator
is a one-argument function that returns a callable, the class
decorator syntax:
@decorator # Decorate class class C: ... x = C(99) # Make an instance
is equivalent to the following—the class is automatically passed to the decorator function, and the decorator’s result is assigned back to the class name:
class C: ... C = decorator(C) # Rebind class name to decorator result x = C(99) # Essentially calls decorator(C)(99)
The net effect is that calling the class name later to create an instance winds up triggering the callable returned by the decorator, instead of calling the original class itself.
New class decorators are coded using many of the same techniques used for function decorators. Because a class decorator is also a callable that returns a callable, most combinations of functions and classes suffice.
However it’s coded, the decorator’s result is what runs when an instance is later created. For example, to simply manage a class just after it is created, return the original class itself:
def decorator(C): # Process class C return C @decorator class C: ... # C = decorator(C)
To instead insert a wrapper layer that intercepts later instance creation calls, return a different callable object:
def decorator(C): # Save or use class C # Return a different callable: nested def, class with __call__, etc. @decorator class C: ... # C = decorator(C)
The callable returned by such a class decorator typically creates and returns a new instance of the original class, augmented in some way to manage its interface. For example, the following inserts an object that intercepts undefined attributes of a class instance:
def decorator(cls): # On @ decoration class Wrapper: def __init__(self, *args): # On instance creation self.wrapped = cls(*args) def __getattr__(self, name): # On attribute fetch return getattr(self.wrapped, name) return Wrapper @decorator class C: # C = decorator(C) def __init__(self, x, y): # Run by Wrapper.__init__ self.attr = 'spam' x = C(6, 7) # Really calls Wrapper(6, 7) print(x.attr) # Runs Wrapper.__getattr__, prints "spam"
In this example, the decorator rebinds the class name to
another class, which retains the original class in an enclosing
scope and creates and embeds an instance of the original class
when it’s called. When an attribute is later fetched from the
instance, it is intercepted by the wrapper’s __getattr__
and delegated to the
embedded instance of the original class. Moreover, each decorated
class creates a new scope, which remembers the original class.
We’ll flesh out this example into some more useful code later in
this chapter.
Like function decorators, class
decorators are commonly coded as either “factory” functions that
create and return callables, classes that use __init__
or __call__
methods to intercept call
operations, or some combination thereof. Factory functions
typically retain state in enclosing scope references, and classes
in attributes.
As with function decorators, with class decorators some callable type combinations work better than others. Consider the following invalid alternative to the class decorator of the prior example:
class Decorator: def __init__(self, C): # On @ decoration self.C = C def __call__(self, *args): # On instance creation self.wrapped = self.C(*args) return self def __getattr__(self, attrname): # On atrribute fetch return getattr(self.wrapped, attrname) @Decorator class C: ... # C = Decorator(C) x = C() y = C() # Overwrites x!
This code handles multiple decorated classes (each makes a
new Decorator
instance) and
will intercept instance creation calls (each runs __call__
). Unlike the prior version,
however, this version fails to handle multiple
instances of a given class—each instance creation call
overwrites the prior saved instance. The original version does
support multiple instances, because each instance creation call
makes a new independent wrapper object. More generally, either of
the following patterns supports multiple wrapped instances:
def decorator(C): # On @ decoration class Wrapper: def __init__(self, *args): # On instance creation self.wrapped = C(*args) return Wrapper class Wrapper: ... def decorator(C): # On @ decoration def onCall(*args): # On instance creation return Wrapper(C(*args)) # Embed instance in instance return onCall
We’ll study this phenomenon in a more realistic context later in the chapter; in practice, though, we must be careful to combine callable types properly to support our intent.
Sometimes one decorator isn’t enough. To support multiple steps of augmentation, decorator syntax allows you to add multiple layers of wrapper logic to a decorated function or method. When this feature is used, each decorator must appear on a line of its own. Decorator syntax of this form:
@A @B @C def f(...): ...
runs the same as the following:
def f(...): ... f = A(B(C(f)))
Here, the original function is passed through three different decorators, and the resulting callable object is assigned back to the original name. Each decorator processes the result of the prior, which may be the original function or an inserted wrapper.
If all the decorators insert wrappers, the net effect is that when the original function name is called, three different layers of wrapping object logic will be invoked, to augment the original function in three different ways. The last decorator listed is the first applied, and the most deeply nested (insert joke about “interior decorators” here...).
Just as for functions, multiple class decorators result in multiple nested function calls, and possibly multiple levels of wrapper logic around instance creation calls. For example, the following code:
@spam @eggs class C: ... X = C()
is equivalent to the following:
class C: ... C = spam(eggs(C)) X = C()
Again, each decorator is free to return either the original
class or an inserted wrapper object. With wrappers, when an instance
of the original C
class is
finally requested, the call is redirected to the wrapping layer
objects provided by both the spam
and eggs
decorators, which may
have arbitrarily different roles.
For example, the following do-nothing decorators simply return the decorated function:
def d1(F): return F def d2(F): return F def d3(F): return F @d1 @d2 @d3 def func(): # func = d1(d2(d3(func))) print('spam') func() # Prints "spam"
The same syntax works on classes, as do these same do-nothing decorators.
When decorators insert wrapper function objects, though, they may augment the original function when called—the following concatenates to its result in the decorator layers, as it runs the layers from inner to outer:
def d1(F): return lambda: 'X' + F() def d2(F): return lambda: 'Y' + F() def d3(F): return lambda: 'Z' + F() @d1 @d2 @d3 def func(): # func = d1(d2(d3(func))) return 'spam' print(func()) # Prints "XYZspam"
We use lambda
functions to
implement wrapper layers here (each retains the wrapped function in
an enclosing scope); in practice, wrappers can take the form of
functions, callable classes, and more. When designed well, decorator
nesting allows us to combine augmentation steps in a wide variety of
ways.
Both function and class decorators can also seem to take arguments, although really these arguments are passed to a callable that in effect returns the decorator, which in turn returns a callable. The following, for instance:
@decorator(A, B) def F(arg): ... F(99)
is automatically mapped into this equivalent form, where
decorator
is a callable that
returns the actual decorator. The returned decorator in turn returns
the callable run later for calls to the original function
name:
def F(arg): ... F = decorator(A, B)(F) # Rebind F to result of decorator's return value F(99) # Essentially calls decorator(A, B)(F)(99)
Decorator arguments are resolved before decoration ever occurs, and they are usually used to retain state information for use in later calls. The decorator function in this example, for instance, might take a form like the following:
def decorator(A, B): # Save or use A, B def actualDecorator(F): # Save or use function F # Return a callable: nested def, class with __call__, etc. return callable return actualDecorator
The outer function in this structure generally saves the decorator arguments away as state information, for use in the actual decorator, the callable it returns, or both. This code snippet retains the state information argument in enclosing function scope references, but class attributes are commonly used as well.
In other words, decorator arguments often imply three levels of callables: a callable to accept decorator arguments, which returns a callable to serve as decorator, which returns a callable to handle calls to the original function or class. Each of the three levels may be a function or class and may retain state in the form of scopes or class attributes. We’ll see concrete examples of decorator arguments employed later in this chapter.
Although much of the rest of this chapter focuses on wrapping later calls to functions and classes, I should underscore that the decorator mechanism is more general than this—it is a protocol for passing functions and classes through a callable immediately after they are created. As such, it can also be used to invoke arbitrary post-creation processing:
def decorator(O): # Save or augment function or class O return O @decorator def F(): ... # F = decorator(F) @decorator class C: ... # C = decorator(C)
As long as we return the original decorated object this way instead of a wrapper, we can manage functions and classes themselves, not just later calls to them. We’ll see more realistic examples later in this chapter that use this idea to register callable objects to an API with decoration and assign attributes to functions when they are created.
On to the code—in the rest of this chapter, we are going to study working examples that demonstrate the decorator concepts we just explored. This section presents a handful of function decorators at work, and the next shows class decorators in action. Following that, we’ll close out with some larger case studies of class and function decorator usage.
To get started, let’s revive the call tracer example we met in Chapter 31. The following defines and applies a function decorator that counts the number of calls made to the decorated function and prints a trace message for each call:
class tracer: def __init__(self, func): # On @ decoration: save original func self.calls = 0 self.func = func def __call__(self, *args): # On later calls: run original func self.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) self.func(*args) @tracer def spam(a, b, c): # spam = tracer(spam) print(a + b + c) # Wraps spam in a decorator object
Notice how each function decorated with this class will create
a new instance, with its own saved function object and calls
counter. Also observe how the *args
argument syntax is used to pack and
unpack arbitrarily many passed-in arguments. This generality enables
this decorator to be used to wrap any function with any number of
arguments (this version doesn’t yet work on class methods, but we’ll
fix that later in this section).
Now, if we import this module’s function and test it interactively, we get the following sort of behavior—each call generates a trace message initially, because the decorator class intercepts it. This code runs under both Python 2.6 and 3.0, as does all code in this chapter unless otherwise noted:
>>>from decorator1 import spam
>>>spam(1, 2, 3)
# Really calls the tracer wrapper object call 1 to spam 6 >>>spam('a', 'b', 'c')
# Invokes __call__ in class call 2 to spam abc >>>spam.calls
# Number calls in wrapper state information 2 >>>spam
<decorator1.tracer object at 0x02D9A730>
When run, the tracer
class
saves away the decorated function, and intercepts later calls to it,
in order to add a layer of logic that counts and prints each call.
Notice how the total number of calls shows up as an attribute of the
decorated function—spam
is really
an instance of the tracer
class
when decorated (a finding that may have ramifications for programs
that do type checking, but is generally benign).
For function calls, the @
decoration syntax can be more convenient than modifying each call to
account for the extra logic level, and it avoids accidentally
calling the original function directly. Consider a nondecorator
equivalent such as the following:
calls = 0 def tracer(func, *args): global calls calls += 1 print('call %s to %s' % (calls, func.__name__)) func(*args) def spam(a, b, c): print(a, b, c) >>>spam(1, 2, 3)
# Normal non-traced call: accidental? 1 2 3 >>>tracer(spam, 1, 2, 3)
# Special traced call without decorators call 1 to spam 1 2 3
This alternative can be used on any function without the
special @
syntax, but unlike the
decorator version, it requires extra syntax at every place where the
function is called in your code; furthermore, its intent may not be
as obvious, and it does not ensure that the extra layer will be
invoked for normal calls. Although decorators are never
required (we can always rebind names manually),
they are often the most convenient option.
The last example of the prior section raises an important issue. Function decorators have a variety of options for retaining state information provided at decoration time, for use during the actual function call. They generally need to support multiple decorated objects and multiple calls, but there are a number of ways to implement these goals: instance attributes, global variables, nonlocal variables, and function attributes can all be used for retaining state.
For example, here is an augmented version of the prior example, which adds support for keyword arguments and returns the wrapped function’s result to support more use cases:
class tracer: # State via instance attributes def __init__(self, func): # On @ decorator self.calls = 0 # Save func for later call self.func = func def __call__(self, *args, **kwargs): # On call to original function self.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs) @tracer def spam(a, b, c): # Same as: spam = tracer(spam) print(a + b + c) # Triggers tracer.__init__ @tracer def eggs(x, y): # Same as: eggs = tracer(eggs) print(x ** y) # Wraps eggs in a tracer object spam(1, 2, 3) # Really calls tracer instance: runs tracer.__call__ spam(a=4, b=5, c=6) # spam is an instance attribute eggs(2, 16) # Really calls tracer instance, self.func is eggs eggs(4, y=4) # self.calls is per-function here (need 3.0 nonlocal)
Like the original, this uses class instance
attributes to save state explicitly. Both the wrapped
function and the calls counter are
per-instance information—each decoration gets
its own copy. When run as a script under either 2.6 or 3.0, the
output of this version is as follows; notice how the spam
and eggs
functions each have their own calls
counter, because each decoration creates a new class
instance:
call 1 to spam 6 call 2 to spam 15 call 1 to eggs 65536 call 2 to eggs 256
While useful for decorating functions, this coding scheme has issues when applied to methods (more on this later).
Enclosing def
scope
references and nested def
s can
often achieve the same effect, especially for static data like the
decorated original function. In this example, though, we would
also need a counter in the enclosing scope that changes on each
call, and that’s not possible in Python 2.6. In 2.6, we can either
use classes and attributes, as we did earlier, or move the state
variable out to the global scope, with global
declarations:
calls = 0 def tracer(func): # State via enclosing scope and global def wrapper(*args, **kwargs): # Instead of class attributes global calls # calls is global, not per-function calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return wrapper @tracer def spam(a, b, c): # Same as: spam = tracer(spam) print(a + b + c) @tracer def eggs(x, y): # Same as: eggs = tracer(eggs) print(x ** y) spam(1, 2, 3) # Really calls wrapper, bound to func spam(a=4, b=5, c=6) # wrapper calls spam eggs(2, 16) # Really calls wrapper, bound to eggs eggs(4, y=4) # Global calls is not per-function here!
Unfortunately, moving the counter out to the common global scope to allow it to be changed like this also means that it will be shared by every wrapped function. Unlike class instance attributes, global counters are cross-program, not per-function—the counter is incremented for any traced function call. You can tell the difference if you compare this version’s output with the prior version’s—the single, shared global call counter is incorrectly updated by calls to every decorated function:
call 1 to spam 6 call 2 to spam 15 call 3 to eggs 65536 call 4 to eggs 256
Shared global state may be what we want in some cases. If we
really want a per-function counter, though, we
can either use classes as before, or make use of the new nonlocal
statement in Python 3.0,
described in Chapter 17. Because this new statement
allows enclosing function scope variables to be changed, they can
serve as per-decoration and
changeable data:
def tracer(func): # State via enclosing scope and nonlocal calls = 0 # Instead of class attrs or global def wrapper(*args, **kwargs): # calls is per-function, not global nonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return wrapper @tracer def spam(a, b, c): # Same as: spam = tracer(spam) print(a + b + c) @tracer def eggs(x, y): # Same as: eggs = tracer(eggs) print(x ** y) spam(1, 2, 3) # Really calls wrapper, bound to func spam(a=4, b=5, c=6) # wrapper calls spam eggs(2, 16) # Really calls wrapper, bound to eggs eggs(4, y=4) # Nonlocal calls _is_ not per-function here
Now, because enclosing scope variables are not cross-program globals, each wrapped function gets its own counter again, just as for classes and attributes. Here’s the new output when run under 3.0:
call 1 to spam 6 call 2 to spam 15 call 1 to eggs 65536 call 2 to eggs 256
Finally, if you are not using Python 3.X and don’t have a
nonlocal
statement, you may
still be able to avoid globals and classes by making use of
function attributes for some changeable state
instead. In recent Pythons, we can assign arbitrary attributes to
functions to attach them, with func.
attr
=
value
. In
our example, we can simply use wrapper.calls
for state. The
following works the same as the preceding nonlocal
version because the counter is
again per-decorated-function, but it also runs in Python
2.6:
def tracer(func): # State via enclosing scope and func attr def wrapper(*args, **kwargs): # calls is per-function, not global wrapper.calls += 1 print('call %s to %s' % (wrapper.calls, func.__name__)) return func(*args, **kwargs) wrapper.calls = 0 return wrapper
Notice that this only works because the name wrapper
is retained in the enclosing
tracer
function’s scope. When
we later increment wrapper.calls
, we are not changing the
name wrapper
itself, so no
nonlocal
declaration is
required.
This scheme was almost relegated to a footnote, because it
is more obscure than nonlocal
in 3.0 and is probably
better saved for cases where other schemes don’t help. However, we
will employ it in an answer to one of the end-of-chapter
questions, where we’ll need to access the saved state from
outside the decorator’s code; nonlocals can
only be seen inside the nested function itself, but function
attributes have wider visibility.
Because decorators often imply multiple levels of callables, you can combine functions with enclosing scopes and classes with attributes to achieve a variety of coding structures. As we’ll see later, though, this sometimes may be subtler than you expect—each decorated function should have its own state, and each decorated class may require state both for itself and for each generated instance.
In fact, as the next section will explain, if we want to apply function decorators to class methods, too, we also have to be careful about the distinction Python makes between decorators coded as callable class instance objects and decorators coded as functions.
When I wrote the first tracer
function decorator above, I naively
assumed that it could also be applied to any
method—decorated methods should work the same,
but the automatic self
instance
argument would simply be included at the front of *args
. Unfortunately, I was wrong: when
applied to a class’s method, the first version of the tracer
fails, because self
is the instance of the decorator
class and the instance of the decorated subject class is not
included in *args
. This is true
in both Python 3.0 and 2.6.
I introduced this phenomenon earlier in this chapter, but now we can see it in the context of realistic working code. Given the class-based tracing decorator:
class tracer: def __init__(self, func): # On @ decorator self.calls = 0 # Save func for later call self.func = func def __call__(self, *args, **kwargs): # On call to original function self.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs)
decoration of simple functions works as advertised earlier:
@tracer def spam(a, b, c): # spam = tracer(spam) print(a + b + c) # Triggers tracer.__init__ spam(1, 2, 3) # Runs tracer.__call__ spam(a=4, b=5, c=6) # spam is an instance attribute
However, decoration of class methods
fails (more lucid readers might recognize this as our Person
class resurrected from the
object-oriented tutorial in Chapter 27):
class Person: def __init__(self, name, pay): self.name = name self.pay = pay @tracer def giveRaise(self, percent): # giveRaise = tracer(giveRaise) self.pay *= (1.0 + percent) @tracer def lastName(self): # lastName = tracer(lastName) return self.name.split()[-1] bob = Person('Bob Smith', 50000) # tracer remembers method funcs bob.giveRaise(.25) # Runs tracer.__call__(???, .25) print(bob.lastName()) # Runs tracer.__call__(???)
The root of the problem here is in the self
argument of the tracer class’s
__call__
method—is it a tracer
instance or a Person
instance? We really need
both as it’s coded: the tracer
for decorator state, and the
Person
for routing on to the
original method. Really, self
must be the tracer
object, to provide access to
tracer
’s state information; this
is true whether decorating a simple function or a method.
Unfortunately, when our decorated method name is rebound to a
class instance object with a __call__
, Python passes only the tracer
instance to
self
; it doesn’t pass along the
Person
subject in the arguments
list at all. Moreover, because the tracer
knows nothing about the Person
instance we are trying to process
with method calls, there’s no way to create a bound method with an
instance, and thus no way to correctly dispatch the call.
In fact, the prior listing winds up passing too few arguments
to the decorated method, and results in an error. Add a line to the
decorator’s __call__
to print all
its arguments to verify this; as you can see, self
is the tracer
, and the Person
instance is entirely absent:
<__main__.tracer object at 0x02D6AD90> (0.25,) {} call 1 to giveRaise Traceback (most recent call last): File "C:/misc/tracer.py", line 56, in <module> bob.giveRaise(.25) File "C:/misc/tracer.py", line 9, in __call__ return self.func(*args, **kwargs) TypeError: giveRaise() takes exactly 2 positional arguments (1 given)
As mentioned earlier, this happens because Python passes the
implied subject instance to self
when a method name is bound to a simple function only; when it is an
instance of a callable class, that class’s instance is passed
instead. Technically, Python only makes a bound method object
containing the subject instance when the method is a simple
function.
If you want your function decorators to work on
both simple functions and class methods, the
most straightforward solution lies in using one of the other state
retention solutions described earlier—code your function decorator
as nested def
s, so that you
don’t depend on a single self
instance argument to be both the wrapper class instance and the
subject class instance.
The following alternative applies this fix using Python 3.0
nonlocals. Because decorated methods are rebound to simple
functions instead of instance objects, Python correctly passes the
Person
object as the first
argument, and the decorator propagates it on in the first item of
*args
to the self
argument of the real, decorated
methods:
# A decorator for both functions and methods def tracer(func): # Use function, not class with __call__ calls = 0 # Else "self" is decorator instance only! def onCall(*args, **kwargs): nonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return onCall # Applies to simple functions @tracer def spam(a, b, c): # spam = tracer(spam) print(a + b + c) # onCall remembers spam spam(1, 2, 3) # Runs onCall(1, 2, 3) spam(a=4, b=5, c=6) # Applies to class method functions too! class Person: def __init__(self, name, pay): self.name = name self.pay = pay @tracer def giveRaise(self, percent): # giveRaise = tracer(giveRaise) self.pay *= (1.0 + percent) # onCall remembers giveRaise @tracer def lastName(self): # lastName = tracer(lastName) return self.name.split()[-1] print('methods...') bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) # Runs onCall(sue, .10) print(sue.pay) print(bob.lastName(), sue.lastName()) # Runs onCall(bob), lastName in scopes
This version works the same on both functions and methods:
call 1 to spam 6 call 2 to spam 15 methods... Bob Smith Sue Jones call 1 to giveRaise 110000.0 call 1 to lastName call 2 to lastName Smith Jones
Although the nested function solution illustrated in the prior section is the most straightforward way to support decorators that apply to both functions and class methods, other schemes are possible. The descriptor feature we explored in the prior chapter, for example, can help here as well.
Recall from our discussion in that chapter that a descriptor
may be a class attribute assigned to objects with a __get__
method run automatically when
that attribute is referenced and fetched (object
derivation is required in Python
2.6, but not 3.0):
class Descriptor(object):
def __get__(self, instance, owner): ...
class Subject:
attr = Descriptor()
X = Subject()
X.attr # Roughly runs Descriptor.__get__(Subject.attr, X, Subject)
Descriptors may also have __set__
and __del__
access methods, but we don’t
need them here. Now, because the descriptor’s __get__
method receives
both the descriptor class and subject class
instances when invoked, it’s well suited to decorating methods
when we need both the decorator’s state and the original class
instance for dispatching calls. Consider the following alternative
tracing decorator, which is also a descriptor:
class tracer(object): def __init__(self, func): # On @ decorator self.calls = 0 # Save func for later call self.func = func def __call__(self, *args, **kwargs): # On call to original func self.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs) def __get__(self, instance, owner): # On method attribute fetch return wrapper(self, instance) class wrapper: def __init__(self, desc, subj): # Save both instances self.desc = desc # Route calls back to decr self.subj = subj def __call__(self, *args, **kwargs): return self.desc(self.subj, *args, **kwargs) # Runs tracer.__call__ @tracer def spam(a, b, c): # spam = tracer(spam)...same as prior...
# Uses __call__ only class Person: @tracer def giveRaise(self, percent): # giveRaise = tracer(giverRaise)...same as prior...
# Makes giveRaise a descriptor
This works the same as the preceding nested function coding.
Decorated functions invoke only its __call__
, while decorated methods invoke
its __get__
first to resolve
the method name fetch (on
instance
.
method
); the
object returned by __get__
retains the subject class instance and is then invoked to complete
the call expression, thereby triggering __call__
(on (
args...
)
). For example, the test code’s call
to:
sue.giveRaise(.10) # Runs __get__ then __call__
run’s tracer.__get__
first, because the giveRaise
attribute in the Person
class
has been rebound to a descriptor by the function decorator. The
call expression then triggers the __call__
method of the returned wrapper
object, which in turn invokes
tracer.__call__
.
The wrapper
object
retains both descriptor and subject instances, so it can route
control back to the original decorator/descriptor class instance.
In effect, the wrapper
object
saves the subject class instance available during method attribute
fetch and adds it to the later call’s arguments list, which is
passed to __call__
. Routing the
call back to the descriptor class instance this way is required in
this application so that all calls to a wrapped method use the
same calls
counter state
information in the descriptor instance object.
Alternatively, we could use a nested function and enclosing scope references to achieve the same effect—the following version works the same as the preceding one, by swapping a class and object attributes for a nested function and scope references, but it requires noticeably less code:
class tracer(object): def __init__(self, func): # On @ decorator self.calls = 0 # Save func for later call self.func = func def __call__(self, *args, **kwargs): # On call to original func self.calls += 1 print('call %s to %s' % (self.calls, self.func.__name__)) return self.func(*args, **kwargs) def __get__(self, instance, owner): # On method fetch def wrapper(*args, **kwargs): # Retain both inst return self(instance, *args, **kwargs) # Runs __call__ return wrapper
Add print
statements to
these alternatives’ methods to trace the two-step get/call process on your own, and run them
with the same test code as in the nested function alternative
shown earlier. In either coding, this descriptor-based scheme is
also substantially subtler than the nested function option, and so
is probably a second choice here; it may be a useful coding
pattern in other contexts, though.
In the rest of this chapter we’re going to be fairly casual
about using classes or functions to code our function decorators,
as long as they are applied only to functions. Some decorators may
not require the instance of the original class, and will still
work on both functions and methods if coded as a class—something
like Python’s own staticmethod
decorator, for
example, wouldn’t require an instance of the subject class
(indeed, its whole point is to remove the instance from the
call).
The moral of this story, though, is that if you want your decorators to work on both simple functions and class methods, you’re better off using the nested-function-based coding pattern outlined here instead of a class with call interception.
To sample the fuller flavor of what function decorators
are capable of, let’s turn to a different use case. Our next
decorator times calls made to a decorated function—both the time for
one call, and the total time among all calls. The decorator is
applied to two functions, in order to compare the time requirements
of list comprehensions and the map
built-in call (for comparison, also
see Chapter 20 for
another nondecorator example that times iteration alternatives like
these):
import time class timer: def __init__(self, func): self.func = func self.alltime = 0 def __call__(self, *args, **kargs): start = time.clock() result = self.func(*args, **kargs) elapsed = time.clock() - start self.alltime += elapsed print('%s: %.5f, %.5f' % (self.func.__name__, elapsed, self.alltime)) return result @timer def listcomp(N): return [x * 2 for x in range(N)] @timer def mapcall(N): return map((lambda x: x * 2), range(N)) result = listcomp(5) # Time for this call, all calls, return value listcomp(50000) listcomp(500000) listcomp(1000000) print(result) print('allTime = %s' % listcomp.alltime) # Total time for all listcomp calls print('') result = mapcall(5) mapcall(50000) mapcall(500000) mapcall(1000000) print(result) print('allTime = %s' % mapcall.alltime) # Total time for all mapcall calls print('map/comp = %s' % round(mapcall.alltime / listcomp.alltime, 3))
In this case, a nondecorator approach would allow the subject
functions to be used with or without timing, but it would also
complicate the call signature when timing is desired (we’d need to
add code at every call instead of once at the def
), and there would be no direct way to
guarantee that all list builder calls in a program are routed
through timer logic, short of finding and potentially changing them
all.
When run in Python 2.6, the output of this file’s self-test code is as follows:
listcomp: 0.00002, 0.00002 listcomp: 0.00910, 0.00912 listcomp: 0.09105, 0.10017 listcomp: 0.17605, 0.27622 [0, 2, 4, 6, 8] allTime = 0.276223304917 mapcall: 0.00003, 0.00003 mapcall: 0.01363, 0.01366 mapcall: 0.13579, 0.14945 mapcall: 0.27648, 0.42593 [0, 2, 4, 6, 8] allTime = 0.425933533452 map/comp = 1.542
Testing subtlety: I didn’t run this under Python 3.0 because,
as described in Chapter 14, the map
built-in returns an
iterator in 3.0, instead of an actual list as
in 2.6. Hence, 3.0’s map
doesn’t
quite compare directly to a list comprehension’s work (as is, the
map
test takes virtually no time
at all in 3.0!).
If you wish to run this under 3.0, too, use list(map())
to force it to build a list
like the list comprehension does, or else you’re not really
comparing apples to apples. Don’t do so in 2.6, though—if you do,
the map
test will be charged for
building two lists, not one.
The following sort of code would pick fairly for 2.6 and 3.0;
note, though, that while this makes the comparison between list
comprehensions and map
more fair
in either 2.6 or 3.0, because range
is also an iterator in 3.0, the
results for 2.6 and 3.0 won’t compare directly:
... import sys @timer def listcomp(N): return [x * 2 for x in range(N)] if sys.version_info[0] == 2: @timer def mapcall(N): return map((lambda x: x * 2), range(N)) else: @timer def mapcall(N): return list(map((lambda x: x * 2), range(N))) ...
Finally, as we learned in the modules part of this book if you
want to be able to reuse this decorator in other modules, you should
indent the self-test code at the bottom of the file under a __name__ == '__main__'
test so it runs
only when the file is run, not when it’s imported. We won’t do this,
though, because we’re about to add another feature to our code.
The timer decorator of the prior section works, but it would be nice if it was more configurable—providing an output label and turning trace messages on and off, for instance, might be useful in a general-purpose tool like this. Decorator arguments come in handy here: when they’re coded properly, we can use them to specify configuration options that can vary for each decorated function. A label, for instance, might be added as follows:
def timer(label=''): def decorator(func): def onCall(*args): # args passed to function ... # func retained in enclosing scope print(label, ... # label retained in enclosing scope return onCall return decorator # Returns that actual decorator @timer('==>') # Like listcomp = timer('==>')(listcomp) def listcomp(N): ... # listcomp is rebound to decorator listcomp(...) # Really calls decorator
This code adds an enclosing scope to retain a decorator
argument for use on a later actual call. When the listcomp
function is defined, it really
invokes decorator
(the result of
timer
, run before decoration
actually occurs), with the label
value available in its enclosing scope. That is, timer
returns the
decorator, which remembers both the decorator argument and the original
function and returns a callable which invokes the original function
on later calls.
We can put this structure to use in our timer to allow a label and a trace control flag to be passed in at decoration time. Here’s an example that does just that, coded in a module file named mytools.py so it can be imported as a general tool:
import time def timer(label='', trace=True): # On decorator args: retain args class Timer: def __init__(self, func): # On @: retain decorated func self.func = func self.alltime = 0 def __call__(self, *args, **kargs): # On calls: call original start = time.clock() result = self.func(*args, **kargs) elapsed = time.clock() - start self.alltime += elapsed if trace: format = '%s %s: %.5f, %.5f' values = (label, self.func.__name__, elapsed, self.alltime) print(format % values) return result return Timer
Mostly all we’ve done here is embed the original Timer
class in an enclosing function, in
order to create a scope that retains the decorator arguments. The
outer timer
function is called
before decoration occurs, and it simply returns the Timer
class to serve as the actual
decorator. On decoration, an instance of Timer
is made that remembers the decorated
function itself, but also has access to the decorator arguments in
the enclosing function scope.
This time, rather than embedding self-test code in this file, we’ll run the decorator in a different file. Here’s a client of our timer decorator, the module file testseqs.py, applying it to sequence iteration alternatives again:
from mytools import timer @timer(label='[CCC]==>') def listcomp(N): # Like listcomp = timer(...)(listcomp) return [x * 2 for x in range(N)] # listcomp(...) triggers Timer.__call__ @timer(trace=True, label='[MMM]==>') def mapcall(N): return map((lambda x: x * 2), range(N)) for func in (listcomp, mapcall): print('') result = func(5) # Time for this call, all calls, return value func(50000) func(500000) func(1000000) print(result) print('allTime = %s' % func.alltime) # Total time for all calls print('map/comp = %s' % round(mapcall.alltime / listcomp.alltime, 3))
Again, if you wish to run this fairly in 3.0, wrap the
map
function in a list
call. When run as-is in 2.6, this
file prints the following output—each decorated function now has a
label of its own, defined by decorator arguments:
[CCC]==> listcomp: 0.00003, 0.00003 [CCC]==> listcomp: 0.00640, 0.00643 [CCC]==> listcomp: 0.08687, 0.09330 [CCC]==> listcomp: 0.17911, 0.27241 [0, 2, 4, 6, 8] allTime = 0.272407666337 [MMM]==> mapcall: 0.00004, 0.00004 [MMM]==> mapcall: 0.01340, 0.01343 [MMM]==> mapcall: 0.13907, 0.15250 [MMM]==> mapcall: 0.27907, 0.43157 [0, 2, 4, 6, 8] allTime = 0.431572169089 map/comp = 1.584
As usual, we can also test this interactively to see how the configuration arguments come into play:
>>>from mytools import timer
>>>@timer(trace=False)
# No tracing, collect total time ...def listcomp(N):
...return [x * 2 for x in range(N)]
... >>>x = listcomp(5000)
>>>x = listcomp(5000)
>>>x = listcomp(5000)
>>>listcomp
<mytools.Timer instance at 0x025C77B0> >>>listcomp.alltime
0.0051938863738243413 >>>@timer(trace=True, label=' =>')
# Turn on tracing ...def listcomp(N):
...return [x * 2 for x in range(N)]
... >>>x = listcomp(5000)
=> listcomp: 0.00155, 0.00155 >>>x = listcomp(5000)
=> listcomp: 0.00156, 0.00311 >>>x = listcomp(5000)
=> listcomp: 0.00174, 0.00486 >>>listcomp.alltime
0.0048562736325408196
This timing function decorator can be used for any function, both in modules and interactively. In other words, it automatically qualifies as a general-purpose tool for timing code in our scripts. Watch for another example of decorator arguments in the section Implementing Private Attributes, and again in A Basic Range-Testing Decorator for Positional Arguments.
Timing methods: This section’s timer decorator works on any function, but a minor rewrite is required to be able to apply it to class methods too. In short, as our earlier section Class Blunders I: Decorating Class Methods illustrated, it must avoid using a nested class. Because this mutation will be a subject of one of our end-of-chapter quiz questions, though, I’ll avoid giving away the answer completely here.
So far we’ve been coding function decorators to manage function calls, but as we’ve seen, Python 2.6 and 3.0 extend decorators to work on classes too. As described earlier, while similar in concept to function decorators, class decorators are applied to classes instead—they may be used either to manage classes themselves, or to intercept instance creation calls in order to manage instances. Also like function decorators, class decorators are really just optional syntactic sugar, though many believe that they make a programmer’s intent more obvious and minimize erroneous calls.
Because class decorators may intercept instance creation
calls, they can be used to either manage all the instances of a
class, or augment the interfaces of those instances. To demonstrate,
here’s a first class decorator example that does the former—managing
all instances of a class. This code implements the classic
singleton coding pattern, where at most one
instance of a class ever exists. Its singleton
function defines and returns a
function for managing instances, and the @
syntax automatically wraps up a subject
class in this function:
instances = {} def getInstance(aClass, *args): # Manage global table if aClass not in instances: # Add **kargs for keywords instances[aClass] = aClass(*args) # One dict entry per class return instances[aClass] def singleton(aClass): # On @ decoration def onCall(*args): # On instance creation return getInstance(aClass, *args) return onCall
To use this, decorate the classes for which you want to enforce a single-instance model:
@singleton # Person = singleton(Person) class Person: # Rebinds Person to onCall def __init__(self, name, hours, rate): # onCall remembers Person self.name = name self.hours = hours self.rate = rate def pay(self): return self.hours * self.rate @singleton # Spam = singleton(Spam) class Spam: # Rebinds Spam to onCall def __init__(self, val): # onCall remembers Spam self.attr = val bob = Person('Bob', 40, 10) # Really calls onCall print(bob.name, bob.pay()) sue = Person('Sue', 50, 20) # Same, single object print(sue.name, sue.pay()) X = Spam(42) # One Person, one Spam Y = Spam(99) print(X.attr, Y.attr)
Now, when the Person
or
Spam
class is later used to
create an instance, the wrapping logic layer provided by the
decorator routes instance construction calls to onCall
, which in turn calls getInstance
to manage and share a single
instance per class, regardless of how many construction calls are
made. Here’s this code’s output:
Bob 400 Bob 400 42 42
Interestingly, you can code a more self-contained solution
here if you’re able to use the nonlocal
statement (available in Python
3.0 and later) to change enclosing scope names, as described
earlier—the following alternative achieves an identical effect, by
using one enclosing scope per class, instead of
one global table entry per class:
def singleton(aClass): # On @ decoration instance = None def onCall(*args): # On instance creation nonlocal instance # 3.0 and later nonlocal if instance == None: instance = aClass(*args) # One scope per class return instance return onCall
This version works the same, but it does not depend on names in the global scope outside the decorator. In either Python 2.6 or 3.0, you can also code a self-contained solution with a class instead—the following uses one instance per class, rather than an enclosing scope or global table, and works the same as the other two versions (in fact, it relies on the same coding pattern that we will later see is a common decorator class blunder; here we want just one instance, but that’s not always the case):
class singleton: def __init__(self, aClass): # On @ decoration self.aClass = aClass self.instance = None def __call__(self, *args): # On instance creation if self.instance == None: self.instance = self.aClass(*args) # One instance per class return self.instance
To make this decorator a fully general-purpose tool, store it
in an importable module file, indent the self-test code under a
__name__
check, and add support
for keyword arguments in construction calls with **kargs
syntax (I’ll leave this as a
suggested exercise).
The singleton example of the prior section illustrated using class decorators to manage all the instances of a class. Another common use case for class decorators augments the interface of each generated instance. Class decorators can essentially install on instances a wrapper logic layer that manages access to their interfaces in some way.
For example, in Chapter 30, the
__getattr__
operator overloading
method is shown as a way to wrap up entire object interfaces of
embedded instances, in order to implement the
delegation coding pattern. We saw similar
examples in the managed attribute coverage of the prior chapter.
Recall that __getattr__
is run
when an undefined attribute name is fetched; we can use this hook to
intercept method calls in a controller class and propagate them to
an embedded object.
For reference, here’s the original nondecorator delegation example, working on two built-in type objects:
class Wrapper: def __init__(self, object): self.wrapped = object # Save object def __getattr__(self, attrname): print('Trace:', attrname) # Trace fetch return getattr(self.wrapped, attrname) # Delegate fetch >>>x = Wrapper([1,2,3])
# Wrap a list >>>x.append(4)
# Delegate to list method Trace: append >>>x.wrapped
# Print my member [1, 2, 3, 4] >>>x = Wrapper({"a": 1, "b": 2})
# Wrap a dictionary >>>list(x.keys())
# Delegate to dictionary method Trace: keys # Use list() in 3.0 ['a', 'b']
In this code, the Wrapper
class intercepts access to any of the wrapped object’s attributes,
prints a trace message, and uses the getattr
built-in to pass off the request
to the wrapped object.
Specifically, it traces attribute accesses made
outside the wrapped object’s class; accesses
inside the wrapped object’s methods are not caught and run normally
by design. This whole-interface model differs from the behavior of
function decorators, which wrap up just one specific method.
Class decorators provide an alternative and convenient way to
code this __getattr__
technique
to wrap an entire interface. In 2.6 and 3.0, for example, the prior
class example can be coded as a class decorator that triggers
wrapped instance creation, instead of passing a pre-made instance
into the wrapper’s constructor (also augmented here to support
keyword arguments with **kargs
and to count the number of accesses made):
def Tracer(aClass): # On @ decorator class Wrapper: def __init__(self, *args, **kargs): # On instance creation self.fetches = 0 self.wrapped = aClass(*args, **kargs) # Use enclosing scope name def __getattr__(self, attrname): print('Trace: ' + attrname) # Catches all but own attrs self.fetches += 1 return getattr(self.wrapped, attrname) # Delegate to wrapped obj return Wrapper @Tracer class Spam: # Spam = Tracer(Spam) def display(self): # Spam is rebound to Wrapper print('Spam!' * 8) @Tracer class Person: # Person = Tracer(Person) def __init__(self, name, hours, rate): # Wrapper remembers Person self.name = name self.hours = hours self.rate = rate def pay(self): # Accesses outside class traced return self.hours * self.rate # In-method accesses not traced food = Spam() # Triggers Wrapper() food.display() # Triggers __getattr__ print([food.fetches]) bob = Person('Bob', 40, 50) # bob is really a Wrapper print(bob.name) # Wrapper embeds a Person print(bob.pay()) print('') sue = Person('Sue', rate=100, hours=60) # sue is a different Wrapper print(sue.name) # with a different Person print(sue.pay()) print(bob.name) # bob has different state print(bob.pay()) print([bob.fetches, sue.fetches]) # Wrapper attrs not traced
It’s important to note that this is very different from the tracer decorator we met earlier. In Coding Function Decorators, we looked at decorators that enabled us to trace and time calls to a given function or method. In contrast, by intercepting instance creation calls, the class decorator here allows us to trace an entire object interface—i.e., accesses to any of its attributes.
The following is the output produced by this code under both
2.6 and 3.0: attribute fetches on instances of both the Spam
and Person
classes invoke the __getattr__
logic in the Wrapper
class, because food
and bob
are really instances of Wrapper
, thanks to the decorator’s
redirection of instance creation calls:
Trace: display Spam!Spam!Spam!Spam!Spam!Spam!Spam!Spam! [1] Trace: name Bob Trace: pay 2000 Trace: name Sue Trace: pay 6000 Trace: name Bob Trace: pay 2000 [4, 2]
Notice that the preceding code decorates a user-defined class.
Just like in the original example in Chapter 30, we can also use the decorator
to wrap up a built-in type such as a list, as long as we either
subclass to allow decoration syntax or perform the decoration
manually—decorator syntax requires a class
statement for the @
line.
In the following, x
is
really a Wrapper
again due to the
indirection of decoration (I moved the decorator class to module
file tracer.py in order to
reuse it this way):
>>>from tracer import Tracer
# Decorator moved to a module file >>>@Tracer
...class MyList(list): pass
# MyList = Tracer(MyList) >>>x = MyList([1, 2, 3])
# Triggers Wrapper() >>>x.append(4)
# Triggers __getattr__, append Trace: append >>>x.wrapped
[1, 2, 3, 4] >>>WrapList = Tracer(list)
# Or perform decoration manually >>>x = WrapList([4, 5, 6])
# Else subclass statement required >>>x.append(7)
Trace: append >>>x.wrapped
[4, 5, 6, 7]
The decorator approach allows us to move instance creation into the decorator itself, instead of requiring a premade object to be passed in. Although this seems like a minor difference, it lets us retain normal instance creation syntax and realize all the benefits of decorators in general. Rather than requiring all instance creation calls to route objects through a wrapper manually, we need only augment classes with decorator syntax:
@Tracer # Decorator approach class Person: ... bob = Person('Bob', 40, 50) sue = Person('Sue', rate=100, hours=60) class Person: ... # Non-decorator approach bob = Wrapper(Person('Bob', 40, 50)) sue = Wrapper(Person('Sue', rate=100, hours=60))
Assuming you will make more than one instance of a class, decorators will generally be a net win in terms of both code size and code maintenance.
Attribute version skew note: As we
learned in Chapter 37, __getattr__
will intercept accesses to
operator overloading methods like __str__
and __repr__
in Python 2.6, but not in
3.0.
In Python 3.0, class instances inherit defaults for some
(but not all) of these names from the class (really, from the
automatic object
superclass),
because all classes are “new-style.” Moreover, in 3.0 implicitly
invoked attributes for built-in operations like printing and
+
are not
routed through __getattr__
(or
its cousin, __getattribute__
).
New-style classes look up such methods in
classes and skip the normal instance lookup entirely.
Here, this means that the __getattr__
-based tracing wrapper will
automatically trace and propagate operator overloading calls in
2.6, but not in 3.0. To see this, display “x” directly at the end
of the preceding interactive session—in 2.6 the attribute __repr__
is traced and the list prints
as expected, but in 3.0 no trace occurs and the list prints using
a default display for the Wrapper
class:
>>> x # 2.6 Trace: __repr__ [4, 5, 6, 7] >>> x # 3.0 <tracer.Wrapper object at 0x026C07D0>
To work the same in 3.0, operator overloading methods
generally need to be redefined redundantly in the wrapper class,
either by hand, by tools, or by definition in superclasses. Only
simple named attributes will work the same in both versions. We’ll
see this version skew at work again in a Private
decorator later in this
chapter.
Curiously, the decorator function in this example can
almost be coded as a class instead of a
function, with the proper operator overloading protocol. The
following slightly simplified alternative works similarly because
its __init__
is triggered when
the @
decorator is applied to the
class, and its __call__
is
triggered when a subject class instance is created. Our objects are
really instances of Tracer
this
time, and we essentially just trade an enclosing scope reference for
an instance attribute here:
class Tracer: def __init__(self, aClass): # On @decorator self.aClass = aClass # Use instance attribute def __call__(self, *args): # On instance creation self.wrapped = self.aClass(*args) # ONE (LAST) INSTANCE PER CLASS! return self def __getattr__(self, attrname): print('Trace: ' + attrname) return getattr(self.wrapped, attrname) @Tracer # Triggers __init__ class Spam: # Like: Spam = Tracer(Spam) def display(self): print('Spam!' * 8) ... food = Spam() # Triggers __call__ food.display() # Triggers __getattr__
As we saw in the abstract earlier, though, this class-only
alternative handles multiple classes as before, but it won’t quite
work for multiple instances of a given class:
each instance construction call triggers __call__
, which overwrites the prior
instance. The net effect is that Tracer
saves just one instance—the last
one created. Experiment with this yourself to see how, but here’s an
example of the problem:
@Tracer class Person: # Person = Tracer(Person) def __init__(self, name): # Wrapper bound to Person self.name = name bob = Person('Bob') # bob is really a Wrapper print(bob.name) # Wrapper embeds a Person Sue = Person('Sue') print(sue.name) # sue overwrites bob print(bob.name) # OOPS: now bob's name is 'Sue'!
This code’s output follows—because this tracer only has a single shared instance, the second overwrites the first:
Trace: name Bob Trace: name Sue Trace: name Sue
The problem here is bad state retention—we make one decorator instance per class, but not per class instance, such that only the last instance is retained. The solution, as in our prior class blunder for decorating methods, lies in abandoning class-based decorators.
The earlier function-based Tracer
version does
work for multiple instances, because each instance construction call
makes a new Wrapper
instance,
instead of overwriting the state of a single shared Tracer
instance; the original nondecorator
version handles multiple instances correctly for the same reason.
Decorators are not only arguably magical, they can also be
incredibly subtle!
Regardless of such subtleties, the Tracer
class decorator example ultimately
still relies on __getattr__
to
intercept fetches on a wrapped and embedded instance object. As we
saw earlier, all we’ve really accomplished is moving the instance
creation call inside a class, instead of passing the instance into a
manager function. With the original nondecorator tracing example, we
would simply code instance creation differently:
class Spam: # Non-decorator version ... # Any class will do food = Wrapper(Spam()) # Special creation syntax @Tracer class Spam: # Decorator version ... # Requires @ syntax at class food = Spam() # Normal creation syntax
Essentially, class decorators shift special syntax requirements from the instance creation call to the class statement itself. This is also true for the singleton example earlier in this section—rather than decorating a class and using normal instance creation calls, we could simply pass the class and its construction arguments into a manager function:
instances = {}
def getInstance(aClass, *args):
if aClass not in instances:
instances[aClass] = aClass(*args)
return instances[aClass]
bob = getInstance(Person, 'Bob', 40, 10) # Versus: bob = Person('Bob', 40, 10)
Alternatively, we could use Python’s introspection facilities to fetch the class from an already-created instance (assuming creating an initial instance is acceptable):
instances = {}
def getInstance(object):
aClass = object.__class__
if aClass not in instances:
instances[aClass] = object
return instances[aClass]
bob = getInstance(Person('Bob', 40, 10)) # Versus: bob = Person('Bob', 40, 10)
The same holds true for function decorators like the tracer we wrote earlier: rather than decorating a function with logic that intercepts later calls, we could simply pass the function and its arguments into a manager that dispatches the call:
def func(x, y): # Nondecorator version ... # def tracer(func, args): ... func(*args) result = tracer(func, (1, 2)) # Special call syntax @tracer def func(x, y): # Decorator version ... # Rebinds name: func = tracer(func) result = func(1, 2) # Normal call syntax
Manager function approaches like this place the burden of using special syntax on calls, instead of expecting decoration syntax at function and class definitions.
So why did I just show you ways to not use decorators to implement singletons? As I mentioned at the start of this chapter, decorators present us with tradeoffs. Although syntax matters, we all too often forget to ask the “why” questions when confronted with new tools. Now that we’ve seen how decorators actually work, let’s step back for a minute to glimpse the big picture here.
Like most language features, decorators have both pros and cons. For example, in the negatives column, class decorators suffer from two potential drawbacks:
As we’ve seen, when wrappers are inserted, a decorated function or class does not retain its original type—its name is rebound to a wrapper object, which might matter in programs that use object names or test object types. In the singleton example, both the decorator and manager function approaches retain the original class type for instances; in the tracer code, neither approach does, because wrappers are required.
A wrapping layer added by decoration incurs the additional performance cost of an extra call each time the decorated object is invoked—calls are relatively time-expensive operations, so decoration wrappers can make a program slower. In the tracer code, both approaches require each attribute to be routed through a wrapper layer; the singleton example avoids extra calls by retaining the original class type.
Similar concerns apply with function decorators: both decoration and manager functions incur extra calls, and type changes generally occur when decorating (but not otherwise).
That said, neither of these is a very serious issue. For most programs, the type difference issue is unlikely to matter and the speed hit of the extra calls will be insignificant; furthermore, the latter occurs only when wrappers are used, can often be negated by simply removing the decorator when optimal performance is required, and is also incurred by nondecorator solutions that add wrapping logic (including metaclasses, as we’ll see in Chapter 39).
Conversely, as we saw at the start of this chapter, decorators have three main advantages. Compared to the manager (a.k.a. “helper”) function solutions of the prior section, decorators offer:
Decorators make augmentation explicit and obvious. Their
@
syntax is easier to
recognize than special code in calls that may appear anywhere
in a source file—in our singleton and tracer examples, for
instance, the decorator lines seem more likely to be noticed
than extra code at calls would be. Moreover, decorators allow
function and instance creation calls to use normal syntax
familiar to all Python programmers.
Decorators avoid repeated augmentation code at each function or class call. Because they appear just once, at the definition of the class or function itself, they obviate redundancy and simplify future code maintenance. For our singleton and tracer cases, we need to use special code at each call to use a manager function approach—extra work is required both initially and for any modifications that must be made in the future.
Decorators make it less likely that a programmer will forget to use required wrapping logic. This derives mostly from the two prior advantages—because decoration is explicit and appears only once, at the decorated objects themselves, decorators promote more consistent and uniform API usage than special code that must be included at each call. In the singleton example, for instance, it would be easy to forget to route all class creation calls through special code, which would subvert the singleton management altogether.
Decorators also promote code encapsulation to reduce redundancy and minimize future maintenance effort; although other code structuring tools do too, decorators make this natural for augmentation tasks.
None of these benefits completely requires decorator syntax to be achieved, though, and decorator usage is ultimately a stylistic choice. That said, most programmers find them to be a net win, especially as a tool for using libraries and APIs correctly.
I can recall similar arguments being made both for and against
constructor functions in classes—prior to the introduction of
__init__
methods, the same effect
was often achieved by running an instance through a method manually
when creating it (e.g., X=Class().init()
). Over time, though,
despite being fundamentally a stylistic choice, the __init__
syntax came to be universally
preferred because it was more explicit, consistent, and
maintainable. Although you should be the judge, decorators seem to
bring many of the same assets to the table.
Most of our examples in this chapter have been designed to intercept function and instance creation calls. Although this is typical for decorators, they are not limited to this role. Because decorators work by running new functions and classes through decorator code, they can also be used to manage function and class objects themselves, not just later calls made to them.
Imagine, for example, that you require methods or classes used by an application to be registered to an API for later processing (perhaps that API will call the objects later, in response to events). Although you could provide a registration function to be called manually after the objects are defined, decorators make your intent more explicit.
The following simple implementation of this idea defines a decorator that can be applied to both functions and classes, to add the object to a dictionary-based registry. Because it returns the object itself instead of a wrapper, it does not intercept later calls:
# Registering decorated objects to an API registry = {} def register(obj): # Both class and func decorator registry[obj.__name__] = obj # Add to registry return obj # Return obj itself, not a wrapper @register def spam(x): return(x ** 2) # spam = register(spam) @register def ham(x): return(x ** 3) @register class Eggs: # Eggs = register(Eggs) def __init__(self, x): self.data = x ** 4 def __str__(self): return str(self.data) print('Registry:') for name in registry: print(name, '=>', registry[name], type(registry[name])) print(' Manual calls:') print(spam(2)) # Invoke objects manually print(ham(2)) # Later calls not intercepted X = Eggs(2) print(X) print(' Registry calls:') for name in registry: print(name, '=>', registry[name](3)) # Invoke from registry
When this code is run the decorated objects are added to the registry by name, but they still work as originally coded when they’re called later, without being routed through a wrapper layer. In fact, our objects can be run both manually and from inside the registry table:
Registry: Eggs => <class '__main__.Eggs'> <class 'type'> ham => <function ham at 0x02CFB738> <class 'function'> spam => <function spam at 0x02CFB6F0> <class 'function'> Manual calls: 4 8 16 Registry calls: Eggs => 81 ham => 27 spam => 9
A user interface might use this technique, for example, to
register callback handlers for user actions. Handlers might be
registered by function or class name, as done here, or decorator
arguments could be used to specify the subject event; an extra
def
statement enclosing our
decorator could be used to retain such arguments for use on
decoration.
This example is artificial, but its technique is very general. For example, function decorators might also be used to process function attributes, and class decorators might insert new class attributes, or even new methods, dynamically. Consider the following function decorators—they assign function attributes to record information for later use by an API, but they do not insert a wrapper layer to intercept later calls:
# Augmenting decorated objects directly >>>def decorate(func):
...func.marked = True
# Assign function attribute for later use ...return func
... >>>@decorate
...def spam(a, b):
...return a + b
... >>>spam.marked
True >>>def annotate(text):
# Same, but value is decorator argument ...def decorate(func):
...func.label = text
...return func
...return decorate
... >>>@annotate('spam data')
...def spam(a, b):
# spam = annotate(...)(spam) ...return a + b
... >>>spam(1, 2), spam.label
(3, 'spam data')
Such decorators augment functions and classes directly, without catching later calls to them. We’ll see more examples of class decorations managing classes directly in the next chapter, because this turns out to encroach on the domain of metaclasses; for the remainder of this chapter, let’s turn to two larger case studies of decorators at work.
The final two sections of this chapter present larger examples of decorator use. Both are presented with minimal description, partly because this chapter has exceeded its size limits, but mostly because you should already understand decorator basics well enough to study these on your own. Being general-purpose tools, these examples give us a chance to see how decorator concepts come together in more useful code.
The following class decorator implements a Private
declaration for class instance
attributes—that is, attributes stored on an instance, or inherited
from one of its classes. It disallows fetch and change access to
such attributes from outside the decorated
class, but still allows the class itself to access those names
freely within its methods. It’s not exactly C++ or Java, but it
provides similar access control as an option in Python.
We saw an incomplete first-cut implementation of instance attribute privacy for changes only in Chapter 29. The version here extends this concept to validate attribute fetches too, and it uses delegation instead of inheritance to implement the model. In fact, in a sense this is just an extension to the attribute tracer class decorator we met earlier.
Although this example utilizes the new syntactic sugar of
class decorators to code attribute privacy, its attribute
interception is ultimately still based upon the __getattr__
and
__setattr__
operator overloading
methods we met in prior chapters. When a private attribute access is
detected, this version uses the raise
statement to raise an exception,
along with an error message; the exception may be caught in a
try
or allowed to terminate the
script.
Here is the code, along with a self test at the bottom of the
file. It will work under both Python 2.6 and 3.0 because it employs
3.0 print
and raise
syntax, though it catches operator
overloading method attributes in 2.6 only (more on this in a
moment):
""" Privacy for attributes fetched from class instances. See self-test code at end of file for a usage example. Decorator same as: Doubler = Private('data', 'size')(Doubler). Private returns onDecorator, onDecorator returns onInstance, and each onInstance instance embeds a Doubler instance. """ traceMe = False def trace(*args): if traceMe: print('[' + ' '.join(map(str, args)) + ']') def Private(*privates): # privates in enclosing scope def onDecorator(aClass): # aClass in enclosing scope class onInstance: # wrapped in instance attribute def __init__(self, *args, **kargs): self.wrapped = aClass(*args, **kargs) def __getattr__(self, attr): # My attrs don't call getattr trace('get:', attr) # Others assumed in wrapped if attr in privates: raise TypeError('private attribute fetch: ' + attr) else: return getattr(self.wrapped, attr) def __setattr__(self, attr, value): # Outside accesses trace('set:', attr, value) # Others run normally if attr == 'wrapped': # Allow my attrs self.__dict__[attr] = value # Avoid looping elif attr in privates: raise TypeError('private attribute change: ' + attr) else: setattr(self.wrapped, attr, value) # Wrapped obj attrs return onInstance # Or use __dict__ return onDecorator if __name__ == '__main__': traceMe = True @Private('data', 'size') # Doubler = Private(...)(Doubler) class Doubler: def __init__(self, label, start): self.label = label # Accesses inside the subject class self.data = start # Not intercepted: run normally def size(self): return len(self.data) # Methods run with no checking def double(self): # Because privacy not inherited for i in range(self.size()): self.data[i] = self.data[i] * 2 def display(self): print('%s => %s' % (self.label, self.data)) X = Doubler('X is', [1, 2, 3]) Y = Doubler('Y is', [−10, −20, −30]) # The followng all succeed print(X.label) # Accesses outside subject class X.display(); X.double(); X.display() # Intercepted: validated, delegated print(Y.label) Y.display(); Y.double() Y.label = 'Spam' Y.display() # The following all fail properly """ print(X.size()) # prints "TypeError: private attribute fetch: size" print(X.data) X.data = [1, 1, 1] X.size = lambda S: 0 print(Y.data) print(Y.size()) """
When traceMe
is True
, the module file’s self-test code
produces the following output. Notice how the decorator catches and
validates both attribute fetches and assignments run outside of the
wrapped class, but does not catch attribute accesses inside the
class itself:
[set: wrapped <__main__.Doubler object at 0x02B2AAF0>] [set: wrapped <__main__.Doubler object at 0x02B2AE70>] [get: label] X is [get: display] X is => [1, 2, 3] [get: double] [get: display] X is => [2, 4, 6] [get: label] Y is [get: display] Y is => [−10, −20, −30] [get: double] [set: label Spam] [get: display] Spam => [−20, −40, −60]
This code is a bit complex, and you’re probably best off tracing through it on your own to see how it works. To help you study, though, here are a few highlights worth mentioning.
The first-cut privacy example shown in Chapter 29 used
inheritance to mix in a __setattr__
to catch accesses.
Inheritance makes this difficult, however, because differentiating
between accesses from inside or outside the class is not
straightforward (inside access should be allowed to run normally,
and outside access should be restricted). To work around this, the
Chapter 29 example requires
inheriting classes to use __dict__
assignments to set
attributes—an incomplete solution at best.
The version here uses delegation
(embedding one object inside another) instead of inheritance; this
pattern is better suited to our task, as it makes it much easier
to distinguish between accesses inside and outside of the subject
class. Attribute accesses from outside the subject class are
intercepted by the wrapper layer’s overloading methods and
delegated to the class if valid; accesses inside the class itself
(i.e., through self
inside its
methods’ code) are not intercepted and are allowed to run normally
without checks, because privacy is not inherited here.
The class decorator used here accepts any number of
arguments, to name private attributes. What really happens,
though, is that the arguments are passed to the Private
function, and Private
returns the decorator function
to be applied to the subject class. So, the arguments are used
before decoration ever occurs; Private
returns the decorator, which in
turn “remembers” the privates list as an enclosing scope
reference.
Speaking of enclosing scopes, there are actually three levels of state retention at work in this code:
The arguments to Private
are used before decoration
occurs and are retained as an enclosing scope reference for
use in both onDecorator
and
onInstance
.
The class argument to onDecorator
is used at decoration
time and is retained as an enclosing scope reference for use
at instance construction time.
The wrapped instance object is retained as an instance
attribute in onInstance
,
for use when attributes are later accessed from outside the
class.
This all works fairly naturally, given Python’s scope and namespace rules.
The __setattr__
in this
code relies on an instance object’s __dict__
attribute namespace dictionary
in order to set onInstance
’s
own wrapped
attribute. As we
learned in the prior chapter, it cannot assign an attribute
directly without looping. However, it uses the setattr
built-in instead of __dict__
to set attributes in the
wrapped object itself. Moreover, getattr
is used to fetch attributes in
the wrapped object, since they may be stored in the object itself
or inherited by it.
Because of that, this code will work for most classes. You
may recall from Chapter 31 that
new-style classes with __slots__
may not store attributes in a __dict__
. However, because we only rely
on a __dict__
at the onInstance
level here, not in the
wrapped instance, and because setattr
and getattr
apply to attributes based on
both __dict__
and __slots__
, our decorator applies to
classes using either storage scheme.
Now that we have a Private
implementation, it’s
straightforward to generalize the code to allow for Public
declarations too—they are
essentially the inverse of Private
declarations, so we need only
negate the inner test. The example listed in this section allows a
class to use decorators to define a set of either Private
or Public
instance attributes (attributes
stored on an instance or inherited from its classes), with the
following semantics:
Private
declares
attributes of a class’s instances that cannot be fetched or
assigned, except from within the code of the class’s methods.
That is, any name declared Private
cannot be accessed from
outside the class, while any name not declared Private
can be freely fetched or
assigned from outside the class.
Public
declares
attributes of a class’s instances that can be fetched or
assigned from both outside the class and within the class’s
methods. That is, any name declared Public
can be freely accessed
anywhere, while any name not declared Public
cannot be accessed from outside
the class.
Private
and Public
declarations are intended to be
mutually exclusive: when using Private
, all undeclared names are
considered Public
, and when using
Public
, all undeclared names are
considered Private
. They are
essentially inverses, though undeclared names not created by class
methods behave slightly differently—they can be assigned and thus
created outside the class under Private
(all undeclared names are
accessible), but not under Public
(all undeclared names are inaccessible).
Again, study this code on your own to get a feel for how this
works. Notice that this scheme adds an additional fourth
level of state retention at the top, beyond that
described in the preceding section: the test functions used by the
lambda
s are saved in an extra
enclosing scope. This example is coded to run under either Python
2.6 or 3.0, though it comes with a caveat when run under 3.0
(explained briefly in the file’s docstring and expanded on after the
code):
""" Class decorator with Private and Public attribute declarations. Controls access to attributes stored on an instance, or inherited by it from its classes. Private declares attribute names that cannot be fetched or assigned outside the decorated class, and Public declares all the names that can. Caveat: this works in 3.0 for normally named attributes only: __X__ operator overloading methods implicitly run for built-in operations do not trigger either __getattr__ or __getattribute__ in new-style classes. Add __X__ methods here to intercept and delegate built-ins. """ traceMe = False def trace(*args): if traceMe: print('[' + ' '.join(map(str, args)) + ']') def accessControl(failIf): def onDecorator(aClass): class onInstance: def __init__(self, *args, **kargs): self.__wrapped = aClass(*args, **kargs) def __getattr__(self, attr): trace('get:', attr) if failIf(attr): raise TypeError('private attribute fetch: ' + attr) else: return getattr(self.__wrapped, attr) def __setattr__(self, attr, value): trace('set:', attr, value) if attr == '_onInstance__wrapped': self.__dict__[attr] = value elif failIf(attr): raise TypeError('private attribute change: ' + attr) else: setattr(self.__wrapped, attr, value) return onInstance return onDecorator def Private(*attributes): return accessControl(failIf=(lambda attr: attr in attributes)) def Public(*attributes): return accessControl(failIf=(lambda attr: attr not in attributes))
See the prior example’s self-test code for a usage example.
Here’s a quick look at these class decorators in action at the
interactive prompt (they work the same in 2.6 and 3.0); as
advertised, non-Private
or
Public
names can be fetched and
changed from outside the subject class, but Private
or non-Public
names cannot:
>>>from access import Private, Public
>>>@Private('age')
# Person = Private('age')(Person) ...class Person:
# Person = onInstance with state ...def __init__(self, name, age):
...self.name = name
...self.age = age
# Inside accesses run normally ... >>>X = Person('Bob', 40)
>>>X.name
# Outside accesses validated 'Bob' >>>X.name = 'Sue'
>>>X.name
'Sue' >>>X.age
TypeError: private attribute fetch: age >>>X.age = 'Tom'
TypeError: private attribute change: age >>>@Public('name')
...class Person:
...def __init__(self, name, age):
...self.name = name
...self.age = age
... >>>X = Person('bob', 40)
# X is an onInstance >>>X.name
# onInstance embeds Person 'bob' >>>X.name = 'Sue'
>>>X.name
'Sue' >>>X.age
TypeError: private attribute fetch: age >>>X.age = 'Tom'
TypeError: private attribute change: age
To help you analyze the code, here are a few final notes on this version. Since this is just a generalization of the preceding section’s example, most of the notes there apply here as well.
Besides generalizing, this version also makes use of
Python’s __
X
pseudoprivate name mangling feature (which we met in Chapter 30) to localize the wrapped
attribute to the control class,
by automatically prefixing it with the class name. This avoids the
prior version’s risk for collisions with a wrapped
attribute that may be used by
the real, wrapped class, and it’s useful in a general tool like
this. It’s not quite “privacy,” though, because the mangled name
can be used freely outside the class. Notice that we also have to
use the fully expanded name string ('_onInstance__wrapped'
) in __setattr__
, because that’s what Python
changes it to.
Although this example does implement access controls for
attributes of an instance and its classes, it is possible to
subvert these controls in various ways—for instance, by going
through the expanded version of the wrapped
attribute explicitly (bob.pay
might not work, but the fully
mangled bob._onInstance__wrapped.pay
could!). If
you have to explicitly try to do so, though, these controls are
probably sufficient for normal intended use. Of course, privacy
controls can generally be subverted in any language if you try
hard enough (#define private
public
may work in some C++ implementations, too).
Although access controls can reduce accidental changes, much of
this is up to programmers in any language; whenever source code
may be changed, access control will always be a bit of a pipe
dream.
We could again achieve the same results without decorators,
by using manager functions or coding the name rebinding of
decorators manually; the decorator syntax, however, makes this
consistent and a bit more obvious in the code. The chief potential
downsides of this and any other wrapper-based approach are that
attribute access incurs an extra call, and instances of decorated
classes are not really instances of the original decorated
class—if you test their type with X.__class__
or isinstance(X, C)
, for example, you’ll
find that they are instances of the wrapper
class. Unless you plan to do introspection on objects’ types,
though, the type issue is probably irrelevant.
As is, this example works as planned under Python 2.6 and 3.0 (provided operator overloading methods to be delegated are redefined in the wrapper). As with most software, though, there is always room for improvement.
Like all delegation-based classes that use __getattr__
, this decorator works
cross-version for normally
named attributes only; operator overloading methods like __str__
and __add__
work differently for new-style
classes and so fail to reach the embedded object if defined there
when this runs under 3.0.
As we learned in the prior chapter, classic classes look up
operator overloading names in instances at runtime normally, but
new-style classes do not—they skip the instance entirely and look
up such methods in classes. Hence, the __
X
__
operator overloading methods
implicitly run for built-in operations do not
trigger either __getattr__
or
__getattribute__
in new-style
classes in 2.6 and all classes in 3.0; such attribute fetches skip
our onInstance.__getattr__
altogether, so they cannot be validated or delegated.
Our decorator’s class is not coded as new-style (by deriving
from object
), so it will catch
operator overloading methods if run under 2.6. Since all classes
are new-style automatically in 3.0, though, such methods will
fail if they are coded on the embedded
object. The simplest workaround in 3.0 is to redefine redundantly
in onInstance
all the operator
overloading methods that can possibly be used in wrapped objects.
Such extra methods can be added by hand, by tools that partly
automate the task (e.g., with class decorators or the metaclasses
discussed in the next chapter), or by definition in superclasses.
To see the difference yourself, try applying the decorator
to a class that uses operator overloading methods under 2.6;
validations work as before, and both the __str__
method used by printing and the
__add__
method run for +
invoke the decorator’s __getattr__
and hence wind up being
validated and delegated to the subject Person
object correctly:
C:misc>c:python26python
>>>from access import Private
>>>@Private('age')
...class Person:
...def __init__(self):
...self.age = 42
...def __str__(self):
...return 'Person: ' + str(self.age)
...def __add__(self, yrs):
...self.age += yrs
... >>>X = Person()
>>>X.age
# Name validations fail correctly TypeError: private attribute fetch: age >>>print(X)
# __getattr__ => runs Person.__str__ Person: 42 >>>X + 10
# __getattr__ => runs Person.__add__ >>>print(X)
# __getattr__ => runs Person.__str__ Person: 52
When the same code is run under Python 3.0, though, the
implicitly invoked __str__
and
__add__
skip the decorator’s
__getattr__
and look for
definitions in or above the decorator class itself; print
winds up finding the default
display inherited from the class type (technically, from the
implied object
superclass in
3.0), and +
generates an error
because no default is inherited:
C:misc>c:python30python
>>>from access import Private
>>>@Private('age')
...class Person:
...def __init__(self):
...self.age = 42
...def __str__(self):
...return 'Person: ' + str(self.age)
...def __add__(self, yrs):
...self.age += yrs
... >>>X = Person()
# Name validations still work >>>X.age
# But 3.0 fails to delegate built-ins! TypeError: private attribute fetch: age >>>print(X)
<access.onInstance object at 0x025E0790> >>>X + 10
TypeError: unsupported operand type(s) for +: 'onInstance' and 'int' >>>print(X)
<access.onInstance object at 0x025E0790>
Using the alternative __getattribute__
method won’t help
here—although it is defined to catch every attribute reference
(not just undefined names), it is also not run by built-in
operations. Python’s property
feature, which we met in Chapter 37,
won’t help here either; recall that properties are automatically
run code associated with specific attributes defined when a class is
written, and are not designed to handle arbitrary attributes in
wrapped objects.
As mentioned earlier, the most straightforward solution
under 3.0 is to redundantly redefine operator overloading names
that may appear in embedded objects in delegation-based classes like our
decorator. This isn’t ideal because it creates some code
redundancy, especially compared to 2.6 solutions. However, it
isn’t too major a coding effort, can be automated to some extent
with tools or superclasses, suffices to make our decorator work in
3.0, and allows operator overloading names to be declared Private
or Public
too (assuming each overloading
method runs the failIf
test
internally):
def accessControl(failIf): def onDecorator(aClass): class onInstance: def __init__(self, *args, **kargs): self.__wrapped = aClass(*args, **kargs) # Intercept and delegate operator overloading methods def __str__(self): return str(self.__wrapped) def __add__(self, other): return self.__wrapped + other def __getitem__(self, index):return self.__wrapped[index]
# If needed def __call__(self, *args, **kargs):return self.__wrapped(*arg, *kargs)
# If needed ...plus any others needed... # Intercept and delegate named attributes def __getattr__(self, attr): ... def __setattr__(self, attr, value): ... return onInstance return onDecorator
With such operator overloading methods added, the prior
example with __str__
and
__add__
works the same under
2.6 and 3.0, although a substantial amount of extra code may be
required to accommodate 3.0—in principle,
every operator overloading method that is not
run automatically will need to be defined redundantly for 3.0 in a
general tool class like this (which is why this extension is
omitted in our code). Since every class is new-style in 3.0,
delegation-based code is more difficult (though not impossible) in
this release.
On the other hand, delegation wrappers could simply inherit from a common superclass that redefines operator overloading methods once, with standard delegation code. Moreover, tools such as additional class decorators or metaclasses might automate some of the work of adding such methods to delegation classes (see the class augmentation examples in Chapter 39 for details). Though still not as simple as the 2.6 solution, such techniques might help make 3.0 delegation classes more general.
Although redundantly defining operator overloading methods in wrappers is probably the most straightforward workaround to Python 3.0 dilemma outlined in the prior section, it’s not necessarily the only one. We don’t have space to explore this issue much further here, so investigating other potential solutions is relegated to a suggested exercise. Because one dead-end alternative underscores class concepts well, though, it merits a brief mention.
One downside of this example is that instance objects are
not truly instances of the original class—they are instances of
the wrapper instead. In some programs that rely on type testing,
this might matter. To support such cases, we might try to achieve
similar effects by inserting a __getattribute__
method into the
original class, to catch every attribute
reference made on its instances. This inserted method would pass
valid requests up to its
superclass to avoid loops, using the techniques we studied in the
prior chapter. Here is the potential change to our class
decorator’s code:
# trace support as before
def accessControl(failIf):
def onDecorator(aClass):
def getattributes(self, attr):
trace('get:', attr)
if failIf(attr):
raise TypeError('private attribute fetch: ' + attr)
else:
return object.__getattribute__(self, attr)
aClass.__getattribute__ = getattributes
return aClass
return onDecorator
def Private(*attributes):
return accessControl(failIf=(lambda attr: attr in attributes))
def Public(*attributes):
return accessControl(failIf=(lambda attr: attr not in attributes))
This alternative addresses the type-testing issue but
suffers from others. For example, it handles only attribute
fetches—as is, this version allows private
names to be assigned freely. Intercepting
assignments would still have to use __setattr__
, and either an instance
wrapper object or another class method insertion. Adding an
instance wrapper to catch assignments would change the type again,
and inserting methods fails if the original class is using a
__setattr__
of its own (or a
__getattribute__
, for that
matter!). An inserted __setattr__
would also have to allow for
a __slots__
in the client
class.
In addition, this scheme does not address the
built-in operation attributes issue described in the prior section, since
__getattribute__
is not run in
these contexts, either. In our case, if Person
had a __str__
it would be run by print
operations, but only because it was actually present in that
class. As before, the __str__
attribute would not be routed to the inserted
__getattribute__
method
generically—printing would bypass this method altogether and call
the class’s __str__
directly.
Although this is probably better than not supporting
operator overloading methods in a wrapped object at all (barring
redefinition, at least), this scheme still cannot intercept and
validate __
X
__
methods, making it impossible for any
of them to be Private
. Although
most operator overloading methods are meant to be public, some
might not be.
Much worse, because this nonwrapper approach works by adding
a __getattribute__
to the decorated
class, it also intercepts attribute accesses made by the
class itself and validates them the same as accesses
made from outside—this means the class’s method won’t be able to
use Private
names,
either!
In fact, inserting methods this way is functionally equivalent to inheriting them, and implies the same constraints as our original Chapter 29 privacy code. To know whether an attribute access originated inside or outside the class, our method might need to inspect frame objects on the Python call stack. This might ultimately yield a solution (replace private attributes with properties or descriptors that check the stack, for example), but it would slow access further and is far too dark a magic for us to explore here.
While interesting, and possibly relevant for some other use cases, this method insertion technique doesn’t meet our goals. We won’t explore this option’s coding pattern further here because we will study class augmentation techniques in the next chapter, in conjunction with metaclasses. As we’ll see there, metaclasses are not strictly required for changing classes this way, because class decorators can often serve the same role.
Now that I’ve gone to such great lengths to add Private
and Public
attribute declarations for Python
code, I must again remind you that it is not entirely
Pythonic to add access controls to your classes
like this. In fact, most Python programmers will probably find this
example to be largely or totally irrelevant, apart from serving as a
demonstration of decorators in action. Most large Python programs
get by successfully without any such controls at all. If you do wish
to regulate attribute access in order to eliminate coding mistakes,
though, or happen to be a soon-to-be-ex-C++-or-Java programmer, most
things are possible with Python’s operator overloading and
introspection tools.
As a final example of the utility of decorators, this section develops a function decorator that automatically tests whether arguments passed to a function or method are within a valid numeric range. It’s designed to be used during either development or production, and it can be used as a template for similar tasks (e.g., argument type testing, if you must). Because this chapter’s size limits has been broached, this example’s code is largely self-study material, with limited narrative; as usual, browse the code for more details.
In the object-oriented tutorial of Chapter 27, we wrote a class that gave a raise to objects representing people based upon a passed-in percentage:
class Person: ... def giveRaise(self, percent): self.pay = int(self.pay * (1 + percent))
There, we noted that if we wanted the code to be robust it
would be a good idea to check the percentage to make sure it’s not
too large or too small. We could implement such a check with either
if
or assert
statements in the method itself,
using inline tests:
class Person: def giveRaise(self, percent): # Validate with inline code if percent < 0.0 or percent > 1.0: raise TypeError, 'percent invalid' self.pay = int(self.pay * (1 + percent)) class Person: # Validate with asserts def giveRaise(self, percent): assert percent >= 0.0 and percent <= 1.0, 'percent invalid' self.pay = int(self.pay * (1 + percent))
However, this approach clutters up the method with inline tests that will probably be useful only during development. For more complex cases, this can become tedious (imagine trying to inline the code needed to implement the attribute privacy provided by the last section’s decorator). Perhaps worse, if the validation logic ever needs to change, there may be arbitrarily many inline copies to find and update.
A more useful and interesting alternative would be to develop a general tool that can perform range tests for us automatically, for the arguments of any function or method we might code now or in the future. A decorator approach makes this explicit and convenient:
class Person:
@rangetest(percent=(0.0, 1.0)) # Use decorator to validate
def giveRaise(self, percent):
self.pay = int(self.pay * (1 + percent))
Isolating validation logic in a decorator simplifies both clients and future maintenance.
Notice that our goal here is different than the attribute validations coded in the prior chapter’s final example. Here, we mean to validate the values of function arguments when passed, rather than attribute values when set. Python’s decorator and introspection tools allow us to code this new task just as easily.
Let’s start with a basic range test implementation. To keep
things simple, we’ll begin by coding a decorator that works only for
positional arguments and assumes they always appear at the same
position in every call; they cannot be passed by keyword name, and
we don’t support additional **args
keywords in calls because this can
invalidate the positions declared in the decorator. Code the
following in a file called devtools.py:
def rangetest(*argchecks): # Validate positional arg ranges def onDecorator(func): if not __debug__: # True if "python -O main.py args..." return func # No-op: call original directly else: # Else wrapper while debugging def onCall(*args): for (ix, low, high) in argchecks: if args[ix] < low or args[ix] > high: errmsg = 'Argument %s not in %s..%s' % (ix, low, high) raise TypeError(errmsg) return func(*args) return onCall return onDecorator
As is, this code is mostly a rehash of the coding patterns we explored earlier: we use decorator arguments, nested scopes for state retention, and so on.
We also use nested def
statements to ensure that this works for both simple functions and
methods, as we learned earlier. When used for a class method,
onCall
receives the subject
class’s instance in the first item in *args
and passes this along to self
in the original method function;
argument numbers in range tests start at 1 in this case, not
0.
Also notice this code’s use of the __debug__
built-in variable, though—Python
sets this to True
, unless it’s
being run with the –O
optimize
command-line flag (e.g., python –O
main.py
). When __debug__
is False
, the decorator returns the origin
function unchanged, to avoid extra calls and their associated
performance penalty.
This first iteration solution is used as follows:
# File devtools_test.py from devtools import rangetest print(__debug__) # False if "python –O main.py" @rangetest((1, 0, 120)) # persinfo = rangetest(...)(persinfo) def persinfo(name, age): # age must be in 0..120 print('%s is %s years old' % (name, age)) @rangetest([0, 1, 12], [1, 1, 31], [2, 0, 2009]) def birthday(M, D, Y): print('birthday = {0}/{1}/{2}'.format(M, D, Y)) class Person: def __init__(self, name, job, pay): self.job = job self.pay = pay @rangetest([1, 0.0, 1.0]) # giveRaise = rangetest(...)(giveRaise) def giveRaise(self, percent): # Arg 0 is the self instance here self.pay = int(self.pay * (1 + percent)) # Comment lines raise TypeError unless "python -O" used on shell command line persinfo('Bob Smith', 45) # Really runs onCall(...) with state #persinfo('Bob Smith', 200) # Or person if –O cmd line argument birthday(5, 31, 1963) #birthday(5, 32, 1963) sue = Person('Sue Jones', 'dev', 100000) sue.giveRaise(.10) # Really runs onCall(self, .10) print(sue.pay) # Or giveRaise(self, .10) if –O #sue.giveRaise(1.10) #print(sue.pay)
When run, valid calls in this code produce the following
output (all the code in this section works the same under Python 2.6
and 3.0, because function decorators are supported in both, we’re
not using attribute delegation, and we use 3.0-style print
calls and exception construction
syntax):
C:misc> C:python30python devtools_test.py
True
Bob Smith is 45 years old
birthday = 5/31/1963
110000
Uncommenting any of the invalid calls causes a TypeError
to be raised by the decorator.
Here’s the result when the last two lines are allowed to run (as
usual, I’ve omitted some of the error message text here to save
space):
C:misc> C:python30python devtools_test.py
True
Bob Smith is 45 years old
birthday = 5/31/1963
110000
TypeError: Argument 1 not in 0.0..1.0
Running Python with its –O
flag at a system command line will disable range testing, but also
avoid the performance overhead of the wrapping layer—we wind up
calling the original undecorated function directly. Assuming this is
a debugging tool only, you can use this flag to optimize your
program for production use:
C:misc> C:python30python –O devtools_test.py
False
Bob Smith is 45 years old
birthday = 5/31/1963
110000
231000
The prior version illustrates the basics we need to employ, but it’s fairly limited—it supports validating arguments passed by position only, and it does not validate keyword arguments (in fact, it assumes that no keywords are passed in a way that makes argument position numbers incorrect). Additionally, it does nothing about arguments with defaults that may be omitted in a given call. That’s fine if all your arguments are passed by position and never defaulted, but less than ideal in a general tool. Python supports much more flexible argument-passing modes, which we’re not yet addressing.
The mutation of our example shown next does better. By
matching the wrapped function’s expected arguments against the
actual arguments passed in a call, it supports range validations for
arguments passed by either position or keyword name, and it skips
testing for default arguments omitted in the call. In short,
arguments to be validated are specified by keyword arguments to the
decorator, which later steps through both the *pargs
positionals tuple and the **kargs
keywords dictionary to
validate.
""" File devtools.py: function decorator that performs range-test validation for passed arguments. Arguments are specified by keyword to the decorator. In the actual call, arguments may be passed by position or keyword, and defaults may be omitted. See devtools_test.py for example use cases. """ trace = True def rangetest(**argchecks): # Validate ranges for both+defaults def onDecorator(func): # onCall remembers func and argchecks if not __debug__: # True if "python –O main.py args..." return func # Wrap if debugging; else use original else: import sys code = func.__code__ allargs = code.co_varnames[:code.co_argcount] funcname = func.__name__ def onCall(*pargs, **kargs): # All pargs match first N expected args by position # The rest must be in kargs or be omitted defaults positionals = list(allargs) positionals = positionals[:len(pargs)] for (argname, (low, high)) in argchecks.items(): # For all args to be checked if argname in kargs: # Was passed by name if kargs[argname] < low or kargs[argname] > high: errmsg = '{0} argument "{1}" not in {2}..{3}' errmsg = errmsg.format(funcname, argname, low, high) raise TypeError(errmsg) elif argname in positionals: # Was passed by position position = positionals.index(argname) if pargs[position] < low or pargs[position] > high: errmsg = '{0} argument "{1}" not in {2}..{3}' errmsg = errmsg.format(funcname, argname, low, high) raise TypeError(errmsg) else: # Assume not passed: default if trace: print('Argument "{0}" defaulted'.format(argname)) return func(*pargs, **kargs) # OK: run original call return onCall return onDecorator
The following test script shows how the decorator is used—arguments to be validated are given by keyword decorator arguments, and at actual calls we can pass by name or position and omit arguments with defaults even if they are to be validated otherwise:
# File devtools_test.py # Comment lines raise TypeError unless "python –O" used on shell command line from devtools import rangetest # Test functions, positional and keyword @rangetest(age=(0, 120)) # persinfo = rangetest(...)(persinfo) def persinfo(name, age): print('%s is %s years old' % (name, age)) @rangetest(M=(1, 12), D=(1, 31), Y=(0, 2009)) def birthday(M, D, Y): print('birthday = {0}/{1}/{2}'.format(M, D, Y)) persinfo('Bob', 40) persinfo(age=40, name='Bob') birthday(5, D=1, Y=1963) #persinfo('Bob', 150) #persinfo(age=150, name='Bob') #birthday(5, D=40, Y=1963) # Test methods, positional and keyword class Person: def __init__(self, name, job, pay): self.job = job self.pay = pay # giveRaise = rangetest(...)(giveRaise) @rangetest(percent=(0.0, 1.0)) # percent passed by name or position def giveRaise(self, percent): self.pay = int(self.pay * (1 + percent)) bob = Person('Bob Smith', 'dev', 100000) sue = Person('Sue Jones', 'dev', 100000) bob.giveRaise(.10) sue.giveRaise(percent=.20) print(bob.pay, sue.pay) #bob.giveRaise(1.10) #bob.giveRaise(percent=1.20) # Test omitted defaults: skipped @rangetest(a=(1, 10), b=(1, 10), c=(1, 10), d=(1, 10)) def omitargs(a, b=7, c=8, d=9): print(a, b, c, d) omitargs(1, 2, 3, 4) omitargs(1, 2, 3) omitargs(1, 2, 3, d=4) omitargs(1, d=4) omitargs(d=4, a=1) omitargs(1, b=2, d=4) omitargs(d=8, c=7, a=1) #omitargs(1, 2, 3, 11) # Bad d #omitargs(1, 2, 11) # Bad c #omitargs(1, 2, 3, d=11) # Bad d #omitargs(11, d=4) # Bad a #omitargs(d=4, a=11) # Bad a #omitargs(1, b=11, d=4) # Bad b #omitargs(d=8, c=7, a=11) # Bad a
When this script is run, out-of-range arguments raise an exception as before, but arguments may be passed by either name or position, and omitted defaults are not validated. This code runs on both 2.6 and 3.0, but extra tuple parentheses print in 2.6. Trace its output and test this further on your own to experiment; it works as before, but its scope has been broadened:
C:misc> C:python30python devtools_test.py
Bob is 40 years old
Bob is 40 years old
birthday = 5/1/1963
110000 120000
1 2 3 4
Argument "d" defaulted
1 2 3 9
1 2 3 4
Argument "c" defaulted
Argument "b" defaulted
1 7 8 4
Argument "c" defaulted
Argument "b" defaulted
1 7 8 4
Argument "c" defaulted
1 2 8 4
Argument "b" defaulted
1 7 7 8
On validation errors, we get an exception as before (unless
the –O
command-line argument is
passed to Python) when one of the method test lines is
uncommented:
TypeError: giveRaise argument "percent" not in 0.0..1.0
This decorator’s code relies on both introspection APIs and subtle constraints of argument passing. To be fully general we could in principle try to mimic Python’s argument matching logic in its entirety to see which names have been passed in which modes, but that’s far too much complexity for our tool. It would be better if we could somehow match arguments passed by name against the set of all expected arguments’ names, in order to determine which position arguments actually appear in during a given call.
It turns out that the introspection API available on function objects and their associated code objects has exactly the tool we need. This API was briefly introduced in Chapter 19, but we’ll actually put it to use here. The set of expected argument names is simply the first N variable names attached to a function’s code object:
# In Python 3.0 (and 2.6 for compatibility): >>>def func(a, b, c, d):
...x = 1
...y = 2
... >>>code = func.__code__
# Code object of function object >>>code.co_nlocals
6 >>>code.co_varnames
# All local var names ('a', 'b', 'c', 'd', 'x', 'y') >>>code.co_varnames[:code.co_argcount]
# First N locals are expected args ('a', 'b', 'c', 'd') >>>import sys
# For backward compatibility >>>sys.version_info
# [0] is major release number (3, 0, 0, 'final', 0) >>>code = func.__code__ if sys.version_info[0] == 3 else func.func_code
The same API is available in older Pythons, but the func.__code__
attribute is spelled as
func.func_code
in 2.5 and
earlier (the newer __code__
attribute is also redundantly available in 2.6 for portability).
Run a dir
call on function and
code objects for more details.
Given this set of expected argument names, the solution relies on two constraints on argument passing order imposed by Python (these still hold true in both 2.6 and 3.0):
At the call, all positional arguments appear before all keyword arguments.
In the def
, all
nondefault arguments appear before all default
arguments.
That is, a nonkeyword argument cannot generally follow a keyword argument at a call, and a nondefault argument cannot follow a default argument at a definition. All “name=value” syntax must appear after any simple “name” in both places.
To simplify our work, we can also make the assumption that a call is valid in general—i.e., that all arguments either will receive values (by name or position), or will be omitted intentionally to pick up defaults. This assumption won’t necessarily hold, because the function has not yet actually been called when the wrapper logic tests validity—the call may still fail later when invoked by the wrapper layer, due to incorrect argument passing. As long as that doesn’t cause the wrapper to fail any more badly, though, we can finesse the validity of the call. This helps, because validating calls before they are actually made would require us to emulate Python’s argument-matching algorithm in full—again, too complex a procedure for our tool.
Now, given these constraints and assumptions, we can allow for both keywords and omitted default arguments in the call with this algorithm. When a call is intercepted, we can make the following assumptions:
All N passed positional arguments
in *pargs
must match the
first N expected arguments obtained from
the function’s code object. This is true per Python’s call
ordering rules, outlined earlier, since all positionals
precede all keywords.
To obtain the names of arguments actually passed by
position, we can slice the list of all expected arguments up
to the length N of the *pargs
positionals tuple.
Any arguments after the first N expected arguments either were passed by keyword or were defaulted by omission at the call.
For each argument name to be validated, if it is in
**kargs
it was passed by
name, and if it is in the first N
expected arguments it was passed by position (in which case
its relative position in the expected list gives its relative
position in *pargs
);
otherwise, we can assume it was omitted in the call and
defaulted and need not be checked.
In other words, we can skip tests for arguments that were
omitted in a call by assuming that the first
N actually passed positional arguments in
*pargs
must match the first
N argument names in the list of all expected
arguments, and that any others must either have been passed by
keyword and thus be in **kargs
,
or have been defaulted. Under this scheme, the decorator will
simply skip any argument to be checked that was omitted between
the rightmost positional argument and the leftmost keyword
argument, between keyword arguments, or after the rightmost
positional in general. Trace through the decorator and its test
script to see how this is realized in code.
Although our range-testing tool works as planned, two caveats remain. First, as mentioned earlier, calls to the original function that are not valid still fail in our final decorator. The following both trigger exceptions, for example:
omitargs() omitargs(d=8, c=7, b=6)
These only fail, though, where we try to invoke the original function, at the end of the wrapper. While we could try to imitate Python’s argument matching to avoid this, there’s not much reason to do so—since the call would fail at this point anyhow, we might as well let Python’s own argument-matching logic detect the problem for us.
Lastly, although our final version handles positional
arguments, keyword arguments, and omitted defaults, it still doesn’t
do anything explicit about *args
and **args
that may be used in a
decorated function that accepts arbitrarily many arguments. We
probably don’t need to care for our purposes, though:
If an extra keyword argument is
passed, its name will show up in **kargs
and can be tested normally if
mentioned to the decorator.
If an extra keyword argument is not
passed, its name won’t be in either **kargs
or the sliced expected
positionals list, and it will thus not be checked—it is treated
as though it were defaulted, even though it is really an
optional extra argument.
If an extra positional argument is
passed, there’s no way to reference it in the decorator
anyhow—its name won’t be in either **kargs
or the sliced expected
arguments list, so it will simply be skipped. Because such
arguments are not listed in the function’s definition, there’s
no way to map a name given to the decorator back to an expected
relative position.
In other words, as it is the code supports testing arbitrary keyword arguments by name, but not arbitrary positionals that are unnamed and hence have no set position in the function’s argument signature.
In principle, we could extend the decorator’s interface to
support *args
in the decorated
function, too, for the rare cases where this might be useful (e.g.,
a special argument name with a test to apply to all arguments in the
wrapper’s *pargs
beyond the
length of the expected arguments list). Since we’ve already
exhausted the space allocation for this example, though, if you care
about such improvements you’ve officially crossed over into the
realm of suggested exercises.
Interestingly, the function annotation feature introduced in Python
3.0 could provide an alternative to the decorator arguments used by
our example to specify range tests. As we learned in Chapter 19, annotations allow us to
associate expressions with arguments and return values, by coding
them in the def
header line
itself; Python collects annotations in a dictionary and attaches it
to the annotated function.
We could use this in our example to code range limits in the header line, instead of in decorator arguments. We would still need a function decorator to wrap the function in order to intercept later calls, but we would essentially trade decorator argument syntax:
@rangetest(a=(1, 5), c=(0.0, 1.0))
def func(a, b, c): # func = rangetest(...)(func)
print(a + b + c)
for annotation syntax like this:
@rangetest def func(a:(1, 5), b, c:(0.0, 1.0)): print(a + b + c)
That is, the range constraints would be moved into the function itself, instead of being coded externally. The following script illustrates the structure of the resulting decorators under both schemes, in incomplete skeleton code. The decorator arguments code pattern is that of our complete solution shown earlier; the annotation alternative requires one less level of nesting, because it doesn’t need to retain decorator arguments:
# Using decorator arguments def rangetest(**argchecks): def onDecorator(func): def onCall(*pargs, **kargs): print(argchecks) for check in argchecks: pass # Add validation code here return func(*pargs, **kargs) return onCall return onDecorator @rangetest(a=(1, 5), c=(0.0, 1.0)) def func(a, b, c): # func = rangetest(...)(func) print(a + b + c) func(1, 2, c=3) # Runs onCall, argchecks in scope # Using function annotations def rangetest(func): def onCall(*pargs, **kargs): argchecks = func.__annotations__ print(argchecks) for check in argchecks: pass # Add validation code here return func(*pargs, **kargs) return onCall @rangetest def func(a:(1, 5), b, c:(0.0, 1.0)): # func = rangetest(func) print(a + b + c) func(1, 2, c=3) # Runs onCall, annotations on func
When run, both schemes have access to the same validation test information, but in different forms—the decorator argument version’s information is retained in an argument in an enclosing scope, and the annotation version’s information is retained in an attribute of the function itself:
{'a': (1, 5), 'c': (0.0, 1.0)} 6 {'a': (1, 5), 'c': (0.0, 1.0)} 6
I’ll leave fleshing out the rest of the annotation-based version as a suggested exercise; its code would be identical to that of our complete solution shown earlier, because range-test information is simply on the function instead of in an enclosing scope. Really, all this buys us is a different user interface for our tool—it will still need to match argument names against expected argument names to obtain relative positions as before.
In fact, using annotation instead of decorator arguments in this example actually limits its utility. For one thing, annotation only works under Python 3.0, so 2.6 is no longer supported; function decorators with arguments, on the other hand, work in both versions.
More importantly, by moving the validation specifications into
the def
header, we essentially
commit the function to a single role—since
annotation allows us to code only one expression per argument, it
can have only one purpose. For instance, we cannot use range-test
annotations for any other role.
By contrast, because decorator arguments are coded outside the function itself, they are both easier to remove and more general—the code of the function itself does not imply a single decoration purpose. In fact, by nesting decorators with arguments, we can apply multiple augmentation steps to the same function; annotation directly supports only one. With decorator arguments, the function itself also retains a simpler, normal appearance.
Still, if you have a single purpose in mind, and you can commit to supporting 3.X only, the choice between annotation and decorator arguments is largely stylistic and subjective. As is so often true in life, one person’s annotation may well be another’s syntactic clutter....
The coding pattern we’ve arrived at for processing arguments in decorators could be applied in other contexts. Checking argument data types at development time, for example, is a straightforward extension:
def typetest(**argchecks): def onDecorator(func): .... def onCall(*pargs, **kargs): positionals = list(allargs)[:len(pargs)] for (argname, type) in argchecks.items(): if argname in kargs: if not isinstance(kargs[argname], type): ... raise TypeError(errmsg) elif argname in positionals: position = positionals.index(argname) if not isinstance(pargs[position], type): ... raise TypeError(errmsg) else: # Assume not passed: default return func(*pargs, **kargs) return onCall return onDecorator @typetest(a=int, c=float) def func(a, b, c, d): # func = typetest(...)(func) ... func(1, 2, 3.0, 4) # Okay func('spam', 2, 99, 4) # Triggers exception correctly
In fact, we might even generalize further by passing in a test
function, much as we did to add Public
decorations earlier; a single copy
of this sort of code would suffice for both range and type testing.
Using function annotations instead of decorator arguments for such a
decorator, as described in the prior section, would make this look
even more like type declarations in other languages:
@typetest def func(a: int, b, c: float, d): # func = typetest(func) ... # Gasp!...
As you should have learned in this book, though, this particular role is generally a bad idea in working code, and not at all Pythonic (in fact, it’s often a symptom of an ex-C++ programmer’s first attempts to use Python).
Type testing restricts your function to work on specific types only, instead of allowing it to operate on any types with compatible interfaces. In effect, it limits your code and breaks its flexibility. On the other hand, every rule has exceptions; type checking may come in handy in isolated cases while debugging and when interfacing with code written in more restrictive languages, such as C++. This general pattern of argument processing might also be applicable in a variety of less controversial roles.
In this chapter, we explored decorators—both the function and class varieties. As we learned, decorators are a way to insert code to be run automatically when a function or class is defined. When a decorator is used, Python rebinds a function or class name to the callable object it returns. This hook allows us to add a layer of wrapper logic to function calls and class instance creation calls, in order to manage functions and instances. As we also saw, manager functions and manual name rebinding can achieve the same effect, but decorators provide a more explicit and uniform solution.
As we’ll see in the next chapter, class decorators can also be used to manage classes themselves, rather than just their instances. Because this functionality overlaps with metaclasses, the topic of the next chapter, you’ll have to read ahead for the rest of this story. First, though, work through the following quiz. Because this chapter was mostly focused on its larger examples, its quiz will ask you to modify some of its code in order to review.
As mentioned in one of this chapter’s Notes, the timer
function decorator with decorator arguments that we wrote in the
section Adding Decorator Arguments can be
applied only to simple functions, because it
uses a nested class with a __call__
operator overloading method to
catch calls. This structure does not work for class
methods because the decorator instance is
passed to self
, not the subject
class instance. Rewrite this decorator so that it can be applied
to both simple functions and class methods, and test it on both
functions and methods. (Hint: see the section Class Blunders I: Decorating Class Methods for
pointers.) Note that you may make use of assigning function object
attributes to keep track of total time, since you won’t have a
nested class for state retention and can’t access nonlocals from
outside the decorator code.
The Public
/Private
class decorators we wrote in
this chapter will add overhead to every attribute fetch in a
decorated class. Although we could simply delete the @
decoration line to gain speed, we
could also augment the decorator itself to check the __debug__
switch and perform no wrapping
at all when the –O
Python flag
is passed on the command line (just as we did for the argument
range-test decorators). That way, we can speed our program without
changing its source, via command-line arguments (python –O main.py...
). Code and test
this extension.
Here’s one way to code the first question’s solution, and
its output (albeit with class methods that run too fast to time).
The trick lies in replacing nested classes with nested
functions, so the self
argument is not the decorator’s
instance, and assigning the total time to the decorator function
itself so it can be fetched later through the original rebound
name (see the section State Information Retention Options of this chapter
for details—functions support arbitrary attribute attachment, and
the function name is an enclosing scope reference in this
context).
import time def timer(label='', trace=True): # On decorator args: retain args def onDecorator(func): # On @: retain decorated func def onCall(*args, **kargs): # On calls: call original start = time.clock() # State is scopes + func attr result = func(*args, **kargs) elapsed = time.clock() - start onCall.alltime += elapsed if trace: format = '%s%s: %.5f, %.5f' values = (label, func.__name__, elapsed, onCall.alltime) print(format % values) return result onCall.alltime = 0 return onCall return onDecorator # Test on functions @timer(trace=True, label='[CCC]==>') def listcomp(N): # Like listcomp = timer(...)(listcomp) return [x * 2 for x in range(N)] # listcomp(...) triggers onCall @timer(trace=True, label='[MMM]==>') def mapcall(N): return list(map((lambda x: x * 2), range(N))) # list() for 3.0 views for func in (listcomp, mapcall): result = func(5) # Time for this call, all calls, return value func(5000000) print(result) print('allTime = %s ' % func.alltime) # Total time for all calls # Test on methods class Person: def __init__(self, name, pay): self.name = name self.pay = pay @timer() def giveRaise(self, percent): # giveRaise = timer()(giveRaise) self.pay *= (1.0 + percent) # tracer remembers giveRaise @timer(label='**') def lastName(self): # lastName = timer(...)(lastName) return self.name.split()[-1] # alltime per class, not instance bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) bob.giveRaise(.10) sue.giveRaise(.20) # runs onCall(sue, .10) print(bob.pay, sue.pay) print(bob.lastName(), sue.lastName()) # runs onCall(bob), remembers lastName print('%.5f %.5f' % (Person.giveRaise.alltime, Person.lastName.alltime)) # Expected output [CCC]==>listcomp: 0.00002, 0.00002 [CCC]==>listcomp: 1.19636, 1.19638 [0, 2, 4, 6, 8] allTime = 1.19637775192 [MMM]==>mapcall: 0.00002, 0.00002 [MMM]==>mapcall: 2.29260, 2.29262 [0, 2, 4, 6, 8] allTime = 2.2926232943 giveRaise: 0.00001, 0.00001 giveRaise: 0.00001, 0.00002 55000.0 120000.0 **lastName: 0.00001, 0.00001 **lastName: 0.00001, 0.00002 Smith Jones 0.00002 0.00002
The following satisfies the second question—it’s been
augmented to return the original class in optimized mode (–O
), so attribute accesses don’t incur a
speed hit. Really, all I did was add the debug mode test
statements and indent the class further to the right. Add operator
overloading method redefinitions to the wrapper class if you want
to support delegation of these to the subject class in 3.0, too
(2.6 routes these through __getattr__
, but 3.0 and new-style
classes in 2.6 do not).
traceMe = False def trace(*args): if traceMe: print('[' + ' '.join(map(str, args)) + ']') def accessControl(failIf): def onDecorator(aClass): if not __debug__: return aClass else: class onInstance: def __init__(self, *args, **kargs): self.__wrapped = aClass(*args, **kargs) def __getattr__(self, attr): trace('get:', attr) if failIf(attr): raise TypeError('private attribute fetch: ' + attr) else: return getattr(self.__wrapped, attr) def __setattr__(self, attr, value): trace('set:', attr, value) if attr == '_onInstance__wrapped': self.__dict__[attr] = value elif failIf(attr): raise TypeError('private attribute change: ' + attr) else: setattr(self.__wrapped, attr, value) return onInstance return onDecorator def Private(*attributes): return accessControl(failIf=(lambda attr: attr in attributes)) def Public(*attributes): return accessControl(failIf=(lambda attr: attr not in attributes)) # Test code: split me off to another file to reuse decorator @Private('age') # Person = Private('age')(Person) class Person: # Person = onInstance with state def __init__(self, name, age): self.name = name self.age = age # Inside accesses run normally X = Person('Bob', 40) print(X.name) # Outside accesses validated X.name = 'Sue' print(X.name) #print(X.age) # FAILS unles "python -O" #X.age = 999 # ditto #print(X.age) # ditto @Public('name') class Person: def __init__(self, name, age): self.name = name self.age = age X = Person('bob', 40) # X is an onInstance print(X.name) # onInstance embeds Person X.name = 'Sue' print(X.name) #print(X.age) # FAILS unless "python –O main.py" #X.age = 999 # ditto #print(X.age) # ditto