In the prior chapter, we explored decorators and studied various examples of their use. In this final chapter of the book, we’re going continue our tool-builders focus and investigate another advanced topic: metaclasses.
In a sense, metaclasses simply extend the code-insertion model of
decorators. As we learned in the prior chapter, function and class
decorators allow us to intercept and augment function calls and class
instance creation calls. In a similar sprit, metaclasses allow us to
intercept and augment class creation—they provide
an API for inserting extra logic to be run at the conclusion of a
class
statement, albeit in different
ways than decorators. As such, they provide a general protocol for
managing class objects in a program.
Like all the subjects dealt with in this part of the book, this is an advanced topic that can be investigated on an as-needed basis. In practice, metaclasses allow us to gain a high level of control over how a set of classes work. This is a powerful concept, and metaclasses are not intended for most application programmers (nor, frankly, the faint of heart!).
On the other hand, metaclasses open the door to a variety of coding patterns that may be difficult or impossible to achieve otherwise, and they are especially of interest to programmers seeking to write flexible APIs or programming tools for others to use. Even if you don’t fall into that category, metaclasses can teach you much about Python’s class model in general.
As in the prior chapter, part of our goal here is also to show more realistic code examples than we did earlier in this book. Although metaclasses are a core language topic and not themselves an application domain, part of this chapter’s goal is to spark your interest in exploring larger application-programming examples after you finish this book.
Metaclasses are perhaps the most advanced topic in this book, if not the Python language as a whole. To borrow a quote from the comp.lang.python newsgroup by veteran Python core developer Tim Peters (who is also the author of the famous “import this” Python motto):
[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).
In other words, metaclasses are primarily intended for programmers building APIs and tools for others to use. In many (if not most) cases, they are probably not the best choice in applications work. This is especially true if you’re developing code that other people will use in the future. Coding something “because it seems cool” is not generally a reasonable justification, unless you are experimenting or learning.
Still, metaclasses have a wide variety of potential roles, and it’s important to know when they can be useful. For example, they can be used to enhance classes with features like tracing, object persistence, exception logging, and more. They can also be used to construct portions of a class at runtime based upon configuration files, apply function decorators to every method of a class generically, verify conformance to expected interfaces, and so on.
In their more grandiose incarnations, metaclasses can even be used to implement alternative coding patterns such as aspect-oriented programming, object/relational mappers (ORMs) for databases, and more. Although there are often alternative ways to achieve such results (as we’ll see, the roles of class decorators and metaclasses often intersect), metaclasses provide a formal model tailored to those tasks. We don’t have space to explore all such applications first-hand in this chapter but you should feel free to search the Web for additional use cases after studying the basics here.
Probably the reason for studying metaclasses most relevant to this book is that this topic can help demystify Python’s class mechanics in general. Although you may or may not code or reuse them in your work, a cursory understanding of metaclasses can impart a deeper understanding of Python at large.
Most of this book has focused on straightforward application-coding techniques, as most programmers spend their time writing modules, functions, and classes to achieve real-world goals. They may use classes and make instances, and might even do a bit of operator overloading, but they probably won’t get too deep into the details of how their classes actually work.
However, in this book we’ve also seen a variety of tools that allow us to control Python’s behavior in generic ways, and that often have more to do with Python internals or tool building than with application-programming domains:
Special attributes like __class__
and __dict__
allow us to inspect
internal implementation aspects of Python objects, in order to
process them generically—to list all attributes of an object,
display a class’s name, and so on.
Specially named methods such as __str__
and __add__
coded in classes intercept
and provide behavior for built-in operations applied to class
instances, such as printing, expression operators, and so on.
They are run automatically in response to built-in operations
and allow classes to conform to expected interfaces.
A special category of operator overloading methods
provide a way to intercept attribute accesses on instances
generically: __getattr__
,
__setattr__
, and __getattribute__
allow wrapper
classes to insert automatically run code that may validate
attribute requests and delegate them to embedded objects. They
allow any number of attributes of an object—either selected
attributes, or all of them—to be computed when
accessed.
The property
built-in allows us to associate code with a
specific class attribute that is automatically run when the
attribute is fetched, assigned, or deleted. Though not as
generic as the prior paragraph’s tools, properties allow for
automatic code invocation on access to specific
attributes.
Really, property
is a succinct way to define
an attribute descriptor that runs functions on access
automatically. Descriptors allow us to code in a separate
class __get__
, __set__
, and __delete__
handler methods that are
run automatically when an attribute assigned to an instance of
that class is accessed. They provide a general way to insert
automatically run code when a specific attribute is accessed,
and they are triggered after an attribute is looked up
normally.
As we saw in Chapter 38, the
special @callable
syntax
for decorators allows us to add logic to be automatically run
when a function is called or a class instance is created. This
wrapper logic can trace or time calls, validate arguments,
manage all instances of a class, augment instances with extra
behavior such as attribute fetch validation, and more.
Decorator syntax inserts name-rebinding logic to be run at the
end of function and class definition statements—decorated
function and class names are rebound to callable objects that
intercept later calls.
As mentioned in this chapter’s
introduction, metaclasses are a continuation of
this story—they allow us to
insert logic to be run automatically when a class object is created,
at the end of a class
statement.
This logic doesn’t rebind the class name to a decorator callable,
but rather routes creation of the class itself to specialized
logic.
In other words, metaclasses are ultimately just another way to define automatically run code. Via metaclasses and the other tools just listed, Python provides ways for us to interject logic in a variety of contexts—at operator evaluation, attribute access, function calls, class instance creation, and now class object creation.
Unlike class decorators, which usually add logic to be run at instance creation time, metaclasses run at class creation time; as such, they are hooks generally used for managing or augmenting classes, instead of their instances.
For example, metaclasses can be used to add decoration to all methods of classes automatically, register all classes in use to an API, add user-interface logic to classes automatically, create or extend classes from simplified specifications in text files, and so on. Because we can control how classes are made (and by proxy the behavior their instances acquire), their applicability is potentially very wide.
As we’ve also seen, many of these advanced Python tools have intersecting roles. For example, attributes can often be managed with properties, descriptors, or attribute interception methods. As we’ll see in this chapter, class decorators and metaclasses can often be used interchangeably as well. Although class decorators are often used to manage instances, they can be used to manage classes instead; similarly, while metaclasses are designed to augment class construction, they can often insert code to manage instances, too. Since the choice of which technique to use is sometimes purely subjective, knowledge of the alternatives can help you pick the right tool for a given task.
Also like the decorators of the prior chapter, metaclasses are often optional, from a theoretical perspective. We can usually achieve the same effect by passing class objects through manager functions (sometimes known as “helper” functions), much as we can achieve the goals of decorators by passing functions and instances through manager code. Just like decorators, though, metaclasses:
Provide a more formal and explicit structure
Help ensure that application programmers won’t forget to augment their classes according to an API’s requirements
Avoid code redundancy and its associated maintenance costs by factoring class customization logic into a single location, the metaclass
To illustrate, suppose we want to automatically insert a method into a set of classes. Of course, we could do this with simple inheritance, if the subject method is known when we code the classes. In that case, we can simply code the method in a superclass and have all the classes in question inherit from it:
class Extras: def extra(self, args): # Normal inheritance: too static ... class Client1(Extras): ... # Clients inherit extra methods class Client2(Extras): ... class Client3(Extras): ... X = Client1() # Make an instance X.extra() # Run the extra methods
Sometimes, though, it’s impossible to predict such
augmentation when classes are coded. Consider the case where classes
are augmented in response to choices made in a user interface at
runtime, or to specifications typed in a configuration file.
Although we could code every class in our imaginary set to
manually check these, too, it’s a lot to ask of
clients (required
is abstract
here—it’s something to be filled in):
def extra(self, arg): ...
class Client1: ... # Client augments: too distributed
if required():
Client1.extra = extra
class Client2: ...
if required():
Client2.extra = extra
class Client3: ...
if required():
Client3.extra = extra
X = Client1()
X.extra()
We can add methods to a class after the class
statement like this because a class
method is just a function that is associated with a class and has a
first argument to receive the self
instance. Although this works, it
puts all the burden of augmentation on client classes (and assumes
they’ll remember to do this at all!).
It would be better from a maintenance perspective to isolate the choice logic in a single place. We might encapsulate some of this extra work by routing classes though a manager function—such a manager function would extend the class as required and handle all the work of runtime testing and configuration:
def extra(self, arg): ...
def extras(Class): # Manager function: too manual
if required():
Class.extra = extra
class Client1: ...
extras(Client1)
class Client2: ...
extras(Client2)
class Client3: ...
extras(Client3)
X = Client1()
X.extra()
This code runs the class through a manager function
immediately after it is created. Although manager functions like
this one can achieve our goal here, they still put a fairly heavy
burden on class coders, who must understand the requirements and
adhere to them in their code. It would be better if there were a
simple way to enforce the augmentation in the subject classes, so
that they don’t need to deal with and can’t forget to use the
augmentation. In other words, we’d like to be able to insert some
code to run automatically at the end of a
class
statement, to augment the
class.
This is exactly what metaclasses do—by declaring a metaclass, we tell Python to route the creation of the class object to another class we provide:
def extra(self, arg): ... class Extras(type): def __init__(Class, classname, superclasses, attributedict): if required(): Class.extra = extra class Client1(metaclass=Extras): ... # Metaclass declaration only class Client2(metaclass=Extras): ... # Client class is instance of meta class Client3(metaclass=Extras): ... X = Client1() # X is instance of Client1 X.extra()
Because Python invokes the metaclass automatically at the end
of the class
statement when the
new class is created, it can augment, register, or otherwise manage
the class as needed. Moreover, the only requirement for the client
classes is that they declare the metaclass; every class that does so
will automatically acquire whatever augmentation the metaclass
provides, both now and in the future if the metaclass changes.
Although it may be difficult to see in this small example,
metaclasses generally handle such tasks better than other approaches.
Having said that, it’s also interesting to note that the class decorators described in the preceding chapter sometimes overlap with metaclasses in terms of functionality. Although they are typically used for managing or augmenting instances, class decorators can also augment classes, independent of any created instances.
For example, suppose we coded our manager function to return the augmented class, instead of simply modifying it in-place. This would allow a greater degree of flexibility, because the manager would be free to return any type of object that implements the class’s expected interface:
def extra(self, arg): ... def extras(Class): if required(): Class.extra = extra return Class class Client1: ... Client1 = extras(Client1) class Client2: ... Client2 = extras(Client2) class Client3: ... Client3 = extras(Client3) X = Client1() X.extra()
If you think this is starting to look reminiscent of class decorators, you’re right. In the prior chapter we presented class decorators as a tool for augmenting instance creation calls. Because they work by automatically rebinding a class name to the result of a function, though, there’s no reason that we can’t use them to augment the class before any instances are ever created. That is, class decorators can apply extra logic to classes, not just instances, at creation time:
def extra(self, arg): ... def extras(Class): if required(): Class.extra = extra return Class @extras class Client1: ... # Client1 = extras(Client1) @extras class Client2: ... # Rebinds class independent of instances @extras class Client3: ... X = Client1() # Makes instance of augmented class X.extra() # X is instance of original Client1
Decorators essentially automate the prior example’s manual name rebinding here. Just like with metaclasses, because the decorator returns the original class, instances are made from it, not from a wrapper object. In fact, instance creation is not intercepted at all.
In this specific case—adding methods to a class when it’s created—the choice between metaclasses and decorators is somewhat arbitrary. Decorators can be used to manage both instances and classes, and they intersect with metaclasses in the second of these roles.
However, this really addresses only one operational mode of
metaclasses. As we’ll see, decorators correspond to metaclass
__init__
methods in this role,
but metaclasses have additional customization hooks. As we’ll also
see, in addition to class initialization, metaclasses can perform
arbitrary construction tasks that might be more difficult with
decorators.
Moreover, although decorators can manage both instances and classes, the converse is not as direct—metaclasses are designed to manage classes, and applying them to managing instances is less straightforward. We’ll explore this difference in code later in this chapter.
Much of this section’s code has been abstract, but we’ll flesh it out into a real working example later in this chapter. To fully understand how metaclasses work, though, we first need to get a clearer picture of their underlying model.
To really understand how metaclasses do their work, you need to
understand a bit more about Python’s type model and what happens at
the end of a class
statement.
So far in this book, we’ve done most of our work by making instances of built-in types like lists and strings, as well as instances of classes we code ourselves. As we’ve seen, instances of classes have some state information attributes of their own, but they also inherit behavioral attributes from the classes from which they are made. The same holds true for built-in types; list instances, for example, have values of their own, but they inherit methods from the list type.
While we can get a lot done with such instance objects, Python’s type model turns out to be a bit richer than I’ve formally described. Really, there’s a hole in the model we’ve seen thus far: if instances are created from classes, what is it that creates our classes? It turns out that classes are instances of something, too:
We explored the notion of types in Chapter 9 and the relationship of classes to types in Chapter 31, but let’s review the basics here so we can see how they apply to metaclasses.
Recall that the type
built-in returns the type of any object (which is itself an object).
For built-in types like lists, the type of the instance is the
built-in list type, but the type of the list type is the type
type
itself—the type
object at the top of the hierarchy
creates specific types, and specific types create instances. You can
see this for yourself at the interactive prompt. In Python 3.0, for
example:
C:misc>c:python30python
>>>type([])
# In 3.0 list is instance of list type <class 'list'> >>>type(type([]))
# Type of list is type class <class 'type'> >>>type(list)
# Same, but with type names <class 'type'> >>>type(type)
# Type of type is type: top of hierarchy <class 'type'>
As we learned when studying new-style class changes in Chapter 31, the same is generally true in
Python 2.6 (and older), but types are not quite the same as
classes—type
is a unique kind of
built-in object that caps the type hierarchy and is used to
construct types:
C:misc>c:python26python
>>>type([])
# In 2.6, type is a bit different <type 'list'> >>>type(type([]))
<type 'type'> >>>type(list)
<type 'type'> >>>type(type)
<type 'type'>
It turns out that the type/instance relationship holds true
for classes as well: instances are created from classes, and classes
are created from type
. In Python
3.0, though, the notion of a “type” is merged with the notion of a
“class.” In fact, the two are essentially synonyms—classes
are types, and types are classes. That is:
Types are defined by classes that derive from type
.
User-defined classes are instances of type classes.
User-defined classes are types that generate instances of their own.
As we saw earlier, this equivalence effects code that tests the type of instances: the type of an instance is the class from which it was generated. It also has implications for the way that classes are created that turn out to be the key to this chapter’s subject. Because classes are normally created from a root type class by default, most programmers don’t need to think about this type/class equivalence. However, it opens up new possibilities for customizing both classes and their instances.
For example, classes in 3.0 (and new-style classes in 2.6) are
instances of the type
class, and
instance objects are instances of their classes; in fact, classes
now have a __class__
that links
to type
, just as an instance has
a __class__
that links to the
class from which it was made:
C:misc>c:python30python
>>>class C: pass
# 3.0 class object (new-style) ... >>>X = C()
# Class instance object >>>type(X)
# Instance is instance of class <class '__main__.C'> >>>X.__class__
# Instance's class <class '__main__.C'> >>>type(C)
# Class is instance of type <class 'type'> >>>C.__class__
# Class's class is type <class 'type'>
Notice especially the last two lines here—classes are
instances of the type
class, just
as normal instances are instances of a class. This works the same
for both built-ins and user-defined class types in 3.0. In fact,
classes are not really a separate concept at all: they are simply
user-defined types, and type
itself is defined by a class.
In Python 2.6, things work similarly for new-style classes
derived from object
, because this
enables 3.0 class behavior:
C:misc>c:python26python
>>>class C(object): pass
# In 2.6 new-style classes, ... # classes have a class too >>>X = C()
>>>type(X)
<class '__main__.C'> >>>type(C)
<type 'type'> >>>X.__class__
<class '__main__.C'> >>>C.__class__
<type 'type'>
Classic classes in 2.6 are a bit different, though—because
they reflect the class model in older Python releases, they do not
have a __class__
link, and like
built-in types in 2.6 they are instances of type
, not a type class:
C:misc>c:python26python
>>>class C: pass
# In 2.6 classic classes, ... # classes have no class themselves >>>X = C()
>>>type(X)
<type 'instance'> >>>type(C)
<type 'classobj'> >>>X.__class__
<class __main__.C at 0x005F85A0> >>>C.__class__
AttributeError: class C has no attribute '__class__'
So why do we care that classes are instances of a
type
class in 3.0? It turns out
that this is the hook that allows us to code metaclasses. Because
the notion of type is the same as
class today, we can subclass type
with normal object-oriented
techniques and class syntax to customize it. And because classes are
really instances of the type
class, creating classes from customized subclasses
of type
allows us to implement
custom kinds of classes. In full detail, this all works out quite
naturally—in 3.0, and in 2.6 new-style classes:
type
is a class that
generates user-defined classes.
Metaclasses are subclasses of the type
class.
Class objects are instances of the type
class, or a subclass
thereof.
Instance objects are generated from a class.
In other words, to control the way classes are created and
augment their behavior, all we need to do is specify that a
user-defined class be created from a user-defined metaclass instead
of the normal type
class.
Notice that this type instance
relationship is not quite the same as
inheritance: user-defined classes may also have
superclasses from which they and their instances inherit attributes
(inheritance superclasses are listed in parentheses in the class
statement and show up in a class’s
__bases__
tuple). The type from
which a class is created, and of which it is an instance, is a
different relationship. The next section describes the procedure
Python follows to implement this instance-of type
relationship.
Subclassing the type
class to
customize it is really only half of the magic behind metaclasses. We
still need to somehow route a class’s creation to the metaclass,
instead of the default type
. To
fully understand how this is arranged, we also need to know how
class statements do their business.
We’ve already learned that when Python reaches a class
statement, it
runs its nested block of code to create its attributes—all the names
assigned at the top level of the nested code block generate
attributes in the resulting class object. These names are usually
method functions created by nested def
s, but they can also be arbitrary
attributes assigned to create
class data shared by all instances.
Technically speaking, Python follows a standard protocol to
make this happen: at the end of a
class
statement, and
after running all its nested code in a namespace dictionary, it
calls the type
object to create
the class
object:
class = type(classname, superclasses, attributedict)
The type
object in turn
defines a __call__
operator
overloading method that runs two other methods when the type
object is called:
type.__new__(typeclass, classname, superclasses, attributedict) type.__init__(class, classname, superclasses, attributedict)
The __new__
method creates
and returns the new class
object,
and then the __init__
method
initializes the newly created object. As we’ll see in a moment,
these are the hooks that metaclass subclasses of type
generally use to customize
classes.
For example, given a class definition like the following:
class Spam(Eggs): # Inherits from Eggs data = 1 # Class data attribute def meth(self, arg): # Class method attribute pass
Python will internally run the nested code block to create two
attributes of the class (data
and
meth
), and then call the type
object to generate the class
object at the end of the class
statement:
Spam = type('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})
Because this call is made at the end of the class
statement, it’s an ideal hook for
augmenting or otherwise processing a class. The trick lies in
replacing type
with a custom
subclass that will intercept this call. The next section shows
how.
As we’ve just seen, classes are created by the type
class by default. To tell Python to
create a class with a custom metaclass instead, you simply need to
declare a metaclass to intercept the normal class creation call. How
you do so depends on which Python version you are using. In Python
3.0, list the desired metaclass as a keyword argument in the class
header:
class Spam(metaclass=Meta): # 3.0 and later
Inheritance superclasses can be listed in the header as well,
before the metaclass. In the following, for example, the new class
Spam
inherits from Eggs
but is also an instance of and is
created by Meta
:
class Spam(Eggs, metaclass=Meta): # Other supers okay
We can get the same effect in Python 2.6, but we must specify
the metaclass differently—using
a class attribute instead of a keyword argument. The object
derivation is required to make this a
new-style class, and this form no longer works in 3.0 as the attribute
is simply ignored:
class spam(object): # 2.6 version (only)
__metaclass__ = Meta
In 2.6, a module-global __metaclass__
variable is also available to link all classes in the module to a
metaclass. This is no longer supported in 3.0, as it was intended as a
temporary measure to make it easier to default to new-style classes
without deriving every class from object
.
When declared in these ways, the call to create the class
object run at the end of the class
statement is modified to invoke the
metaclass instead of the type
default:
class = Meta(classname, superclasses, attributedict)
And because the metaclass is a subclass of type
, the type
class’s __call__
delegates the calls to create and
initialize the new class
object to
the metaclass, if it defines custom versions of these methods:
Meta.__new__(Meta, classname, superclasses, attributedict) Meta.__init__(class, classname, superclasses, attributedict)
To demonstrate, here’s the prior section’s example again, augmented with a 3.0 metaclass specification:
class Spam(Eggs, metaclass=Meta): # Inherits from Eggs, instance of Meta data = 1 # Class data attribute def meth(self, arg): # Class method attribute pass
At the end of this class
statement, Python internally runs the following to create the class
object:
Spam = Meta('Spam', (Eggs,), {'data': 1, 'meth': meth, '__module__': '__main__'})
If the metaclass defines its own versions of __new__
or __init__
, they will be invoked in turn
during this call by the inherited type
class’s __call__
method, to create and initialize
the new class. The next section shows how we might go about coding
this final piece of the metaclass puzzle.
So far, we’ve seen how Python routes class creation calls to a
metaclass, if one is provided. How, though, do we actually code a
metaclass that customizes type
?
It turns out that you already know most of the story—metaclasses
are coded with normal Python class
statements and semantics. Their only substantial distinctions are that
Python calls them automatically at the end of a class
statement, and that they must adhere
to the interface expected by the type
superclass.
Perhaps the simplest metaclass you can code is simply a
subclass of type
with a __new__
method that creates the class
object by running the default version in type
. A metaclass __new__
like this is run by the __call__
method inherited from type
; it typically performs whatever
customization is required and calls the type
superclass’s __new__
method to create and return the
new class object:
class Meta(type):
def __new__(meta, classname, supers, classdict):
# Run by inherited type.__call__
return type.__new__(meta, classname, supers, classdict)
This metaclass doesn’t really do anything (we might as well
let the default type
class create
the class), but it demonstrates the way a metaclass taps into the
metaclass hook to customize—because the metaclass is called at the
end of a class
statement, and
because the type
object’s
__call__
dispatches to the
__new__
and __init__
methods, code we provide in these
methods can manage all the classes created from the
metaclass.
Here’s our example in action again, with prints added to the metaclass and the file at large to trace:
class MetaOne(type): def __new__(meta, classname, supers, classdict): print('In MetaOne.new:', classname, supers, classdict, sep=' ...') return type.__new__(meta, classname, supers, classdict) class Eggs: pass print('making class') class Spam(Eggs, metaclass=MetaOne): # Inherits from Eggs, instance of Meta data = 1 # Class data attribute def meth(self, arg): # Class method attribute pass print('making instance') X = Spam() print('data:', X.data)
Here, Spam
inherits from
Eggs
and is an instance of
MetaOne
, but X
is an instance of and inherits from
Spam
. When this code is run with
Python 3.0, notice how the metaclass is invoked at the
end of the class
statement, before we ever make an
instance—metaclasses are for
processing classes, and classes are for processing instances:
making class In MetaOne.new: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02AEBA08>} making instance data: 1
Metaclasses can also tap into the __init__
protocol invoked by the type
object’s __call__
: in general,
__new__
creates and returns the
class object, and __init__
initializes the already created class. Metaclasses can use both
hooks to manage the class at creation time:
class MetaOne(type): def __new__(meta, classname, supers, classdict): print('In MetaOne.new: ', classname, supers, classdict, sep=' ...') return type.__new__(meta, classname, supers, classdict) def __init__(Class, classname, supers, classdict): print('In MetaOne init:', classname, supers, classdict, sep=' ...') print('...init class object:', list(Class.__dict__.keys())) class Eggs: pass print('making class') class Spam(Eggs, metaclass=MetaOne): # Inherits from Eggs, instance of Meta data = 1 # Class data attribute def meth(self, arg): # Class method attribute pass print('making instance') X = Spam() print('data:', X.data)
In this case, the class initialization method is run after the
class construction method, but both run at the end of the class
statement before any instances are
made (conversely, an __init__
in Spam would run
at instance creation time, and is not affected or run by the
metaclass’s __init__
):
making class In MetaOne.new: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02AAB810>} In MetaOne init: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02AAB810>} ...init class object: ['__module__', 'data', 'meth', '__doc__'] making instance data: 1
Although redefining the type
superclass’s __new__
and __init__
methods is the most common way
metaclasses insert logic into the class object creation process,
other schemes are possible, too.
For example, metaclasses need not really be classes at
all. As we’ve learned, the class
statement issues a simple call to
create a class at the conclusion of its processing. Because of
this, any callable object can in principle be
used as a metaclass, provided it accepts the arguments passed and
returns an object compatible with the intended class. In fact, a
simple object factory function will serve just as well as a
class:
# A simple function can serve as a metaclass too def MetaFunc(classname, supers, classdict): print('In MetaFunc: ', classname, supers, classdict, sep=' ...') return type(classname, supers, classdict) class Eggs: pass print('making class') class Spam(Eggs, metaclass=MetaFunc): # Run simple function at end data = 1 # Function returns class def meth(self, args): pass print('making instance') X = Spam() print('data:', X.data)
When run, the function is called at the end of the declaring
class
statement, and it returns
the expected new class object. The function is simply catching the
call that the type
object’s
__call__
normally intercepts by
default:
making class In MetaFunc: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02B8B6A8>} making instance data: 1
Since they participate in normal OOP mechanics, it’s also
possible for metaclasses to catch the creation
call at the end of a class
statement directly, by redefining
the type
object’s __call__
. The required protocol is a bit
involved, though:
# __call__ can be redefined, metas can have metas
class SuperMeta(type):
def __call__(meta, classname, supers, classdict):
print('In SuperMeta.call: ', classname, supers, classdict, sep='
...')
return type.__call__(meta, classname, supers, classdict)
class SubMeta(type, metaclass=SuperMeta):
def __new__(meta, classname, supers, classdict):
print('In SubMeta.new: ', classname, supers, classdict, sep='
...')
return type.__new__(meta, classname, supers, classdict)
def __init__(Class, classname, supers, classdict):
print('In SubMeta init:', classname, supers, classdict, sep='
...')
print('...init class object:', list(Class.__dict__.keys()))
class Eggs:
pass
print('making class')
class Spam(Eggs, metaclass=SubMeta):
data = 1
def meth(self, arg):
pass
print('making instance')
X = Spam()
print('data:', X.data)
When this code is run, all three redefined methods run in
turn. This is essentially what the type
object does by default:
making class In SuperMeta.call: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02B7BA98>} In SubMeta.new: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02B7BA98>} In SubMeta init: ...Spam ...(<class '__main__.Eggs'>,) ...{'__module__': '__main__', 'data': 1, 'meth': <function meth at 0x02B7BA98>} ...init class object: ['__module__', 'data', 'meth', '__doc__'] making instance data: 1
The preceding example is complicated by the fact that
metaclasses are used to create class objects, but don’t generate
instances of themselves. Because of this, with metaclasses name
lookup rules are somewhat different than what we are accustomed
to. The __call__
method, for
example, is looked up in the class of an object; for metaclasses,
this means the metaclass of a metaclass.
To use normal inheritance-based name lookup, we can achieve
the same effect with normal classes and instances. The output of
the following is the same as the preceding version, but note that
__new__
and __init__
must have different names here,
or else they will run when the SubMeta
instance is created, not when it
is later called as a metaclass:
class SuperMeta: def __call__(self, classname, supers, classdict): print('In SuperMeta.call: ', classname, supers, classdict, sep=' ...') Class = self.__New__(classname, supers, classdict) self.__Init__(Class, classname, supers, classdict) return Class class SubMeta(SuperMeta): def __New__(self, classname, supers, classdict): print('In SubMeta.new: ', classname, supers, classdict, sep=' ...') return type(classname, supers, classdict) def __Init__(self, Class, classname, supers, classdict): print('In SubMeta init:', classname, supers, classdict, sep=' ...') print('...init class object:', list(Class.__dict__.keys())) class Eggs: pass print('making class') class Spam(Eggs, metaclass=SubMeta()): # Meta is normal class instance data = 1 # Called at end of statement def meth(self, arg): pass print('making instance') X = Spam() print('data:', X.data)
Although these alternative forms work, most metaclasses get
their work done by redefining the type
superclass’s __new__
and __init__
; in practice, this is usually
as much control as is required, and it’s often simpler than other
schemes. However, we’ll see later that a simple function-based
metaclass can often work much like a class decorator, which allows
the metaclasses to manage instances as well as classes.
Because metaclasses are specified in similar ways to inheritance superclasses, they can be a bit confusing at first glance. A few key points should help summarize and clarify the model:
Metaclasses inherit from the type
class. Although they have a special role, metaclasses are coded with class
statements and follow the usual
OOP model in Python. For
example, as subclasses of type
, they can redefine the type
object’s methods, overriding and customizing them as needed.
Metaclasses typically redefine the type
class’s __new__
and __init__
to customize class creation
and initialization, but they can also redefine __call__
if they wish to catch the
end-of-class creation call directly. Although it’s unusual, they
can even be simple functions that return arbitrary objects,
instead of type
subclasses.
Metaclass declarations are
inherited by subclasses. The metaclass=
M
declaration in a user-defined class is
inherited by the class’s subclasses, too,
so the metaclass will run for the construction of each class
that inherits this specification in a superclass chain.
Metaclass attributes are not
inherited by class instances. Metaclass declarations
specify an instance relationship, which is
not the same as inheritance. Because classes are instances of
metaclasses, the behavior defined in a metaclass applies to the
class, but not the class’s later instances. Instances obtain
behavior from their classes and superclasses, but not from any
metaclasses. Technically, instance attribute lookups usually
search only the __dict__
dictionaries of the instance and all its classes; the metaclass
is not included in inheritance lookup.
To illustrate the last two points, consider the following example:
class MetaOne(type): def __new__(meta, classname, supers, classdict): # Redefine type method print('In MetaOne.new:', classname) return type.__new__(meta, classname, supers, classdict) def toast(self): print('toast') class Super(metaclass=MetaOne): # Metaclass inherited by subs too def spam(self): # MetaOne run twice for two classes print('spam') class C(Super): # Superclass: inheritance versus instance def eggs(self): # Classes inherit from superclasses print('eggs') # But not from metclasses X = C() X.eggs() # Inherited from C X.spam() # Inherited from Super X.toast() # Not inherited from metaclass
When this code is run, the metaclass handles construction of both client classes, and instances inherit class attributes but not metaclass attributes:
In MetaOne.new: Super In MetaOne.new: C eggs spam AttributeError: 'C' object has no attribute 'toast'
Although detail matters, it’s important to keep the big picture in mind when dealing with metaclasses. Metaclasses like those we’ve seen here will be run automatically for every class that declares them. Unlike the helper function approaches we saw earlier, such classes will automatically acquire whatever augmentation the metaclass provides. Moreover, changes in such augmentation only need to be coded in one place—the metaclass—which simplifies making modifications as our needs evolve. Like so many tools in Python, metaclasses ease maintenance work by eliminating redundancy. To fully sample their power, though, we need to move on to some larger use-case examples.
In this and the following section, we’re going to study examples of two common use cases for metaclasses: adding methods to a class, and decorating all methods automatically. These are just two of the many metaclass roles, which unfortunately consume the space we have left for this chapter; again, you should consult the Web for more advanced applications. These examples are representative of metaclasses in action, though, and they suffice to illustrate the basics.
Moreover, both give us an opportunity to contrast class decorators and metaclasses—our first example compares metaclass- and decorator-based implementations of class augmentation and instance wrapping, and the second applies a decorator with a metaclass first and then with another decorator. As you’ll see, the two tools are often interchangeable, and even complementary.
Earlier in this chapter, we looked at skeleton code that augmented classes by adding methods to them in various ways. As we saw, simple class-based inheritance suffices if the extra methods are statically known when the class is coded. Composition via object embedding can often achieve the same effect too. For more dynamic scenarios, though, other techniques are sometimes required—helper functions can usually suffice, but metaclasses provide an explicit structure and minimize the maintenance costs of changes in the future.
Let’s put these ideas in action here with working code. Consider the following example of manual class augmentation—it adds two methods to two classes, after they have been created:
# Extend manually - adding new methods to classes
class Client1:
def __init__(self, value):
self.value = value
def spam(self):
return self.value * 2
class Client2:
value = 'ni?'
def eggsfunc(obj):
return obj.value * 4
def hamfunc(obj, value):
return value + 'ham'
Client1.eggs = eggsfunc
Client1.ham = hamfunc
Client2.eggs = eggsfunc
Client2.ham = hamfunc
X = Client1('Ni!')
print(X.spam())
print(X.eggs())
print(X.ham('bacon'))
Y = Client2()
print(Y.eggs())
print(Y.ham('bacon'))
This works because methods can always be assigned to a class
after it’s been created, as long as the methods assigned are
functions with an extra first argument to receive the subject
self
instance—this argument can
be used to access state information accessible from the class
instance, even though the function is defined independently of the
class.
When this code runs, we receive the output of a method coded inside the first class, as well as the two methods added to the classes after the fact:
Ni!Ni! Ni!Ni!Ni!Ni! baconham ni?ni?ni?ni? baconham
This scheme works well in isolated cases and can be used to fill out a class arbitrarily at runtime. It suffers from a potentially major downside, though: we have to repeat the augmentation code for every class that needs these methods. In our case, it wasn’t too onerous to add the two methods to both classes, but in more complex scenarios this approach can be time-consuming and error-prone. If we ever forget to do this consistently, or we ever need to change the augmentation, we can run into problems.
Although manual augmentation works, in larger programs it would be better if we could apply such changes to an entire set of classes automatically. That way, we’d avoid the chance of the augmentation being botched for any given class. Moreover, coding the augmentation in a single location better supports future changes—all classes in the set will pick up changes automatically.
One way to meet this goal is to use metaclasses. If we code the augmentation in a metaclass, every class that declares that metaclass will be augmented uniformly and correctly and will automatically pick up any changes made in the future. The following code demonstrates:
# Extend with a metaclass - supports future changes better
def eggsfunc(obj):
return obj.value * 4
def hamfunc(obj, value):
return value + 'ham'
class Extender(type):
def __new__(meta, classname, supers, classdict):
classdict['eggs'] = eggsfunc
classdict['ham'] = hamfunc
return type.__new__(meta, classname, supers, classdict)
class Client1(metaclass=Extender):
def __init__(self, value):
self.value = value
def spam(self):
return self.value * 2
class Client2(metaclass=Extender):
value = 'ni?'
X = Client1('Ni!')
print(X.spam())
print(X.eggs())
print(X.ham('bacon'))
Y = Client2()
print(Y.eggs())
print(Y.ham('bacon'))
This time, both of the client classes are extended with the new methods because they are instances of a metaclass that performs the augmentation. When run, this version’s output is the same as before—we haven’t changed what the code does, we’ve just refactored it to encapsulate the augmentation more cleanly:
Ni!Ni! Ni!Ni!Ni!Ni! baconham ni?ni?ni?ni? baconham
Notice that the metaclass in this example still performs a fairly static task: adding two known methods to every class that declares it. In fact, if all we need to do is always add the same two methods to a set of classes, we might as well code them in a normal superclass and inherit in subclasses. In practice, though, the metaclass structure supports much more dynamic behavior. For instance, the subject class might also be configured based upon arbitrary logic at runtime:
# Can also configure class based on runtime tests
class MetaExtend(type):
def __new__(meta, classname, supers, classdict):
if sometest():
classdict['eggs'] = eggsfunc1
else:
classdict['eggs'] = eggsfunc2
if someothertest():
classdict['ham'] = hamfunc
else:
classdict['ham'] = lambda *args: 'Not supported'
return type.__new__(meta, classname, supers, classdict)
Just in case this chapter has not yet managed to make your head explode, keep in mind again that the prior chapter’s class decorators often overlap with this chapter’s metaclasses in terms of functionality. This derives from the fact that:
Class decorators rebind class names to the result of a
function at the end of a class
statement.
Metaclasses work by routing class object creation through
an object at the end of a class
statement.
Although these are slightly different models, in practice they can usually achieve the same goals, albeit in different ways. In fact, class decorators can be used to manage both instances of a class and the class itself. While decorators can manage classes naturally, though, it’s somewhat less straightforward for metaclasses to manage instances. Metaclasses are probably best used for class object management.
For example, the prior section’s metaclass example,
which adds methods to a class on creation, can also be coded as a
class decorator; in this mode, decorators roughly correspond to
the __init__
method of
metaclasses, since the class object has already been created by
the time the decorator is invoked. Also like with metaclasses, the
original class type is retained, since no wrapper object layer is
inserted. The output of the following is the same as that of the
prior metaclass code:
# Extend with a decorator: same as providing __init__ in a metaclass def eggsfunc(obj): return obj.value * 4 def hamfunc(obj, value): return value + 'ham' def Extender(aClass): aClass.eggs = eggsfunc # Manages class, not instance aClass.ham = hamfunc # Equiv to metaclass __init__ return aClass @Extender class Client1: # Client1 = Extender(Client1) def __init__(self, value): # Rebound at end of class stmt self.value = value def spam(self): return self.value * 2 @Extender class Client2: value = 'ni?' X = Client1('Ni!') # X is a Client1 instance print(X.spam()) print(X.eggs()) print(X.ham('bacon')) Y = Client2() print(Y.eggs()) print(Y.ham('bacon'))
In other words, at least in certain cases, decorators can manage classes as easily as metaclasses. The converse isn’t quite so straightforward, though; metaclasses can be used to manage instances, but only with a certain amount of magic. The next section demonstrates.
As we’ve just seen, class decorators can often serve the same class-management role as metaclasses. Metaclasses can often serve the same instance-management role as decorators, too, but this is a bit more complex. That is:
Class decorators can manage both classes and instances.
Metaclasses can manage both classes and instances, but instances take extra work.
That said, certain applications may be better coded in one or the other. For example, consider the following class decorator example from the prior chapter; it’s used to print a trace message whenever any normally named attribute of a class instance is fetched:
# Class decorator to trace external instance attribute fetches def Tracer(aClass): # On @ decorator class Wrapper: def __init__(self, *args, **kargs): # On instance creation self.wrapped = aClass(*args, **kargs) # Use enclosing scope name def __getattr__(self, attrname): print('Trace:', attrname) # Catches all but .wrapped return getattr(self.wrapped, attrname) # Delegate to wrapped object return Wrapper @Tracer class Person: # Person = Tracer(Person) def __init__(self, name, hours, rate): # Wrapper remembers Person self.name = name self.hours = hours self.rate = rate # In-method fetch not traced def pay(self): return self.hours * self.rate bob = Person('Bob', 40, 50) # bob is really a Wrapper print(bob.name) # Wrapper embeds a Person print(bob.pay()) # Triggers __getattr__
When this code is run, the decorator uses class name rebinding to wrap instance objects in an object that produces the trace lines in the following output:
Trace: name Bob Trace: pay 2000
Although it’s possible for a metaclass to achieve the same effect, it seems less straightforward conceptually. Metaclasses are designed explicitly to manage class object creation, and they have an interface tailored for this purpose. To use a metaclass to manage instances, we have to rely on a bit more magic. The following metaclass has the same effect and output as the prior decorator:
# Manage instances like the prior example, but with a metaclass def Tracer(classname, supers, classdict): # On class creation call aClass = type(classname, supers, classdict) # Make client class class Wrapper: def __init__(self, *args, **kargs): # On instance creation self.wrapped = aClass(*args, **kargs) def __getattr__(self, attrname): print('Trace:', attrname) # Catches all but .wrapped return getattr(self.wrapped, attrname) # Delegate to wrapped object return Wrapper class Person(metaclass=Tracer): # Make Person with Tracer def __init__(self, name, hours, rate): # Wrapper remembers Person self.name = name self.hours = hours self.rate = rate # In-method fetch not traced def pay(self): return self.hours * self.rate bob = Person('Bob', 40, 50) # bob is really a Wrapper print(bob.name) # Wrapper embeds a Person print(bob.pay()) # Triggers __getattr__
This works, but it relies on two tricks. First, it must use
a simple function instead of a class, because type
subclasses must adhere to object
creation protocols. Second, it must manually create the subject
class by calling type
manually;
it needs to return an instance wrapper, but metaclasses are also
responsible for creating and returning the subject class. Really,
we’re using the metaclass protocol to imitate decorators in this
example, rather than vice versa; because both run at the
conclusion of a class
statement, in many roles they are just variations on a theme. This
metaclass version produces the same output as the decorator when
run live:
Trace: name Bob Trace: pay 2000
You should study both versions of these examples for yourself to weigh their tradeoffs. In general, though, metaclasses are probably best suited to class management, due to their design; class decorators can manage either instances or classes, though they may not be the best option for more advanced metaclass roles that we don’t have space to cover in this book (if you want to learn more about decorators and metaclasses after reading this chapter, search the Web or Python’s standard manuals). The next section concludes this chapter with one more common use case—applying operations to a class’s methods automatically.
As we saw in the prior section, because they are both run
at the end of a class
statement,
metaclasses and decorators can often be used
interchangeably, albeit with different syntax.
The choice between the two is arbitrary in many contexts. It’s also
possible to use them in combination, as
complementary tools. In this section, we’ll explore an example of just
such a combination—applying a function decorator to all the methods of
a class.
In the prior chapter we coded two function decorators, one that traced and counted all calls made to a decorated function and another that timed such calls. They took various forms there, some of which were applicable to both functions and methods and some of which were not. The following collects both decorators’ final forms into a module file for reuse and reference here:
# File mytools.py: assorted decorator tools def tracer(func): # Use function, not class with __call__ calls = 0 # Else self is decorator instance only def onCall(*args, **kwargs): nonlocal calls calls += 1 print('call %s to %s' % (calls, func.__name__)) return func(*args, **kwargs) return onCall import time def timer(label='', trace=True): # On decorator args: retain args def onDecorator(func): # On @: retain decorated func def onCall(*args, **kargs): # On calls: call original start = time.clock() # State is scopes + func attr result = func(*args, **kargs) elapsed = time.clock() - start onCall.alltime += elapsed if trace: format = '%s%s: %.5f, %.5f' values = (label, func.__name__, elapsed, onCall.alltime) print(format % values) return result onCall.alltime = 0 return onCall return onDecorator
As we learned in the prior chapter, to use these decorators
manually, we simply import them from the module and code the
decoration @
syntax before each
method we wish to trace or time:
from mytools import tracer class Person: @tracer def __init__(self, name, pay): self.name = name self.pay = pay @tracer def giveRaise(self, percent): # giveRaise = tracer(giverRaise) self.pay *= (1.0 + percent) # onCall remembers giveRaise @tracer def lastName(self): # lastName = tracer(lastName) return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) # Runs onCall(sue, .10) print(sue.pay) print(bob.lastName(), sue.lastName()) # Runs onCall(bob), remembers lastName
When this code is run, we get the following output—calls to decorated methods are routed to logic that intercepts and then delegates the call, because the original method names have been bound to the decorator:
call 1 to __init__ call 2 to __init__ Bob Smith Sue Jones call 1 to giveRaise 110000.0 call 1 to lastName call 2 to lastName Smith Jones
The manual decoration scheme of the prior section works, but it requires us to add decoration syntax before each method we wish to trace and to later remove that syntax when we no longer desire tracing. If we want to trace every method of a class, this can become tedious in larger programs. It would be better if we could somehow apply the tracer decorator to all of a class’s methods automatically.
With metaclasses, we can do exactly that—because they are run when a class is constructed, they are a natural place to add decoration wrappers to a class’s methods. By scanning the class’s attribute dictionary and testing for function objects there, we can automatically run methods through the decorator and rebind the original names to the results. The effect is the same as the automatic method name rebinding of decorators, but we can apply it more globally:
# Metaclass that adds tracing decorator to every method of a client class from types import FunctionType from mytools import tracer class MetaTrace(type): def __new__(meta, classname, supers, classdict): for attr, attrval in classdict.items(): if type(attrval) is FunctionType: # Method? classdict[attr] = tracer(attrval) # Decorate it return type.__new__(meta, classname, supers, classdict) # Make class class Person(metaclass=MetaTrace): def __init__(self, name, pay): self.name = name self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print(sue.pay) print(bob.lastName(), sue.lastName())
When this code is run, the results are the same as before—calls to methods are routed to the tracing decorator first for tracing, and then propagated on to the original method:
call 1 to __init__ call 2 to __init__ Bob Smith Sue Jones call 1 to giveRaise 110000.0 call 1 to lastName call 2 to lastName Smith Jones
The result you see here is a combination of decorator and metaclass work—the metaclass automatically applies the function decorator to every method at class creation time, and the function decorator automatically intercepts method calls in order to print the trace messages in this output. The combination “just works,” thanks to the generality of both tools.
The prior metaclass example works for just one specific function decorator—tracing. However, it’s trivial to generalize this to apply any decorator to all the methods of a class. All we have to do is add an outer scope layer to retain the desired decorator, much like we did for decorators in the prior chapter. The following, for example, codes such a generalization and then uses it to apply the tracer decorator again:
# Metaclass factory: apply any decorator to all methods of a class from types import FunctionType from mytools import tracer, timer def decorateAll(decorator): class MetaDecorate(type): def __new__(meta, classname, supers, classdict): for attr, attrval in classdict.items(): if type(attrval) is FunctionType: classdict[attr] = decorator(attrval) return type.__new__(meta, classname, supers, classdict) return MetaDecorate class Person(metaclass=decorateAll(tracer)): # Apply a decorator to all def __init__(self, name, pay): self.name = name self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print(sue.pay) print(bob.lastName(), sue.lastName())
When this code is run as it is, the output is again the same as that of the previous examples—we’re still ultimately decorating every method in a client class with the tracer function decorator, but we’re doing so in a more generic fashion:
call 1 to __init__ call 2 to __init__ Bob Smith Sue Jones call 1 to giveRaise 110000.0 call 1 to lastName call 2 to lastName Smith Jones
Now, to apply a different decorator to the methods, we can
simply replace the decorator name in the class
header line. To use the timer
function decorator shown earlier, for example, we could use either
of the last two header lines in the following when defining our
class—the first accepts the timer’s default arguments, and the
second specifies label text:
class Person(metaclass=decorateAll(tracer)): # Apply tracer class Person(metaclass=decorateAll(timer())): # Apply timer, defaults class Person(metaclass=decorateAll(timer(label='**'))): # Decorator arguments
Notice that this scheme cannot support nondefault decorator arguments differing per method, but it can pass in decorator arguments that apply to all methods, as done here. To test, use the last of these metaclass declarations to apply the timer, and add the following lines at the end of the script:
# If using timer: total time per method
print('-'*40)
print('%.5f' % Person.__init__.alltime)
print('%.5f' % Person.giveRaise.alltime)
print('%.5f' % Person.lastName.alltime)
The new output is as follows—the metaclass wraps methods in timer decorators now, so we can tell how long each and every call takes, for every method of the class:
**__init__: 0.00001, 0.00001 **__init__: 0.00001, 0.00002 Bob Smith Sue Jones **giveRaise: 0.00001, 0.00001 110000.0 **lastName: 0.00001, 0.00001 **lastName: 0.00001, 0.00002 Smith Jones ---------------------------------------- 0.00002 0.00001 0.00002
Class decorators intersect with metaclasses here, too. The following version replaces the preceding example’s metaclass with a class decorator. It defines and uses a class decorator that applies a function decorator to all methods of a class. Although the prior sentence may sound more like a Zen statement than a technical description, this all works quite naturally—Python’s decorators support arbitrary nesting and combinations:
# Class decorator factory: apply any decorator to all methods of a class from types import FunctionType from mytools import tracer, timer def decorateAll(decorator): def DecoDecorate(aClass): for attr, attrval in aClass.__dict__.items(): if type(attrval) is FunctionType: setattr(aClass, attr, decorator(attrval)) # Not __dict__ return aClass return DecoDecorate @decorateAll(tracer) # Use a class decorator class Person: # Applies func decorator to methods def __init__(self, name, pay): # Person = decorateAll(..)(Person) self.name = name # Person = DecoDecorate(Person) self.pay = pay def giveRaise(self, percent): self.pay *= (1.0 + percent) def lastName(self): return self.name.split()[-1] bob = Person('Bob Smith', 50000) sue = Person('Sue Jones', 100000) print(bob.name, sue.name) sue.giveRaise(.10) print(sue.pay) print(bob.lastName(), sue.lastName())
When this code is run as it is, the class decorator applies the tracer function decorator to every method and produces a trace message on calls (the output is the same as that of the preceding metaclass version of this example):
call 1 to __init__ call 2 to __init__ Bob Smith Sue Jones call 1 to giveRaise 110000.0 call 1 to lastName call 2 to lastName Smith Jones
Notice that the class decorator returns the original,
augmented class, not a wrapper layer for it (as is common when
wrapping instance objects instead). As for the metaclass version, we
retain the type of the original class—an instance of Person
is an instance of Person
, not of some wrapper class. In
fact, this class decorator deals with class creation only; instance
creation calls are not intercepted at all.
This distinction can matter in programs that require type testing for instances to yield the original class, not a wrapper. When augmenting a class instead of an instance, class decorators can retain the original class type. The class’s methods are not their original functions because they are rebound to decorators, but this is less important in practice, and it’s true in the metaclass alternative as well.
Also note that, like the metaclass version, this structure cannot support function decorator arguments that differ per method, but it can handle such arguments if they apply to all methods. To use this scheme to apply the timer decorator, for example, either of the last two decoration lines in the following will suffice if coded just before our class definition—the first uses decorator argument defaults, and the second provides one explicitly:
@decorateAll(tracer) # Decorate all with tracer @decorateAll(timer()) # Decorate all with timer, defaults @decorateAll(timer(label='@@')) # Same but pass a decorator argument
As before, let’s use the last of these decorator lines and add the following at the end of the script to test our example with a different decorator:
# If using timer: total time per method
print('-'*40)
print('%.5f' % Person.__init__.alltime)
print('%.5f' % Person.giveRaise.alltime)
print('%.5f' % Person.lastName.alltime)
The same sort of output appears—for every method we get timing data for each and all calls, but we’ve passed a different label argument to the timer decorator:
@@__init__: 0.00001, 0.00001 @@__init__: 0.00001, 0.00002 Bob Smith Sue Jones @@giveRaise: 0.00001, 0.00001 110000.0 @@lastName: 0.00001, 0.00001 @@lastName: 0.00001, 0.00002 Smith Jones ---------------------------------------- 0.00002 0.00001 0.00002
As you can see, metaclasses and class decorators are not only often interchangeable, but also commonly complementary. Both provide advanced but powerful ways to customize and manage both class and instance objects, because both ultimately allow you to insert code into the class creation process. Although some more advanced applications may be better coded with one or the other, the way you choose or combine these two tools in many cases is largely up to you.
In this chapter, we studied metaclasses and explored examples of them in action. Metaclasses allow us to tap into the class creation protocol of Python, in order to manage or augment user-defined classes. Because they automate this process, they can provide better solutions for API writers then manual code or helper functions; because they encapsulate such code, they can minimize maintenance costs better than some other approaches.
Along the way, we also saw how the roles of class decorators and
metaclasses often intersect: because both run at the conclusion of a
class
statement, they can sometimes
be used interchangeably. Class decorators can be used to manage both
class and instance objects; metaclasses can, too, although they are
more directly targeted toward classes.
Since this chapter covered an advanced topic, we’ll work through just a few quiz questions to review the basics (if you’ve made it this far in a chapter on metaclasses, you probably already deserve extra credit!). Because this is the last part of the book, we’ll forego the end-of-part exercises. Be sure to see the appendixes that follow for pointers on installation steps, and the solutions to the prior parts’ exercises.
Once you finish the quiz, you’ve officially reached the end of this book. Now that you know Python inside and out, your next step, should you choose to take it, is to explore the libraries, techniques, and tools available in the application domains in which you work. Because Python is so widely used, you’ll find ample resources for using it in almost any application you can think of—from GUIs, the Web, and databases to numeric programming, robotics, and system administration.
This is where Python starts to become truly fun, but this is also where this book’s story ends, and others’ begin. For pointers on where to turn after this book, see the list of recommended follow-up texts in the Preface. Good luck with your journey. And of course, “Always look on the bright side of Life!”
What is a metaclass?
How do you declare the metaclass of a class?
How do class decorators overlap with metaclasses for managing classes?
How do class decorators overlap with metaclasses for managing instances?
Would you rather count decorators or metaclasses amongst your weaponry? (And please phrase your answer in terms of a popular Monty Python skit.)
A metaclass is a class used to create a class. Normal
classes are instances of the type
class by default. Metaclasses are
usually subclasses of the type
class, which redefines class creation protocol methods in order to
customize the class creation call issued at the end of a class
statement; they typically redefine
the methods __new__
and
__init__
to tap into the class
creation protocol. Metaclasses can also be coded other ways—as
simple functions, for example—but they are responsible for making
and returning an object for the new class.
In Python 3.0 and later, use a keyword argument in the
class
header line: class
C(metaclass=
M
)
. In Python 2.X, use a class attribute
instead: __metaclass__ =
M
. In 3.0, the class
header line can also name normal
superclasses (a.k.a. base classes) before the metaclass
keyword argument.
Because both are automatically triggered at the end of a
class
statement, class
decorators and metaclasses can both be used to manage classes.
Decorators rebind a class name to a callable’s result and
metaclasses route class creation through a callable, but both
hooks can be used for similar purposes. To manage classes,
decorators simply augment and return the original class objects.
Metaclasses augment a class after they create it.
Because both are automatically triggered at the end of a
class
statement, class
decorators and metaclasses can both be used to manage class
instances, by inserting a wrapper object to catch instance
creation calls. Decorators may rebind the class name to a callable
run on instance creation that retains the original class object.
Metaclasses can do the same, but they must also create the class
object, so their usage is somewhat more complex in this
role.
Our chief weapon is decorators...decorators and metaclasses...metaclasses and decorators.... Our two weapons are metaclasses and decorators...and ruthless efficiency.... Our three weapons are metaclasses, decorators, and ruthless efficiency...and an almost fanatical devotion to Guido.... Our four...no....Amongst our weapons.... Amongst our weaponry...are such elements as metaclasses, decorators.... I’ll come in again....