The call to the built-in type() class can be used as a dynamic equivalent of the class statement. It creates a new class object given its name, its base classes, and a mapping containing its attributes:

def method(self):
    return 1

klass = type('MyClass', (object,), {'method': method})

The following is the output:

>>> instance = klass()
>>> instance.method()
1

This is equivalent to the explicit definition of the class:

class MyClass:
    def method(self):
        return 1

Here is what you will get:

>>> instance = MyClass()
>>> instance.method()
1

Every class created with the class statement implicitly uses type as its metaclass. This default behavior can be changed by providing the metaclass keyword argument to the class statement:

class ClassWithAMetaclass(metaclass=type):
    pass

The value provided as a metaclass argument is usually another class object, but it can be any other callable that accepts the same arguments as the type class and is expected to return another class object. The call signature is type(name, bases, namespace), which is explained as follows:

One way of thinking about metaclasses is the __new__() method, but at a higher level of class definition.

Despite the fact that functions that explicitly call type() can be used in place of metaclasses, the usual approach is to use a different class that inherits from type for this purpose. The common template for a metaclass is as follows:

class Metaclass(type):
    def __new__(mcs, name, bases, namespace):
        return super().__new__(mcs, name, bases, namespace)

    @classmethod
    def __prepare__(mcs, name, bases, **kwargs):
        return super().__prepare__(name, bases, **kwargs)

    def __init__(cls, name, bases, namespace, **kwargs):
        super().__init__(name, bases, namespace)

    def __call__(cls, *args, **kwargs):
        return super().__call__(*args, **kwargs)

The name, bases, and namespace arguments have the same meaning as in the type() call explained earlier, but each of these four methods can have different purposes:

Each of the preceding methods can accept additional extra keyword arguments here represented by **kwargs. These arguments can be passed to the metaclass object using extra keyword arguments in the class definition in the form of the following code:

class Klass(metaclass=Metaclass, extra="value"):
    pass

Such amount of information can be overwhelming at the beginning without proper examples, so let's trace the creation of metaclasses, classes, and instances with some print() calls:

class RevealingMeta(type):
    def __new__(mcs, name, bases, namespace, **kwargs):
        print(mcs, "__new__ called")
        return super().__new__(mcs, name, bases, namespace)

    @classmethod
    def __prepare__(mcs, name, bases, **kwargs):
        print(mcs, "__prepare__ called")
        return super().__prepare__(name, bases, **kwargs)

    def __init__(cls, name, bases, namespace, **kwargs):
        print(cls, "__init__ called")
        super().__init__(name, bases, namespace)

    def __call__(cls, *args, **kwargs):
        print(cls, "__call__ called")
        return super().__call__(*args, **kwargs)

Using RevealingMeta as a metaclass to create a new class definition will give the following output in the Python interactive session:

>>> class RevealingClass(metaclass=RevealingMeta):
...     def __new__(cls):
...         print(cls, "__new__ called")
...         return super().__new__(cls)
...     def __init__(self):
...         print(self, "__init__ called")
...         super().__init__()
... 
<class 'RevealingMeta'> __prepare__ called
<class 'RevealingMeta'> __new__ called
<class 'RevealingClass'> __init__ called
>>> instance = RevealingClass()
<class 'RevealingClass'> __call__ called
<class 'RevealingClass'> __new__ called
<RevealingClass object at 0x1032b9fd0> __init__ called

Metaclasses are not a new feature and are available in Python since version 2.2. Anyway, the syntax of this changed significantly and this change is neither backwards nor forwards compatible. While the new syntax is:

class ClassWithAMetaclass(metaclass=type):
    pass

In Python 2, this must be written as follows:

class ClassWithAMetaclass(object):
    __metaclass__ = type

Class statements in Python 2 do not accept keyword arguments, so Python 3 syntax for defining metaclasses will raise the SyntaxError exception on import. It is still possible to write a code using metaclasses that will run on both Python versions, but it requires some extra work. Fortunately, compatibility-related packages such as six provide simple and reusable solutions to this problem:

from six import with_metaclass


class Meta(type):
    pass


class Base(object):
    pass


class MyClass(with_metaclass(Meta, Base)):
    pass

The other important difference is the lack of the __prepare__() hook in Python 2 metaclasses. Implementing such a function will not raise any exceptions under Python 2 but is pointless because it will not be called in order to provide a clean namespace object. This is why packages that need to maintain Python 2 compatibility need to rely on more complex tricks if they want to achieve things that are a lot easier to implement using __prepare__(). For instance, the Django REST Framework (http://www.django-rest-framework.org) uses the following approach to preserve the order in which attributes are added to a class:

class SerializerMetaclass(type):
    @classmethod
    def _get_declared_fields(cls, bases, attrs):
        fields = [(field_name, attrs.pop(field_name))
                  for field_name, obj in list(attrs.items())
                  if isinstance(obj, Field)]
        fields.sort(key=lambda x: x[1]._creation_counter)

        # If this class is subclassing another Serializer, add 
        # that Serializer's fields. 
        # Note that we loop over the bases in *reverse*. 
        # This is necessary in order to maintain the 
        # correct order of fields.
        for base in reversed(bases):
            if hasattr(base, '_declared_fields'):
                fields = list(base._declared_fields.items()) + fields

        return OrderedDict(fields)

    def __new__(cls, name, bases, attrs):
        attrs['_declared_fields'] = cls._get_declared_fields(
            bases, attrs
        )
        return super(SerializerMetaclass, cls).__new__(
            cls, name, bases, attrs
        )

This is the workaround if the default namespace type, which is dict, does not guarantee to preserve the order of the key-value tuples. The _creation_counter attribute is expected to be in every instance of the Field class. This Field.creation_counter attribute is created in the same way as InstanceCountingClass.instance_number that was presented in the section about the __new__() method. This is a rather complex solution that breaks a single responsibility principle by sharing its implementation across two different classes only to ensure a trackable order of attributes. In Python 3, this could be simpler because __prepare__() can return other mapping types such as OrderedDict:

from collections import OrderedDict


class OrderedMeta(type):
    @classmethod
    def __prepare__(cls, name, bases, **kwargs):
        return OrderedDict()

    def __new__(mcs, name, bases, namespace):
        namespace['order_of_attributes'] = list(namespace.keys())
        return super().__new__(mcs, name, bases, namespace)


class ClassWithOrder(metaclass=OrderedMeta):
    first = 8
    second = 2

Here is what you will see:

>>> ClassWithOrderedAttributes.order_of_attributes
['__module__', '__qualname__', 'first', 'second']
>>> ClassWithOrderedAttributes.__dict__.keys()
dict_keys(['__dict__', 'first', '__weakref__', 'second', 'order_of_attributes', '__module__', '__doc__'])

Metaclasses once mastered are a powerful feature but always complicate the code. They might also make the code less robust that is intended to work on any kind of class. For instance, you might encounter bad interactions when slots are used in the class, or when some base class already implements a metaclass, which conflicts with what yours does. They just do not compose well.

For simple things like changing the read/write attributes or adding new ones, metaclasses can be avoided in favor of simpler solutions such as properties, descriptors, or class decorators.

It is also true that often metaclasses can be replaced with other simpler approaches, but there are situations where things cannot be easily done without them. For instance, it is hard to imagine Django's ORM implementation built without extensive use of metaclasses. It could be possible, but it is rather unlikely that the resulting solution would be similarly easy to use. And frameworks are the place where metaclasses are really well-suited. They usually have a lot of complex solutions that are not easy to understand and follow, but eventually allow other programmers to write more condensed and readable code that operates on a higher level of abstraction.

Like some other advanced Python features, metaclasses are very elastic and can be easily abused. While the call signature of the class is rather strict, Python does not enforce the type of the return parameter. It can be anything as long as it accepts incoming arguments on calls and has the required attributes whenever it is needed.

One such object that can be anything-anywhere is the instance of the Mock class provided in the unittest.mock module. Mock is not a metaclass and also does not inherit from the type class. It also does not return the class object on instantiating. Still, it can be included as a metaclass keyword argument in the class definition and this will not raise any issues, despite, it is pointless to do so:

>>> from unittest.mock import Mock
>>> class Nonsense(metaclass=Mock):  # pointless, but illustrative
...     pass
... 
>>> Nonsense
<Mock spec='str' id='4327214664'>

The preceding example, of course, completely does not make sense and will fail on any attempt to instantiate such a Nonsense pseudo-class. It is still important to know that such things are possible because issues with metaclass types that do not result in the creation of the type subclass are sometimes very hard to spot and understand. As a proof, here is a traceback of the exception raised when we try to create a new instance of the Nonsense class presented earlier:

>>> Nonsense()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/mock.py", line 917, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/unittest/mock.py", line 976, in _mock_call
    result = next(effect)
StopIteration

Python provides three built-in functions to manually execute, evaluate, and compile arbitrary Python code:

The exec() and eval() functions are the easiest to start with when trying to dynamically generate code because they can operate on strings. If you already know how to program in Python, then you may know how to correctly generate a working source code programmatically. I hope you do.

The most useful in the context of metaprogramming is obviously exec() because it allows us to execute any sequence of Python statements. And the word any should be alarming for you. Even eval(), which only allows evaluation of expressions in the hands of a skillful programmer (when fed with the user input), can lead to serious security holes. Note that crashing the Python interpreter is the least scary scenario you should be afraid of. Introducing vulnerability to remote execution exploits due to irresponsible use of exec() and eval() can cost you your image as a professional developer, or even your job.

Even if used with a trusted input, there is a long list of little details about exec() and eval() that is too long to be included here, but might affect how your application works in the ways you would not expect. Armin Ronacher has a good article that lists the most important of them called Be careful with exec and eval in Python (refer to http://lucumr.pocoo.org/2011/2/1/exec-in-python/).

Despite all these frightening warnings, there are natural situations where the usage of exec() and eval() is really justified. The popular statement about when you have to use them is: you will know. In other words, in case of even the tiniest doubt, you should not use them and try to find a different solution.

The Python syntax is converted to Abstract Syntax Tree (AST) before it is compiled to byte code. This is a tree representation of the abstract syntactic structure of the source code. The processing of Python grammar is available thanks to the built-in ast module. Raw AST of Python code can be created using the compile() function with the ast.PyCF_ONLY_AST flag, or using the ast.parse() helper. Direct translation in reverse is not that simple and there is no function provided in the built-ins for that. Some projects, such as PyPy, do such things though.

The ast module provides some helper functions that allow working with the AST:

>>> tree = ast.parse('def hello_world(): print("hello world!")')
>>> tree
<_ast.Module object at 0x00000000038E9588>
>>> ast.dump(tree)
"Module(
    body=[
        FunctionDef(
           name='hello_world', 
           args=arguments(
               args=[], 
               vararg=None, 
               kwonlyargs=[], 
               kw_defaults=[], 
               kwarg=None, 
               defaults=[]
           ), 
           body=[
               Expr(
                   value=Call(
                       func=Name(id='print', ctx=Load()), 
                       args=[Str(s='hello world!')], 
                       keywords=[]
                   )
               )
           ], 
           decorator_list=[], 
           returns=None
       )
    ]
)"

The output of ast.dump() in the preceding example was reformatted to increase the readability and better show the tree-like structure of the AST. It is important to know that the AST can be modified before being passed to the compile() call that gives many new possibilities. For instance, new syntax nodes can be used for additional instrumentation such as test coverage measurement. It is also possible to modify the existing code tree in order to add new semantics to the existing syntax. Such a technique is used by the MacroPy project (https://github.com/lihaoyi/macropy) to add syntactic macros to Python using the already existing syntax (refer to Figure 5):

Figure 5: How MacroPy adds syntactic macros to Python modules on import

AST can also be created in a purely artificial manner and there is no need to parse any source at all. This gives Python programmers the ability to create Python bytecode for custom domain-specific languages or even completely implement other existing programming languages on top of Python VM.

It is really hard to find a really usable implementation of the library that relies on code generation patterns that is not only an experiment or simple proof of concepts. The reasons for that situation are fairly obvious:

Despite these difficulties, there are some projects that successfully take this approach either to improve performance or achieve things that would be impossible by other means.

# sample.py
import falcon
import json
 
class QuoteResource:
    def on_get(self, req, resp):
        """Handles GET requests"""
        quote = {
            'quote': 'I've always been more interested in '
                     'the future than in the past.',
            'author': 'Grace Hopper'
        }

        resp.body = json.dumps(quote)
 
api = falcon.API()
api.add_route('/quote', QuoteResource())

>>> api._router._find.__code__
<code object find at 0x00000000033C29C0, file "<string>", line 1>
>>> api.add_route('/none', None)
>>> api._router._find.__code__
<code object find at 0x00000000033C2810, file "<string>", line 1>

;; hyllo.hy
(defn hello [] (print "hello world!"))

>>> import hy
>>> import hyllo
>>> hyllo.hello()
hello world!

>>> import dis
>>> dis.dis(hyllo.hello)
  2           0 LOAD_GLOBAL        0 (print)
              3 LOAD_CONST         1 ('hello world!')
              6 CALL_FUNCTION      1 (1 positional, 0 keyword pair)
              9 RETURN_VALUE
>>> def hello(): print("hello world!")
>>> dis.dis(hello)
  1           0 LOAD_GLOBAL        0 (print)
              3 LOAD_CONST         1 ('hello world!')
              6 CALL_FUNCTION      1 (1 positional, 0 keyword pair)
              9 POP_TOP
             10 LOAD_CONST         0 (None)
             13 RETURN_VALUE