One of the most common uses of metaclasses is to have classes automatically register themselves as plugins/handlers. Examples of these can be seen in many projects, such as web frameworks. Those codebases are too extensive to usefully explain here though. Hence, we'll show a simpler example showing the power of metaclasses as a self-registering plugin system:
>>> import abc >>> class Plugins(abc.ABCMeta): ... plugins = dict() ... ... def __new__(metaclass, name, bases, namespace): ... cls = abc.ABCMeta.__new__(metaclass, name, bases, ... namespace) ... if isinstance(cls.name, str): ... metaclass.plugins[cls.name] = cls ... return cls ... ... @classmethod ... def get(cls, name): ... return cls.plugins[name] >>> class PluginBase(metaclass=Plugins): ... @property ... @abc.abstractmethod ... def name(self): ... raise NotImplemented() >>> class SpamPlugin(PluginBase): ... name = 'spam' >>> class EggsPlugin(PluginBase): ... name = 'eggs' >>> Plugins.get('spam') <class '...SpamPlugin'> >>> Plugins.plugins {'spam': <class '...SpamPlugin'>, 'eggs': <class '...EggsPlugin'>}
This example is a tad simplistic of course, but it's the basis for many plugin systems. Which is a very important thing to note while implementing systems like these; however, while metaclasses run at definition time, the module still needs to be imported to work. There are several options to do this; loading on-demand through the get method has my vote as that also doesn't add load time if the plugin is not used.
The following examples will use the following file structure to get reproducible results. All files will be contained in a plugins directory.
The __init__.py file is used to create shortcuts, so a simple import plugins will result in having plugins.Plugins available, instead of requiring importing plugins.base explicitly.
# plugins/__init__.py from .base import Plugin from .base import Plugins __all__ = ['Plugin', 'Plugins']
The base.py file containing the Plugins collection and the Plugin base class:
# plugins/base.py
import abc
class Plugins(abc.ABCMeta):
plugins = dict()
def __new__(metaclass, name, bases, namespace):
cls = abc.ABCMeta.__new__(
metaclass, name, bases, namespace)
if isinstance(cls.name, str):
metaclass.plugins[cls.name] = cls
return cls
@classmethod
def get(cls, name):
return cls.plugins[name]
class Plugin(metaclass=Plugins):
@property
@abc.abstractmethod
def name(self):
raise NotImplemented()And two simple plugins, spam.py:
from . import base
class Spam(base.Plugin):
name = 'spam'And eggs.py:
from . import base
class Eggs(base.Plugin):
name = 'eggs'The first of the solutions for the import problem is simply taking care of it in the get method of the Plugins metaclass. Whenever the plugin is not found in the registry, it should automatically load the module from the plugins directory.
The advantages of this approach are that not only the plugins don't explicitly need to be preloaded but also that the plugins are only loaded when the need is there. Unused plugins are not touched, so this method can help in reducing your applications' load times.
The downside is that the code will not be run or tested, so it might be completely broken and you won't know about it until it is finally loaded. Solutions for this problem will be covered in the testing chapter, Chapter 10, Testing and Logging – Preparing for Bugs. The other problem is that if the code self-registers itself into other parts of an application then that code won't be executed either.
Modifying the Plugins.get method, we get the following:
import abc
import importlib
class Plugins(abc.ABCMeta):
plugins = dict()
def __new__(metaclass, name, bases, namespace):
cls = abc.ABCMeta.__new__(
metaclass, name, bases, namespace)
if isinstance(cls.name, str):
metaclass.plugins[cls.name] = cls
return cls
@classmethod
def get(cls, name):
if name not in cls.plugins:
print('Loading plugins from plugins.%s' % name)
importlib.import_module('plugins.%s' % name)
return cls.plugins[name]This results in the following when executing:
>>> import plugins >>> plugins.Plugins.get('spam') Loading plugins from plugins.spam <class 'plugins.spam.Spam'> >>> plugins.Plugins.get('spam') <class 'plugins.spam.Spam'>
As you can see, this approach only results in running import once. The second time, the plugin will be available in the plugins dictionary so no loading will be necessary.
While only loading the needed plugins is generally a better idea, there is something to be said to preload the plugins you will likely need. As explicit is better than implicit, an explicit list of plugins to load is generally a good solution. The added advantages of this method are that firstly you are able to make the registration a bit more advanced as you are guaranteed that it is run and secondly you can load plugins from multiple packages.
Instead of importing in the get method, we will add a load method this time; a load method that imports all the given module names:
import abc
import importlib
class Plugins(abc.ABCMeta):
plugins = dict()
def __new__(metaclass, name, bases, namespace):
cls = abc.ABCMeta.__new__(
metaclass, name, bases, namespace)
if isinstance(cls.name, str):
metaclass.plugins[cls.name] = cls
return cls
@classmethod
def get(cls, name):
return cls.plugins[name]
@classmethod
def load(cls, *plugin_modules):
for plugin_module in plugin_modules:
plugin = importlib.import_module(plugin_module)Which can be called using the following code:
>>> import plugins >>> plugins.Plugins.load( ... 'plugins.spam', ... 'plugins.eggs', ... ) >>> plugins.Plugins.get('spam') <class 'plugins.spam.Spam'>
A fairly simple and straightforward system to load the plugins based on settings, this can easily be combined with any type of settings system to fill the load method.
Whenever possible, it is best to avoid having systems depend on automatic detection of modules on a filesystem as it goes directly against PEP8. Specifically, "explicit is better than implicit". While these systems can work fine in specific cases, they often make debugging much more difficult. Similar automatic import systems in Django have caused me a fair share of headaches as they tend to obfuscate the errors. Having said that, automatic plugin loading based on all the files in a plugins directory is still a possibility warranting a demonstration.
import os
import re
import abc
import importlib
MODULE_NAME_RE = re.compile('[a-z][a-z0-9_]*', re.IGNORECASE)
class Plugins(abc.ABCMeta):
plugins = dict()
def __new__(metaclass, name, bases, namespace):
cls = abc.ABCMeta.__new__(
metaclass, name, bases, namespace)
if isinstance(cls.name, str):
metaclass.plugins[cls.name] = cls
return cls
@classmethod
def get(cls, name):
return cls.plugins[name]
@classmethod
def load_directory(cls, module, directory):
for file_ in os.listdir(directory):
name, ext = os.path.splitext(file_)
full_path = os.path.join(directory, file_)
import_path = [module]
if os.path.isdir(full_path):
import_path.append(file_)
elif ext == '.py' and MODULE_NAME_RE.match(name):
import_path.append(name)
else:
# Ignoring non-matching files/directories
continue
plugin = importlib.import_module('.'.join(import_path))
@classmethod
def load(cls, **plugin_directories):
for module, directory in plugin_directories.items():
cls.load_directory(module, directory)If possible, I would try to avoid using a fully automatic import system as it's very prone to accidental errors and can make debugging more difficult, not to mention that the import order cannot easily be controlled this way. To make this system a bit smarter (even importing packages outside of your Python path), you can create a plugin loader using the abstract base classes in importlib.abc. Note that you will most likely still need to list the directories through os.listdir or os.walk though.