Structural patterns are really important in big applications. They decide how the code is organized and give developers recipes on how to interact with each part of the application.
For a long time, the most well-known implementation of many structural patterns in the Python world provided the Zope project with its Zope Component Architecture (ZCA). It implements most of the patterns described in this section and provides a rich set of tools to work with them. The ZCA is intended to run not only in the Zope framework, but also in other frameworks such as Twisted. It provides an implementation of interfaces and adapters among other things.
Unfortunately (or not), Zope lost almost all of its momentum and is not as popular as it used to be. But its ZCA may still be a good reference on implementing structural patterns in Python. Baiju Muthukadan created A Comprehensive Guide to Zope Component Architecture. It is available both in print and freely online (refer to http://muthukadan.net/docs/zca.html). It was written in 2009, so it does not cover the latest versions of Python but should still be a good read because it provides a lot of rationale for some of the mentioned patterns.
Python already provides some of the popular structural patterns through its syntax. For instance, the class and function decorators can be considered a flavor of the decorator pattern. Also, support for creating and importing modules is an emanation of module pattern.
The list of common structural patterns is actually quite long. The original Design Patterns book featured as many as seven of them and the list was later extended by other literature. We won't discuss all of them but will focus only on the three most popular and recognized ones, which are:
The Adapter pattern allows the interface of an existing class to be used from another interface. In other words, an adapter wraps a class or an object A so that it works in a context intended for a class or an object B.
Creating adapters in Python is actually very straightforward due to how typing in this language works. The typing philosophy in Python is commonly referred to as duck-typing:
"If it walks like a duck and talks like a duck, then it's a duck!"
According to this rule, if a value for a function or method is accepted, the decision should not be based on its type but rather on its interface. So, as long as the object behaves as expected, that is, has proper method signatures and attributes, its type is considered compatible. This is completely different from many statically typed languages where such a thing is rarely available.
In practice, when some code is intended to work with a given class, it is fine to feed it with objects from another class as long as they provide the methods and attributes used by the code. Of course, this assumes that the code isn't calling an instance
to verify that the instance is of a specific class.
The adapter pattern is based on this philosophy and defines a wrapping mechanism where a class or an object is wrapped in order to make it work in a context that was not primarily intended for it. StringIO
is a typical example, as it adapts the str
type, so it can be used as a file
type:
>>> from io import StringIO >>> my_file = StringIO('some content') >>> my_file.read() 'some content' >>> my_file.seek(0) >>> my_f ile.read(1) 's'
Let's take another example. A DublinCoreInfos
class knows how to display the summary of some subset of Dublin Core information (see http://dublincore.org/) for a given document provided as a dict
. It reads a few fields, such as the author's name or the title, and prints them. To be able to display Dublin Core for a file, it has to be adapted in the same way StringIO
does. The following figure shows a UML-like diagram for such a kind of adapter pattern implementation.
DublinCoreAdapter
wraps a file instance and provides metadata access over it:
from os.path import split, splitext class DublinCoreAdapter: def __init__(self, filename): self._filename = filename @property def title(self): return splitext(split(self._filename)[-1])[0] @property def languages(self): return ('en',) def __getitem__(self, item): return getattr(self, item, 'Unknown') class DublinCoreInfo(object): def summary(self, dc_dict): print('Title: %s' % dc_dict['title']) print('Creator: %s' % dc_dict['creator']) print('Languages: %s' % ', '.join(dc_dict['languages']))
And here is the example usage:
>>> adapted = DublinCoreAdapter('example.txt') >>> infos = DublinCoreInfo() >>> infos.summary(adapted) Title: example Creator: Unknown Languages: en
Besides the fact that it allows substitution, the adapter pattern can also change the way developers work. Adapting an object to work in a specific context makes the assumption that the class of the object does not matter at all. What matters is that this class implements what DublinCoreInfo
is waiting for and this behavior is fixed or completed by an adapter. So, the code can, somehow, simply tell whether it is compatible with objects that are implementing a specific behavior. This can be expressed by interfaces.
An interface is a definition of an API. It describes a list of methods and attributes a class should have to implement with the desired behavior. This description does not implement any code but just defines an explicit contract for any class that wishes to implement the interface. Any class can then implement one or several interfaces in whichever way it wants.
While Python prefers duck-typing over explicit interface definitions, it may be better to use them sometimes. For instance, explicit interface definition makes it easier for a framework to define functionalities over interfaces.
The benefit is that classes are loosely coupled, which is considered as a good practice. For example, to perform a given process, a class A
does not depend on a class B
, but rather on an interface I
. Class B
implements I
, but it could be any other class.
The support for such a technique is built-in in many statically typed languages such as Java or Go. The interfaces allow the functions or methods to limit the range of acceptable parameter objects that implement a given interface, no matter what kind of class it comes from. This allows for more flexibility than restricting arguments to given types or their subclasses. It is like an explicit version of duck-typing behavior: Java uses interfaces to verify a type safety at compile time rather than use duck-typing to tie things together at run time.
Python has a completely different typing philosophy to Java, so it does not have native support for interfaces. Anyway, if you would like to have more explicit control on application interfaces, there are generally two solutions to choose from:
There are a few frameworks that allow you to build explicit interfaces in Python. The most notable one is a part of the Zope project. It is the zope.interface
package. Although, nowadays, Zope is not as popular as it used to be, the zope.interface
package is still one of the main components of the Twisted framework.
The core class of the zope.interface
package is the Interface
class. It allows you to explicitly define a new interface by subclassing. Let's assume that we want to define the obligatory interface for every implementation of a rectangle:
from zope.interface import Interface, Attribute class IRectangle(Interface): width = Attribute("The width of rectangle") height = Attribute("The height of rectangle") def area(): """ Return area of rectangle """ def perimeter(): """ Return perimeter of rectangle """
Some important things to remember when defining interfaces with zope.interface
are as follows:
I
as the name suffix.self
parameter.pass
statement, raise NotImplementedError
, or provide a docstring (preferred).Attribute
class.When you have such a contract defined, you can then define new concrete classes that provide implementation for our IRectangle
interface. In order to do that, you need to use the implementer()
class decorator and implement all of the defined methods and attributes:
@implementer(IRectangle) class Square: """ Concrete implementation of square with rectangle interface """ def __init__(self, size): self.size = size @property def width(self): return self.size @property def height(self): return self.size def area(self): return self.size ** 2 def perimeter(self): return 4 * self.size @implementer(IRectangle) class Rectangle: """ Concrete implementation of rectangle """ def __init__(self, width, height): self.width = width self.height = height def area(self): return self.width * self.height def perimeter(self): return self.width * 2 + self.height * 2
It is common to say that the interface defines a contract that a concrete implementation needs to fulfill. The main benefit of this design pattern is being able to verify consistency between contract and implementation before the object is being used. With the ordinary duck-typing approach, you only find inconsistencies when there is a missing attribute or method at runtime. With zope.interface
, you can introspect the actual implementation using two methods from the zope.interface.verify
module to find inconsistencies early on:
verifyClass(interface, class_object)
: This verifies the class object for existence of methods and correctness of their signatures without looking for attributesverifyObject(interface, instance)
: This verifies the methods, their signatures, and also attributes of the actual object instanceSince we have defined our interface and two concrete implementations, let's verify their contracts in an interactive session:
>>> from zope.interface.verify import verifyClass, verifyObject >>> verifyObject(IRectangle, Square(2)) True >>> verifyClass(IRectangle, Square) True >>> verifyObject(IRectangle, Rectangle(2, 2)) True >>> verifyClass(IRectangle, Rectangle) True
Nothing impressive. The Rectangle
and Square
classes carefully follow the defined contract so there is nothing more to see than a successful verification. But what happens when we make a mistake? Let's see an example of two classes that fail to provide full IRectangle
interface implementation:
@implementer(IRectangle) class Point: def __init__(self, x, y): self.x = x self.y = y @implementer(IRectangle) class Circle: def __init__(self, radius): self.radius = radius def area(self): return math.pi * self.radius ** 2 def perimeter(self): return 2 * math.pi * self.radius
The Point
class does not provide any method or attribute of the IRectangle
interface, so its verification will show inconsistencies already on the class level:
>>> verifyClass(IRectangle, Point) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zope/interface/verify.py", line 102, in verifyClass return _verify(iface, candidate, tentative, vtype='c') File "zope/interface/verify.py", line 62, in _verify raise BrokenImplementation(iface, name) zope.interface.exceptions.BrokenImplementation: An object has failed to implement interface <InterfaceClass __main__.IRectangle> The perimeter attribute was not provided.
The Circle
class is a bit more problematic. It has all the interface methods defined but breaks the contract on the instance attribute level. This is the reason why, in most cases, you need to use the verifyObject()
function to completely verify the interface implementation:
>>> verifyObject(IRectangle, Circle(2)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "zope/interface/verify.py", line 105, in verifyObject return _verify(iface, candidate, tentative, vtype='o') File "zope/interface/verify.py", line 62, in _verify raise BrokenImplementation(iface, name) zope.interface.exceptions.BrokenImplementation: An object has failed to implement interface <InterfaceClass __main__.IRectangle> The width attribute was not provided.
Using zope.inteface
is an interesting way to decouple your application. It allows you to enforce proper object interfaces without the need for the overblown complexity of multiple inheritance, and it also allows to catch inconsistencies early. However, the biggest downside of this approach is the requirement that you explicitly define that the given class follows some interface in order to be verified. This is especially troublesome if you need to verify instances coming from external classes of built-in libraries. zope.interface
provides some solutions for that problem, and you can of course handle such issues on your own by using the adapter pattern, or even monkey-patching. Anyway, the simplicity of such solutions is at least arguable.
Design patterns are meant to make problem solving easier and not to provide you with more layers of complexity. The zope.interface
is a great concept and may greatly fit some projects, but it is not a silver bullet. By using it, you may soon find yourself spending more time on fixing issues with incompatible interfaces for third-party classes and providing never-ending layers of adapters instead of writing the actual implementation. If you feel that way, then this is a sign that something went wrong. Fortunately, Python supports for building lightweight alternative to the interfaces. It's not a full-fledged solution like zope.interface
or its alternatives but it generally provides more flexible applications. You may need to write a bit more code, but in the end you will have something that is more extensible, better handles external types, and may be more future proof.
Note that Python in its core does not have explicit notions of interfaces, and probably will never have, but has some of the features that allow you to build something that resembles the functionality of interfaces. The features are:
The core of our solution is abstract base classes, so we will feature them first.
As you probably know, the direct type comparison is considered harmful and not pythonic. You should always avoid comparisons as follows:
assert type(instance) == list
Comparing types in functions or methods that way completely breaks the ability to pass a class subtype as an argument to the function. The slightly better approach is to use the isinstance()
function that will take the inheritance into account:
assert isinstance(instance, list)
The additional advantage of isinstance()
is that you can use a larger range of types to check the type compatibility. For instance, if your function expects to receive some sort of sequence as the argument, you can compare against the list of basic types:
assert isinstance(instance, (list, tuple, range))
Such a way of type compatibility checking is OK in some situations but it is still not perfect. It will work with any subclass of list
, tuple
, or range
, but will fail if the user passes something that behaves exactly the same as one of these sequence types but does not inherit from any of them. For instance, let's relax our requirements and say that you want to accept any kind of iterable as an argument. What would you do? The list of basic types that are iterable is actually pretty long. You need to cover list, tuple, range, str, bytes, dict, set, generators, and a lot more. The list of applicable built-in types is long, and even if you cover all of them it will still not allow you to check against the custom class that defines the __iter__()
method, but will instead inherit directly from object
.
And this is the kind of situation where abstract base classes (ABC) are the proper solution. ABC is a class that does not need to provide a concrete implementation but instead defines a blueprint of a class that may be used to check against type compatibility. This concept is very similar to the concept of abstract classes and virtual methods known in the C++ language.
Abstract base classes are used for two purposes:
So, let's assume we want to define an interface which ensures that a class has a push()
method. We need to create a new abstract base class using a special ABCMeta
metaclass and an abstractmethod()
decorator from the standard abc
module:
from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """
The abc
module also provides an ABC base class that can be used instead of the metaclass syntax:
from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """
Once it is done, we can use that Pushable
class as a base class for concrete implementation and it will guard us from the instantiation of objects that would have incomplete implementation. Let's define DummyPushable
, which implements all interface methods and the IncompletePushable
that breaks the expected contract:
class DummyPushable(Pushable): def push(self, x): return class IncompletePushable(Pushable): pass
If you want to obtain the DummyPushable
instance, there is no problem because it implements the only required push()
method:
>>> DummyPushable() <__main__.DummyPushable object at 0x10142bef0>
But if you try to instantiate IncompletePushable
, you will get TypeError
because of missing implementation of the interface()
method:
>>> IncompletePushable() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Can't instantiate abstract class IncompletePushable with abstract methods push
The preceding approach is a great way to ensure implementation completeness of base classes but is as explicit as the zope.interface
alternative. The DummyPushable
instances are of course also instances of Pushable
because Dummy is a subclass of Pushable
. But how about other classes with the same methods but not descendants of Pushable
? Let's create one and see:
>>> class SomethingWithPush: ... def push(self, x): ... pass ... >>> isinstance(SomethingWithPush(), Pushable) False
Something is still missing. The SomethingWithPush
class definitely has a compatible interface but is not considered as an instance of Pushable
yet. So, what is missing? The answer is the __subclasshook__(subclass)
method that allows you to inject your own logic into the procedure that determines whether the object is an instance of a given class. Unfortunately, you need to provide it by yourself, as abc
creators did not want to constrain the developers in overriding the whole isinstance()
mechanism. We got full power over it, but we are forced to write some boilerplate code.
Although you can do whatever you want to, usually the only reasonable thing to do in the __subclasshook__()
method is to follow the common pattern. The standard procedure is to check whether the set of defined methods are available somewhere in the MRO of the given class:
from abc import ABCMeta, abstractmethod class Pushable(metaclass=ABCMeta): @abstractmethod def push(self, x): """ Push argument no matter what it means """ @classmethod def __subclasshook__(cls, C): if cls is Pushable: if any("push" in B.__dict__ for B in C.__mro__): return True return NotImplemented
With the __subclasshook__()
method defined that way, you can now confirm that the instances that implement the interface implicitly are also considered instances of the interface:
>>> class SomethingWithPush: ... def push(self, x): ... pass ... >>> isinstance(SomethingWithPush(), Pushable) True
Unfortunately, this approach to the verification of type compatibility and implementation completeness does not take into account the signatures of class methods. So, if the number of expected arguments is different in implementation, it will still be considered compatible. In most cases, this is not an issue, but if you need such fine-grained control over interfaces, the zope.interface
package allows for that. As already said, the __subclasshook__()
method does not constrain you in adding more complexity to the isinstance()
function's logic to achieve a similar level of control.
The two other features that complement abstract base classes are function annotations and type hints. Function annotation is the syntax element described briefly in Chapter 2, Syntax Best Practices – below the Class Level. It allows you to annotate functions and their arguments with arbitrary expressions. As explained in Chapter 2, Syntax Best Practices – below the Class Level, this is only a feature stub that does not provide any syntactic meaning. There is no utility in the standard library that uses this feature to enforce any behavior. Anyway, you can use it as a convenient and lightweight way to inform the developer of the expected argument interface. For instance, consider this IRectangle
interface rewritten from zope.interface
to abstract the base class:
from abc import ( ABCMeta, abstractmethod, abstractproperty ) class IRectangle(metaclass=ABCMeta): @abstractproperty def width(self): return @abstractproperty def height(self): return @abstractmethod def area(self): """ Return rectangle area """ @abstractmethod def perimeter(self): """ Return rectangle perimeter """ @classmethod def __subclasshook__(cls, C): if cls is IRectangle: if all([ any("area" in B.__dict__ for B in C.__mro__), any("perimeter" in B.__dict__ for B in C.__mro__), any("width" in B.__dict__ for B in C.__mro__), any("height" in B.__dict__ for B in C.__mro__), ]): return True return NotImplemented
If you have a function that works only on rectangles, let's say draw_rectangle()
, you could annotate the interface of the expected argument as follows:
def draw_rectangle(rectangle: IRectange): ...
This adds nothing more than information for the developer about expected information. And even this is done through an informal contract because, as we know, bare annotations contain no syntactic meaning. However, they are accessible at runtime, so we can do something more. Here is an example implementation of a generic decorator that is able to verify interface from function annotation if it is provided using abstract base classes:
def ensure_interface(function): signature = inspect.signature(function) parameters = signature.parameters @wraps(function) def wrapped(*args, **kwargs): bound = signature.bind(*args, **kwargs) for name, value in bound.arguments.items(): annotation = parameters[name].annotation if not isinstance(annotation, ABCMeta): continue if not isinstance(value, annotation): raise TypeError( "{} does not implement {} interface" "".format(value, annotation) ) function(*args, **kwargs) return wrapped
Once it is done, we can create some concrete class that implicitly implements the IRectangle
interface (without inheriting from IRectangle
) and update the implementation of the draw_rectangle()
function to see how the whole solution works:
class ImplicitRectangle: def __init__(self, width, height): self._width = width self._height = height @property def width(self): return self._width @property def height(self): return self._height def area(self): return self.width * self.height def perimeter(self): return self.width * 2 + self.height * 2 @ensure_interface def draw_rectangle(rectangle: IRectangle): print( "{} x {} rectangle drawing" "".format(rectangle.width, rectangle.height) )
If we feed the draw_rectangle()
function with an incompatible object, it will now raise TypeError
with a meaningful explanation:
>>> draw_rectangle('foo') Traceback (most recent call last): File "<input>", line 1, in <module> File "<input>", line 101, in wrapped TypeError: foo does not implement <class 'IRectangle'> interface
But if we use ImplicitRectangle
or anything else that resembles the IRectangle
interface, the function executes as it should:
>>> draw_rectangle(ImplicitRectangle(2, 10)) 2 x 10 rectangle drawing
Our example implementation of ensure_interface()
is based on the typechecked()
decorator from the typeannotations
project that tries to provide run-time checking capabilities (refer to https://github.com/ceronman/typeannotations). Its source code might give you some interesting ideas about how to process type annotations to ensure run-time interface checking.
The last feature that can be used to complement this interface pattern landscape are type hints. Type hints are described in detail by PEP 484 and were added to the language quite recently. They are exposed in the new typing
module and are available from Python 3.5. Type hints are built on top of function annotations and reuse this slightly forgotten syntax feature of Python 3. They are intended to guide type hinting and check for various yet-to-come Python type checkers. The typing
module and PEP 484 document aim to provide a standard hierarchy of types and classes that should be used for describing type annotations.
Still, type hints do not seem to be something revolutionary because this feature does not come with any type checker built-in into the standard library. If you want to use type checking or enforce strict interface compatibility in your code, you need to create your own tool because there is none worth recommendation yet. This is why we won't dig into details of PEP 484. Anyway, type hints and the documents describing them are worth mentioning because if some extraordinary solution emerges in the field of type checking in Python, it is highly probable that it will be based on PEP 484.
Abstract base classes are like small building blocks for creating a higher level of abstraction. They allow you to implement really usable interfaces but are very generic and designed to handle lot more than this single design pattern. You can unleash your creativity and do magical things but building something generic and really usable may require a lot of work. Work that may never pay off.
This is why custom abstract base classes are not used so often. Despite that, the collections.abc
module provides a lot of predefined ABCs that allow to verify interface compatibility of many basic Python types. With base classes provided in this module, you can check, for example, whether a given object is callable, mapping, or if it supports iteration. Using them with the isinstance()
function is way better than comparing them against the base python types. You should definitely know how to use these base classes even if you don't want to define your own custom interfaces with ABCMeta
.
The most common abstract base classes from collections.abc
that you will use from time to time are:
Container
: This interface means that the object supports the in
operator and implements the __contains__()
methodIterable
: This interface means that the object supports the iteration and implements the __iter__()
methodCallable
: This interface means that it can be called like a function and implements the __call__()
methodHashable
: This interface means that the object is hashable (can be included in sets and as key in dictionaries) and implements the __hash__
methodSized
: This interface means that the object has size (can be a subject of the len()
function) and implements the __len__()
methodA full list of the available abstract base classes from the collections.abc
module is available in the official Python documentation (refer to https://docs.python.org/3/library/collections.abc.html).
Proxy provides indirect access to an expensive or a distant resource. A Proxy is between a Client and a Subject, as shown in the following figure:
It is intended to optimize Subject accesses if they are expensive. For instance, the memoize()
and lru_cache()
decorators described in Chapter 12, Optimization – Some Powerful Techniques, can be considered as proxies.
A proxy can also be used to provide smart access to a subject. For instance, big video files can be wrapped into proxies to avoid loading them into memory when the user just asks for their titles.
An example is given by the urllib.request
module. urlopen
is a proxy for the content located at a remote URL. When it is created, headers can be retrieved independently from the content itself without the need to read the rest of the response:
>>> class Url(object): ... def __init__(self, location): ... self._url = urlopen(location) ... def headers(self): ... return dict(self._url.headers.items()) ... def get(self): ... return self._url.read() ... >>> python_org = Url('http://python.org') >>> python_org.headers().keys() dict_keys(['Accept-Ranges', 'Via', 'Age', 'Public-Key-Pins', 'X-Clacks-Overhead', 'X-Cache-Hits', 'X-Cache', 'Content-Type', 'Content-Length', 'Vary', 'X-Served-By', 'Strict-Transport-Security', 'Server', 'Date', 'Connection', 'X-Frame-Options'])
This can be used to decide whether the page has been changed before getting its body to update a local copy, by looking at the last-modified
header. Let's take an example with a big file:
>>> ubuntu_iso = Url('http://ubuntu.mirrors.proxad.net/hardy/ubuntu-8.04-desktop-i386.iso') >>> ubuntu_iso.headers()['Last-Modified'] 'Wed, 23 Apr 2008 01:03:34 GMT'
Another use case of proxies is data uniqueness.
For example, let's consider a website that presents the same document in several locations. Extra fields specific to each location are appended to the document, such as a hit counter and a few permission settings. A proxy can be used in that case to deal with location-specific matters and also to point to the original document instead of copying it. So, a given document can have many proxies, and if its content changes, all locations will benefit from it without having to deal with version synchronization.
Generally speaking, proxy pattern is useful for implementing a local handle of something that may live somewhere else to:
Facade provides high-level, simpler access to a subsystem.
A facade is nothing but a shortcut to use a functionality of the application, without having to deal with the underlying complexity of a subsystem. This can be done, for instance, by providing high-level functions at the package level.
Facade is usually done on existing systems, where a package's frequent usage is synthesized in high-level functions. Usually, no classes are needed to provide such a pattern and simple functions in the __init__.py
module are sufficient.
A good example of project that provides a big facade over complicated and complex interfaces is the requests
package (refer to http://docs.python-requests.org/). It really simplifies the madness of dealing with HTTP requests and responses in Python by providing a clean API that is easily readable to developers. It is actually even advertised as HTTP for humans. Such ease of use always comes at some price but eventual tradeoffs and additional overhead does not scare most people from using the Requests project as their HTTP tool of choice. In the end, it allows us to finish projects faster and a developer's time is usually more expensive than hardware.