Namespace packages can be understood as a way of grouping related packages or modules higher than a meta-package level, where each of these packages can be installed independently.

Namespace packages are especially useful if you have your application components developed, packaged, and versioned independently but you still want to access them from the same namespace. This helps to make clear to which organization or project every package belongs. For instance, for some imaginary Acme company, the common namespace could be acme. The result could lead to the creation of the general acme namespace package that will serve as a container for other packages from this organization. For example, if someone from Acme wants to contribute to this namespace with, for example, an SQL-related library, he can create a new acme.sql package that registers itself in acme.

It is important to know the difference between normal and namespace packages and what problems they solve. Normally (without namespace packages), you would create a package acme with an sql subpackage/submodule with the following file structure:

$ tree acme/
acme/
├── acme
│   ├── __init__.py
│   └── sql
│       └── __init__.py
└── setup.py

2 directories, 3 files

Whenever you want to add a new subpackage, let's say templating, you are forced to include it in the source tree of acme:

$ tree acme/
acme/
├── acme
│   ├── __init__.py
│   ├── sql
│   │   └── __init__.py
│   └── templating
│       └── __init__.py
└── setup.py

3 directories, 4 files

Such an approach makes independent development of acme.sql and acme.templating almost impossible. The setup.py script will also have to specify all dependencies for every subpackage, so it is impossible (or at least very hard) to have an installation of just some of the acme components optionally. Also, it is an unresolvable issue if some of the subpackages have conflicting requirements.

With namespace packages, you can store the source tree for each of these subpackages independently:

$ tree acme.sql/
acme.sql/
├── acme
│   └── sql
│       └── __init__.py
└── setup.py

2 directories, 2 files

$ tree acme.templating/
acme.templating/
├── acme
│   └── templating
│       └── __init__.py
└── setup.py

2 directories, 2 files

You can also register them independently in PyPI or any package index you use. Users can choose which of the subpackages they want to install from the acme namespace but they never install the general acme package (it does not exist):

$ pip install acme.sql acme.templating

Note that independent source trees are not enough to create namespace packages in Python. You need a bit of additional work if you don't want your packages to overwrite each other. Also, proper handling may be different depending on the Python language version you target. Details of that are described in the next two sections.

If you use and target only Python 3, then there is good news for you. PEP 420 (Implicit Namespace Packages) introduced a new way to define namespace packages. It is a part of the standards track and became an official part of the language since the 3.3 version. In short, every directory that contains Python packages or modules (including namespace packages too) is considered a namespace package if it does not contain the __init__.py file. So, the following are examples of file structures presented in the previous section:

$ tree acme.sql/
acme.sql/
├── acme
│   └── sql
│       └── __init__.py
└── setup.py

2 directories, 2 files

$ tree acme.templating/
acme.templating/
├── acme
│   └── templating
│       └── __init__.py
└── setup.py

2 directories, 2 files

They are enough to define that acme is a namespace package in Python 3.3 and later. Minimal setup.py scripts using setup tools will look like the following:

from setuptools import setup


setup(
    name='acme.templating',
    packages=['acme.templating'],
)

Unfortunately, setuptools.find_packages() does not support PEP 420 at the time of writing this book. Anyway, this may change in the future. Also, a requirement to explicitly define a list of packages seems to be a very small price for easy integration of namespace packages.

There is no way to make the namespaces packages in PEP 420 layout to work in Python versions older than 3.3. Still, this concept is very old and commonly used in such mature projects like Zope, so it is definitely possible to use them but without implicit definition. In older versions of Python, there are several ways to define that the package should be treated as a namespace.

The simplest one is to create a file structure for each component that resembles an ordinary package layout without namespace packages and leave everything to setuptools. So, the example layout for acme.sql and acme.templating could be the following:

$ tree acme.sql/
acme.sql/
├── acme
│   ├── __init__.py
│   └── sql
│       └── __init__.py
└── setup.py

2 directories, 3 files

$ tree acme.templating/
acme.templating/
├── acme
│   ├── __init__.py
│   └── templating
│       └── __init__.py
└── setup.py

2 directories, 3 files

Note that for both acme.sql and acme.templating, there is an additional source file acme/__init__.py. This must be left empty. The acme namespace package will be created if we provide this name as a value of the namespace_packages keyword argument of the setuptools.setup() function:

from setuptools import setup

setup(
    name='acme.templating',
    packages=['acme.templating'],
    namespace_packages=['acme'],
)

Easiest does not mean best. setuptools, in order to register a new namespace, will call for the pkg_resources.declare_namespace() function in your __init__.py file. It will happen even if the __init__.py file is empty. Anyway, as the official documentation says, it is your own responsibility to declare namespaces in the __init__.py file, and this implicit behavior of setuptools may be dropped in the future. In order to be safe and "future-proof", you need to add the following line to the file acme/__init__.py:

__import__('pkg_resources').declare_namespace(__name__)