Though buildout can be installed in the system-level site-packages, it also provides a bootstrapping script that can be used to set up and configure a project without the developer needing to install anything beyond a Python interpreter. The bootstrapping process installs the buildout package itself into the project, which can then be used to actually create the project build, installing all the relevant Python packages and providing a special project-specific Python interpreter.

This two-step process is an excellent way to get other developers up and running on a project. The bootstrapping tool and buildout configuration can be placed in a version control repository such that all a new developer needs to do is check out the code, run bootstrap.py to prepare for a buildout, and then run the buildout tool itself. This also holds true for server deployments, where the buildout can happen on a fresh server with just these simple commands.

There are other ways to use buildout without bootstrapping. If it is installed in the system site packages, then bootstrap.py is unnecessary and new developers or deployment scripts only need to run the buildout tool directly.

After running the buildout tool, our project directory will contain a half dozen or so additional files and directories. This is where buildout does its work. These subdirectories include: bin, developer-eggs, eggs, and parts. The bin is for executable files, like the buildout tool itself and the special Python interpreters. The eggs location is where buildout stores third-party eggs, while developer-eggs stores the eggs that we're currently developing (our application modules). Parts functions as a workspace that buildout uses while building things.

Let's take a moment to talk about eggs. Eggs are created by the setuptools module and are a quasi-standard way of packaging modules into a tidy unit (the standard way is built-in to Python and called distutils, of which setuptools is an enhanced version). They are typically used to package up a specific version of a Python module such that it can be easily shared with other developers.

The .egg format differs from a standard Python module (a simple directory with __init__.py) in that it includes metadata in addition to code. This is accomplished through the use of either an EGG-INFO subdirectory at the module level or an .egg-info directory at the module's parent level. An .egg's metadata typically describes the package, including the author information, license, a homepage URL, and, importantly, any dependencies.

Typically when you easy_install something into your system-level Python, you are downloading the module from the Python Package Index (PyPI, formerly known as The Cheese Shop). This comes in .egg format and gets placed into your system's site-packages directory. Since the .egg includes additional information, like dependencies, easy_install will attempt to install these as well.

In our case, we want to avoid installing things at the system level. So buildout creates and uses the eggs and developer-eggs directories in our project hierarchy as a place to easy_install modules. It will also create a Python interpreter that is aware of these special locations for our project. After the buildout finishes, we will use this interpreter, not the system Python, to run our application.

The buildout tool uses a configuration file to figure out what it needs to do to install our project and its dependencies. This is a text file called buildout.cfg that uses a very simple format.

The first thing we define is a buildout section, which is the only required portion of buildout.cfg. This section has two goals. First, it defines the parts our project needs to build in order to function. These are almost equivalent to dependencies our project needs in order to run. In our case this would be, at the very least, Django. The second goal is to define the modules we're developing (our web application, for example, the Coleman project we've built in this book). These will be translated into developer-eggs.

Before going any farther, we should look at an example [buildout] section from buildout.cfg:

[buildout]
develop = .
parts = django
eggs = django-coleman

Here you can see the develop and parts definitions. In the example, we only supply a single value to each configuration option, but in real-world projects these options can have as many values as needed, each separated by a space character.

The develop option points to .—the current directory. This tells buildout that the modules we're developing live in the project's root directory and that it will find a setup.py file there, which it can use to turn our modules into an .egg. In order for scripts in our buildout installation to access this egg, though, it needs to be added to the eggs/ directory. We tell buildout to install our egg there with the eggs = django-coleman option. The name of our egg must match the name we specify in our setup script. Clearly we have additional work to do for this to work. We must now write this setup script, and then we will revisit the parts option.

The setup.py file is a setup script for our project. It is used to package up the modules we develop. When using setuptools, this package will be an egg. There are other cases, such as using pure distutils, where the setup script doesn't generate an egg. We will discuss the differences between setuptools and distutils setup scripts later in the chapter, but since zc.buildout uses setuptools, we will write our setup.py accordingly.

Our setup script can include all kinds of metadata, which setuptools will automatically roll up into the generated .egg file. For this example, we will keep it extremely simple, however, and specify only what is needed to get our module built:

from setuptools import setup
setup(
name='django-coleman',
version='0.0.1',
package_dir={'': 'coleman'},
       )

The name and version arguments are self explanatory, though important because they affect the ultimate name of the generated .egg file (django-coleman-0.0.1.egg). The package_dir argument points setuptools to a file-system location relative to the setup.py file where it will find our Python module. This setup script implies the following project layout (minus buildout's configuration and working directories):

setup.py 
coleman/
coleman/__init__.py
 ...

The setup script will package the contents of the coleman/ subdirectory into our egg.

This more than satisfies the basic requirements for our setup script. When we run our buildout command, it will be able to use setup.py to turn our project into an egg, and then store it in eggs/ where our buildout scripts or interpreters can find it. We have effectively built our project inside a fully isolated Python environment, shut off from the rest of our system. The final step is to finish up our buildout.cfg by discussing our project's parts.

Parts are buildout's way of defining additional modules or components that our project needs in order to function. A part can be almost anything, but are often third-party packages from Python Package Index. We specify our project's parts in the buildout section of our buildout.cfg file using the parts = option. This is a list of part names separated by whitespace. Each part will have its own section later in our buildout.cfg file.

For each part we define, buildout needs to know how to install and/or configure it. It does this using a mechanism called a recipe. Several recipes come included with buildout, but we can also write custom ones. Recipes are just normal Python classes so they can do anything a Python script can do. This includes interacting with the operating system, local storage, or even compiling C extension modules.

A recipe is a Python class with three required methods: __init__, install, and update. These methods must perform a certain set of functions for buildout to successfully build the part for our project. We will not write a custom recipe here because most of the time the built-in recipes will work for us. In addition, hundreds of additional recipes have been contributed by the Python community and are available for download from PyPI. If none of these fit your own needs, it is easy to create a custom one and the process is well documented in the buildout documentation.

Among the recipes that come included with buildout or are available in PyPI is djangorecipe. This is a recipe designed specifically for building out Django projects. It has a lot of features, including the ability to set up a Django project and point to a settings file. The end result is a script in our buildout bin/ called Django that wraps up all of these things and wraps Django's standard manage.py utility.

To help explain this, let's look at an example. Earlier we wrote the buildout section of our buildout.cfg file, here we will add another section for our Django part:

[buildout]
develop = .
parts = django
eggs = django-coleman

[django]
recipe = djangorecipe
version = 1.0.2
eggs = ${buildout:eggs}

The [django] section corresponds to our django part. We tell buildout to use the djangorecipe, which it will obtain automatically from PyPI, to build this part. We also tell the recipe we want Django version 1.0.2, using the version option. Finally, the eggs option tells buildout we want the same eggs to be available to Django as those defined in our buildout section.

When we run buildout in the directory with buildout.cfg the output should resemble the following:

Getting distribution for 'zc.recipe.egg'.
Got zc.recipe.egg 1.2.2.
Getting distribution for 'djangorecipe'.
Got djangorecipe 0.20.
Uninstalling django.
Installing django.
django: Downloading Django from: http://www.djangoproject.com/download/1.0.2/tarball/
Generated script '/Users/jesse/src/django-coleman/bin/django'.

The bin/ directory now contains a django script. This is the wrapper around the standard manage.py and supports all the commands you might expect: runserver, syncdb, shell, and so on. Only this script now comes prepared with our modules and any other eggs we specify available to Django. It also creates a project/ directory in our buildout location, which contains a stock Django project configuration, including settings.py file and root urls.py. If we already have these things created in a Django project, we can include them in the options for our django part:

[django]
recipe = djangorecipe
version = 1.0.2
eggs = ${buildout:eggs}
project = myproject

If a myproject directory exists and contains our own settings and URLs files, the djangorecipe will not create new ones.

Now we have an isolated Python environment with four things:

We're all set to work with this fully configured instance of our Django project. We can add additional parts to our buildout file as needed and rerun buildout to download and install any additions. The buildout.cfg file can now be put into version control and, if we really want to get fancy, we can set up our development server to automatically run buildout whenever our configuration changes.

One thing is missing, however, and that is a straightforward way to deploy this isolated version of our project to an actual web server. The djangorecipe can help here too. We just need to add the following option to our django part in buildout.cfg:

wsgi = True

With this addition, buildout will create a django.wsgi file in the bin/ directory. This WSGI script is configured to load our isolated project as a WSGI application and can be dropped directly into our Apache configuration file if we're using mod_wsgi. For the curious, this WSGI script looks like this:

#!/usr/bin/python
import sys
sys.path[0:0] = [
'/Users/jesse/src/django-coleman/eggs/django-coleman-0.0.1.egg',
'/Users/jesse/src/django-coleman/eggs/djangorecipe-0.20-py2.6.egg',
'/Users/jesse/src/django-coleman/eggs/zc.recipe.egg-1.2.2-py2.6.egg',
'/Users/jesse/src/django-coleman/parts/django',
'/Users/jesse/src/django-coleman,
  ]

import djangorecipe.wsgi
application = djangorecipe.wsgi.main('myproject.development', logfile='')

As an even further level of automation, if we run mod_wsgi in daemon mode, whenever our buildout script runs and makes changes to our isolated environment, it will automatically update the WSGI script. This update will update the file's time stamp, which will act as a notice to Apache's mod_wsgi that the script has changed and needs to be reloaded. The mod_wsgi module will reload our entire project without restarting Apache and without requiring us to type any additional commands. Now that's automation!

Sometimes we need to include additional third-party modules, but don't want them installed into our project's isolated environment. Maybe we want to try out some functions in a package we're considering for use. We can do this with buildout as well by defining a new part, installing an egg to it and having buildout generate a custom interpreter. Such a parts section would look like this:

[testout]
recipe = zc.recipe.egg
interpreter = testout
eggs = elementtree

If we were to add a parts section like this to our buildout.cfg file, a couple of things will happen. First, an elementtree egg will be downloaded from PyPI and setup in our eggs location. Second, a custom Python interpreter will be created in the bin/ location called bin/testout. When we run this interpreter, we will see a standard Python shell with only the elementtree egg available from our isolated environment's eggs. Our Django instance will not include this egg, keeping them nicely separated. This is especially useful when trying out different versions of a PyPI package.

Adding packages individually in this way is sometimes useful, but often too complicated. Buildout seems to work best as a tool for deployment onto servers or for building project packages. It can be used in many ways, but for local development tasks, it can feel like overkill to make changes to a configuration just to test out a package. There are better solutions for this use case, one of which we will discuss in the next section on virtual environments.

Finally, buildout can be used for more than just deployments. If you're aim is to write reusable modules that you want to contribute to the community, you can use buildout to automatically run unit tests and then package everything up into an egg. It can even upload and register your eggs with PyPI. There is much, much more than buildout can do and an excellent starting point for additional information is the buildout tutorial at: http://www.buildout.org/docs/tutorial.html.