In the previous section we saw how zc.buildout
could be used to generate an isolated project environment, including project files and dependencies. Buildout automated the process using a configuration file and a special filesystem hierarchy. It is a powerful tool, but may not suit every developer's style. Getting an environment up and running is a fair amount of work and making changes to try out new things requires editing the configuration file and rebuilding everything.
Environment isolation is a definite need for any serious Python or Django developer, though. It becomes very difficult to work in a situation where multiple projects are overlapping and there are conflicting package needs. Developing enhancements to our applications while maintaining a working production copy, like upgrading to a newer version of Django, for example, is impossible without some tool to manage our packages.
A lot of developers will take the obvious approach of creating special local package directories in their own workspace and attaching this location to the $PYTHONPATH
environment variable or sys.path
, swapping around a .pth
file in the system site-packages
location, or managing a collection of symbolic links.
If you've ever tried these approaches you'll probably recognize the difficulties this creates. First among them is that distributed Python applications that use setuptools
or distutils
and a setup.py
script require a lot of special attention to install. Second, tools like easy_install
install to the system site-packages
by default. Both of these kill productivity and defeat the purpose of having a package index.
Buildout solves this problem, but it's not the only solution. Virtual environments take a different approach. Instead of writing configuration files and running build scripts, you can configure a virtual environment, and then use all the standard Python practices for installing packages and configuring tools. No changes to $PYTHONPATH
or sys.path
, no symbolic links, no funny business.
A virtual environment in some ways resembles the isolated environments we saw with buildout. A virtual environment is isolated from all other virtual environments, but not from the system environment. This means if we have installed packages to our system site-packages
location, by default they will be available in our virtual environments as well. Otherwise packages are not available until we install them to the virtual environment.
There are currently several different virtual environment tools. Among these, virtualenv has gained a large following and provides an excellent all-around solution to the virtual environment problem. Virtualenv was written by top Python developer Ian Bicking and is available from PyPI at http://pypi.python.org/pypi/virtualenv.
Once installed, we can use the shell command virtualenv to manipulate environments. Every developer will have different preferences, but it's likely that you'll want to store all of your virtual environments in a single location, perhaps /home/username/virtualenvs
.
When we change to our virtualenvs
directory, we can issue the virtualenv
command, following by a destination directory. This could be the name of our project or some other identifier to distinguish the environment we're creating. Let's create an environment for this book's coleman
project:
$ virtualenv coleman
The results of this command are shown in the following screenshot:
We've just created a complete virtual environment. If we navigate to our environment directory, we'll notice three subdirectories: bin
, include
, and lib
. So far this is similar to what happened when we used buildout to create an environment.
Inside our virtual environment's bin
, we see a copy of our Python interpreter. This interpreter is an actual interpreter binary, copied from our system's Python installation. This is unlike buildout, which creates wrappers around our system Python and provides it with an updated set of paths.
Even though the virtualenv interpreter is not editable, it does have knowledge of the virtual environment. If we were to run it as a Python shell, the path variable will reflect our virtual environment installation path. In order for this interpreter to be recognized as the default when we type python
into a shell, however, we must first activate the environment.
Inside the bin
location there is a shell script called activate
. In order to activate the environment we have to run this shell script using the source command. Source is a built-in bash command that runs a script in our current shell (as opposed to starting a sub-shell). Running activate
with source updates our shell environment and also adds a useful indicator to our shell prompt to inform us of what virtual environment we're currently using, as shown below:
The activate
command has done two things: updated our shell's $PATH
environment variable to include our virtual environment's bin location and updated a $VIRTUAL_ENV
shell environment variable that points to our virtual environment directory.
When we ran virtualenv earlier, it actually performed some magic whereby our virtual environment's interpreter can detect our activated environment and adjust its path accordingly. The results of these operations are shown in the following screenshot:
Once we have activated an environment, we can deactivate it in a similar way by just typing deactivate
at the shell prompt. Our virtual environment indicator should disappear from our prompt and we'll be back to using the system Python interpreter. We can activate other virtual environments by navigating to their location in the shell and running the new environment's bin/activate
.
Now that we've created a virtual environment for our project and activated it using the bin/activate
script, we can go about our work. Because virtualenv has updated our shell path, every time we run python, our virtual environment's python
will run.
Since we can now use the python command at will, something cool happens: we can begin installing packages using setup.py
or easy_install
. As long as our virtual environment is activated, we can easy_install
to our heart's content and it all goes into our virtual environment.
In general, it is good practice when using virtual environments to keep the system site-packages
location as light as possible. This prevents confusion and other potential mishaps. Our easy_install
s from an activated virtual environment will include the system site-packages
when looking for dependencies and such, but it is still recommended that most packages be installed virtually.
This will lead to some duplication if you are working on multiple projects that use the same set of packages. For these cases you may consider installing the package system-wide, especially if it's particularly important or difficult to install. The MySQLdb
package and other database interfaces are good examples of candidates for system-wide installation.
Django would be an example of something not to install system wide, unless you are absolutely certain you won't be developing two separate projects that are running against different versions of the framework. Even if you are, still consider keeping all packages installed virtually. The disk space is there, why not use it?
Another important point to note is that the package manager for your operating system may, either by default or through a dependency negotiation, have installed packages to your Python's system site-packages
without you necessarily being aware. It's a good idea to check out what packages and modules are installed in your system before switching to virtualenv. In fact, you may wish to empty your system site-packages
altogether. You can uninstall any module by deleting its .egg
, directory, and other metadata files.
Doug
Hellman has released a set of extension scripts for virtualenv called virtualenvwrapper
. These are a set of shell functions that enhance the virtual environment configuration, primarily by organizing and structuring our environment locations. It includes pre and post hooks for several operations, including creating an environment, activating a new environment, and deactivating an environment.
The activation hooks allow for lots of interesting possibilities. For example, you could attach a post-activate hook that automatically runs a shell script or AppleScript to launch and configure your text editor whenever you change virtual environments.
The virtualenvwrapper scripts are a highly recommended enhancement to virtualenv. They are available from: http://www.doughellmann.com/projects/virtualenvwrapper/.