The import Statement

You can use any Python source file as a module by executing an import statement in another Python source file. import has the following syntax:

import modname [as varname][,...]

After the import keyword comes one or more module specifiers separated by commas. In the simplest, most common case, a module specifier is just modname, an identifier—a variable that Python binds to the module object when the import statement finishes. In this case, Python looks for the module of the same name to satisfy the import request. For example:

import mymodule

looks for the module named mymodule and binds the variable named mymodule in the current scope to the module object. modname can also be a sequence of identifiers separated by dots (.) to name a module in a package, as covered in “Packages”.

When as varname is part of a module specifier, Python looks for a module named modname and binds the module object to the variable varname. For example:

import mymodule as alias

looks for the module named mymodule and binds the module object to variable alias in the current scope. varname must always be a simple identifier.

import mymodule
a = mymodule.f()

import mymodule as alias
a = alias.f()

# abs takes a numeric argument; let's make it accept a string as well
import builtins
_abs = builtins.abs                          # save original built-in
def abs(str_or_num):
    if isinstance(str_or_num, str):          # if arg is a string
        return ''.join(sorted(set(str_or_num)))  # get this instead
    return _abs(str_or_num)                  # call real built-in
builtins.abs = abs                    # override built-in w/wrapper 

The from Statement

Python’s from statement lets you import specific attributes from a module into the current namespace. from has two syntax variants:

from modname import attrname [as varname][,...]
from modname import *

A from statement specifies a module name, followed by one or more attribute specifiers separated by commas. In the simplest and most common case, an attribute specifier is just an identifier attrname, which is a variable that Python binds to the attribute of the same name in the module named modname. For example:

from mymodule import f

modname can also be a sequence of identifiers separated by dots (.) to name a module within a package, as covered in “Packages”.

When as varname is part of an attribute specifier, Python gets from the module the value of attribute attrname and binds it to variable varname. For example:

from mymodule import f as foo

attrname and varname are always simple identifiers.

You may optionally enclose in parentheses all the attribute specifiers that follow the keyword import in a from statement. This is sometimes useful when you have many attribute specifiers, in order to split the single logical line of the from statement into multiple logical lines more elegantly than by using backslashes ():

from some_module_with_a_long_name import (
    another_name, and_another as x, one_more, and_yet_another as y)

from mymodule import *

Built-in Modules

When a module is loaded, __import__ first checks whether the module is built-in. The tuple sys.builtin_module_names names all built-in modules, but rebinding that tuple does not affect module loading. When Python loads a built-in module, as when it loads any other extension, Python calls the module’s initialization function. The search for built-in modules also looks for modules in platform-specific locations, such as the Registry in Windows.

Searching the Filesystem for a Module

If module M is not built-in, __import__ looks for M’s code as a file on the filesystem. __import__ looks at the strings, which are the items of list sys.path, in order. Each item is the path of a directory, or the path of an archive file in the popular ZIP format. sys.path is initialized at program startup, using the environment variable PYTHONPATH (covered in “Environment Variables”), if present. The first item in sys.path is always the directory from which the main program is loaded. An empty string in sys.path indicates the current directory.

Your code can mutate or rebind sys.path, and such changes affect which directories and ZIP archives __import__ searches to load modules. Changing sys.path does not affect modules that are already loaded (and thus already recorded in sys.modules) when you change sys.path.

If a text file with the extension .pth is found in the PYTHONHOME directory at startup, the file’s contents are added to sys.path, one item per line. .pth files can contain blank lines and comment lines starting with the character #; Python ignores any such lines. .pth files can also contain import statements (which Python executes before your program starts to execute), but no other kinds of statements.

When looking for the file for the module M in each directory and ZIP archive along sys.path, Python considers the following extensions in this order:

1. .pyd and .dll (Windows) or .so (most Unix-like platforms), which indicate Python extension modules. (Some Unix dialects use different extensions; e.g., .sl on HP-UX.) On most platforms, extensions cannot be loaded from a ZIP archive—only source or bytecode-compiled Python modules can.

2. .py, which indicates Python source modules.

3. .pyc (or .pyo, in v2, if Python is run with option -O), which indicates bytecode-compiled Python modules.

4. If a .py file is found, in v3 only, then Python also looks for a directory called ___pycache__; if such a directory is found, then Python looks in that directory for the extension .<tag>.pyc, where <tag> is a string specific to the version of Python that is looking for the module.

One last path in which Python looks for the file for module M is __init__.py, meaning a file named __init__.py in a directory named M, as covered in “Packages”.

Upon finding source file M.py, Python (v3) compiles it to M.<tag>.pyc, unless the bytecode file is already present, is newer than M.py, and was compiled by the same version of Python. If M.py is compiled from a writable directory, Python creates a __pycache__ directory if necessary and saves the bytecode file to the filesystem in that subdirectory so that future runs won’t needlessly recompile. (In v2, M.py compiles to M.pyc or M.pyo, and Python saves the bytecode file in the same directory as M.py.) When the bytecode file is newer than the source file (based on an internal timestamp in the bytecode file, not on trusting the date as recorded in the filesystem), Python does not recompile the module.

Once Python has the bytecode, whether just built by compilation or read from the filesystem, Python executes the module body to initialize the module object. If the module is an extension, Python calls the module’s initialization function.

The Main Program

Execution of a Python application normally starts with a top-level script (also known as the main program), as explained in “The python Program”. The main program executes like any other module being loaded, except that Python keeps the bytecode in memory without saving it to disk. The module name for the main program is always '__main__', both as the __name__ global variable (module attribute) and as the key in sys.modules.

Code in a Python module can test if the module is being used as the main program by checking if global variable __name__ has the value '__main__'. The idiom:

if __name__ == '__main__':

is often used to guard some code so that it executes only when the module is run as the main program. If a module is meant only to be imported, it should normally execute unit tests when it is run as the main program, as covered in “Unit Testing and System Testing”.

Reloading Modules

Python loads a module only the first time you import the module during a program run. When you develop interactively, you need to make sure you explicitly reload your modules each time you edit them (some development environments provide automatic reloading).

To reload a module, in v3, pass the module object (not the module name) as the only argument to the function reload from the importlib module (in v2, call the built-in function reload instead, to the same effect). importlib.reload(M) ensures the reloaded version of M is used by client code that relies on import M and accesses attributes with the syntax M.A. However, importlib.reload(M) has no effect on other existing references bound to previous values of M’s attributes (e.g., with a from statement). In other words, already-bound variables remain bound as they were, unaffected by reload. reload’s inability to rebind such variables is a further incentive to use import rather than from.

reload is not recursive: when you reload module M, this does not imply that other modules imported by M get reloaded in turn. You must arrange to reload, by explicit calls to the reload function, each and every module you have modified.

Circular Imports

Python lets you specify circular imports. For example, you can write a module a.py that contains import b, while module b.py contains import a.

If you decide to use a circular import for some reason, you need to understand how circular imports work in order to avoid errors in your code.

Say that the main script executes import a. As discussed earlier, this import statement creates a new empty module object as sys.modules['a'] and then the body of module a starts executing. When a executes import b, this creates a new empty module object as sys.modules['b'], and then the body of module b starts executing. The execution of a’s module body cannot proceed until b’s module body finishes.

Now, when b executes import a, the import statement finds sys.modules['a'] already bound, and therefore binds global variable a in module b to the module object for module a. Since the execution of a’s module body is currently blocked, module a is usually only partly populated at this time. Should the code in b’s module body immediately try to access some attribute of module a that is not yet bound, an error results.

If you insist on keeping a circular import, you must carefully manage the order in which each module binds its own globals, imports other modules, and accesses globals of other modules. You can have greater control on the sequence in which things happen by grouping your statements into functions, and calling those functions in a controlled order, rather than just relying on sequential execution of top-level statements in module bodies. Usually, removing circular dependencies is easier than ensuring bomb-proof ordering with circular dependencies.

sys.modules Entries

__import__ never binds anything other than a module object as a value in sys.modules. However, if __import__ finds an entry already in sys.modules, it returns that value, whatever type it may be. The import and from statements internally rely on __import__, so they too can end up using objects that are not modules. This lets you set class instances as entries in sys.modules, in order to exploit features such as __getattr__ and __setattr__ special methods, covered in “General-Purpose Special Methods”. This rarely needed advanced technique lets you import module-like objects whose attributes you can compute on the fly. Here’s a toy-like example:

class TT(object):
    def __getattr__(self, name): return 23
import sys
sys.modules[__name__] = TT()

The first import of this code as a module overwrites the module’s sys.modules entry with an instance of class TT. Any attribute name you try to get from it then appears to have the integer value 23.

Custom Importers

An advanced, rarely needed functionality that Python offers is the ability to change the semantics of some or all import and from statements.

import sys, types

class ImporterAndLoader(object):
     '''importer and loader are often a single class'''
     fake_path = '!dummy!'
     def __init__(self, path):
         # only handle our own fake-path marker
         if path != self.fake_path: raise ImportError
     def find_module(self, fullname):
         # don't even try to handle any qualified module name
         if '.' in fullname: return None
         return self
     def create_module(self, spec):
         # run in v3 only: create module "the default way"
         return None
     def exec_module(self, mod):
         # run in v3 only: populate the already-initialized module
         # just print a message in this toy example
         print('NOTE: module {!r} not yet written'.format(mod))
     def load_module(self, fullname):
         # run in v2 only: make and populate the module
         if fullname in sys.modules: return sys.modules[fullname]
         # just print a message in this toy example
         print('NOTE: module {!r} not written yet'.format(fullname))
         # make new empty module, put it in sys.modules
         mod = sys.modules[fullname] = types.ModuleType(fullname)
         # minimally initialize new module and return it
         mod.__file__ = 'dummy_{}'.format(fullname)
         mod.__loader__ = self
         return mod
# add the class to the hook and its fake-path marker to the path
sys.path_hooks.append(ImporterAndLoader)
sys.path.append(ImporterAndLoader.fake_path)

if __name__ == '__main__':      # self-test when run as main script
    import missing_module       # importing a simple missing module
    print(missing_module)       # ...should succeed
    print(sys.modules.get('missing_module'))  # ...should also succeed

Special Attributes of Package Objects

A package P’s __file__ attribute is the string that is the path of P’s module body—that is, the path of the file P/__init__.py. P’s __package__ attribute is the name of P’s package.

A package P’s module body—that is, the Python source that is in the file P/__init__.py—can optionally set a global variable named __all__ (just like any other module can) to control what happens if some other Python code executes the statement from P import *. In particular, if __all__ is not set, from P import * does not import P’s modules, but only names that are set in P’s module body and lack a leading _. In any case, this is not recommended usage.

A package P’s __path__ attribute is the list of strings that are the paths to the directories from which P’s modules and subpackages are loaded. Initially, Python sets __path__ to a list with a single element: the path of the directory containing the file __init__.py that is the module body of the package. Your code can modify this list to affect future searches for modules and subpackages of this package. This advanced technique is rarely necessary, but can be useful when you want to place a package’s modules in several disjoint directories. In v3 only, a namespace package, as covered next, is the usual way to accomplish this goal.

Namespace Packages (v3 Only)

In v3 only, on import foo, when one or more directories that are immediate children of sys.path members are named foo, and none of them contains a file named __init__.py, Python deduces that foo is a namespace package. As a result, Python creates (and assigns to sys.modules['foo']) a package object foo without a __file__ attribute; foo.__path__ is the list of all the various directories that make up the package (and, like for a normal package, your code may optionally choose to further alter it). This advanced approach is rarely needed.

Absolute Versus Relative Imports

As mentioned in “Packages”, an import statement normally expects to find its target somewhere on sys.path, a behavior known as an absolute import (to ensure this reliable behavior in v2, start your module with from __future__ import absolute_import). Alternatively, you can explicitly use a relative import, meaning an import of an object from within the current package. Relative imports use module or package names beginning with one or more dots, and are only available in the from statement. from . import X looks for the module or object named X in the current package; from .X import y looks in module or subpackage X within the current package for the module or object named y. If your package has subpackages, their code can access higher-up objects in the package by using multiple dots at the start of the module or subpackage name you place between from and import. Each additional dot ascends the directory hierarchy one level. Getting too fancy with this feature can easily damage your code’s clarity, so use it with care, and only when necessary.

Python Wheels (and Eggs)

Python wheels (like their precursor, eggs, still supported but not recommended for future development) are an archive format including structured metadata as well as Python code. Both formats, especially wheels, offer excellent ways to package and distribute your Python packages, and setuptools (with the wheel extension, easily installed with, of course, pip install wheel) works seamlessly with them. Read all about them online and in Chapter 25.

Enter the Virtual Environment

The introduction of the pip utility created a simple way to install (and, for the first time, to uninstall) packages and modules in a Python environment. Modifying the system Python’s site-packages still requires administrative privileges, and hence so does pip (although it can optionally install somewhere other than site-packages). Installed modules are still visible to all programs.

The missing piece is the ability to make controlled changes to the Python environment, to direct the use of a specific interpreter and a specific set of Python libraries. That is just what virtual environments (virtualenvs) give you. Creating a virtualenv based on a specific Python interpreter copies or links to components from that interpreter’s installation. Critically, though, each one has its own site-packages directory, into which you can install the Python resources of your choice.

Creating a virtualenv is much simpler than installing Python, and requires far less system resources (a typical newly created virtualenv takes less than 20 MB). You can easily create and activate them on demand, and deactivate and destroy them just as easily. You can activate and deactivate a virtualenv as many times as you like during its lifetime, and if necessary use pip to update the installed resources. When you are done with it, removing its directory tree reclaims all storage occupied by the virtualenv. A virtualenv’s lifetime can be from minutes to months.

What Is a Virtual Environment?

A virtualenv is essentially a self-contained subset of your Python environment that you can switch in or out on demand. For a Python X.Y interpreter it includes, among other things, a bin directory containing a Python X.Y interpreter and a lib/pythonX.Y/site-packages directory containing preinstalled versions of easy-install, pip, pkg_resources, and setuptools. Maintaining separate copies of these important distribution-related resources lets you update them as necessary rather than forcing reliance on the base Python distribution.

A virtualenv has its own copies of (on Windows), or symbolic links to (on other platforms), Python distribution files. It adjusts the values of sys.prefix and sys.exec_prefix, from which the interpreter and various installation utilities determine the location of some libraries. This means that pip can install dependencies in isolation from other environments, in the virtualenv’s site-packages directory. In effect the virtualenv redefines which interpreter runs when you run the python command and which libraries are available to it, but leaves most aspects of your Python environment (such as the PYTHONPATH and PYTHONHOME variables) alone. Since its changes affect your shell environment they also affect any subshells in which you run commands.

With separate virtualenvs you can, for example, test two different versions of the same library with a project, or test your project with multiple versions of Python (very useful to check v2/v3 compatibility of your code). You can also add dependencies to your Python projects without needing any special privileges, since you normally create your virtualenvs somewhere you have write permission.

For a long time the only way to create virtual environments was the third-party virtualenv package, with or without help from virtualenvwrapper, both of which are still available for v2. You can read more about these tools in the Python Packaging User Guide. They also work with v3, but the 3.3 release added the venv module, making virtual environments a native feature of Python for the first time. New in 3.6: use python -m venv envpath in preference to the pyvenv command, which is now deprecated.

Creating and Deleting Virtual Environments

In v3 the command python -m venv envpath creates a virtual environment (in the envpath directory, which it also creates if necessary) based on the Python interpreter used to run the command. You can give multiple directory arguments to create with a single command several virtual environments, into which you then install different sets of dependencies. venv can take a number of options, as shown in Table 6-1.

v2 users must use the python -m virtualenv command, which does not accept multiple directory arguments.

The following terminal session shows the creation of a virtualenv and the structure of the directory tree created. The listing of the bin subdirectory shows that this particular user by default uses a v3 interpreter installed in /usr/local/bin.

machine:~ user$ python3 -m venv /tmp/tempenv
machine:~ user$ tree -dL 4 /tmp/tempenv
/tmp/tempenv
├── bin
├── include
└── lib
    └── python3.5
        └── site-packages
            ├── __pycache__
            ├── pip
            ├── pip-8.1.1.dist-info
            ├── pkg_resources
            ├── setuptools
            └── setuptools-20.10.1.dist-info

11 directories
machine:~ user$ ls -l /tmp/tempenv/bin/
total 80
-rw-r--r-- 1 sh wheel 2134 Oct 24 15:26 activate
-rw-r--r-- 1 sh wheel 1250 Oct 24 15:26 activate.csh
-rw-r--r-- 1 sh wheel 2388 Oct 24 15:26 activate.fish
-rwxr-xr-x 1 sh wheel  249 Oct 24 15:26 easy_install
-rwxr-xr-x 1 sh wheel  249 Oct 24 15:26 easy_install-3.5
-rwxr-xr-x 1 sh wheel  221 Oct 24 15:26 pip
-rwxr-xr-x 1 sh wheel  221 Oct 24 15:26 pip3
-rwxr-xr-x 1 sh wheel  221 Oct 24 15:26 pip3.5
lrwxr-xr-x 1 sh wheel    7 Oct 24 15:26 python->python3
lrwxr-xr-x 1 sh wheel   22 Oct 24 15:26 python3->/usr/local/bin/python3

Deletion of the virtualenv is as simple as removing the directory in which it resides (and all subdirectories and files in the tree: rm -rf envpath in Unix-like systems). Ease of removal is a helpful aspect of using virtualenvs.

The venv module includes features to help the programmed creation of tailored environments (e.g., by preinstalling certain modules in the environment or performing other post-creation steps). It is comprehensively documented online, and we therefore do not cover the API further in this book.

Working with Virtual Environments

To use a virtualenv you activate it from your normal shell environment. Only one virtualenv can be active at a time—activations don’t “stack” like function calls. Activation conditions your Python environment to use the virtualenv’s Python interpreter and site-packages (along with the interpreter’s full standard library). When you want to stop using those dependencies, deactivate the virtualenv and your standard Python environment is once again available. The virtualenv directory tree continues to exist until deleted, so you can activate and deactivate it at will.

Activating a virtualenv in Unix-based environments requires use of the source shell command so that the commands in the activation script make changes to the current shell environment. Simply running the script would mean its commands were executed in a subshell, and the changes would be lost when the subshell terminated. For bash and similar shells, you activate an environment located at path envpath with the command:

source envpath/bin/activate

Users of other shells are accommodated with activate.csh and activate.fish scripts located in the same directory. On Windows systems, use activate.bat:

envpath/Scripts/activate.bat 

Activation does several things, most importantly:

• Adds the virtualenv’s bin directory at the beginning of the shell’s PATH environment variable, so its commands get run in preference to anything of the same name already on the PATH

• Defines a deactivate command to remove all effects of activation and return the Python environment to its former state

• Modifies the shell prompt to include the virtualenv’s name at the start

• Defines a VIRTUAL_ENV environment variable as the path to the virtualenv’s root directory (scripting can use this to introspect the virtualenv)

As a result of these actions, once a virtualenv is activated the python command runs the interpreter associated with that virtualenv. The interpreter sees the libraries (modules and packages) that have been installed in that environment, and pip—now the one from the virtualenv, since installing the module also installed the command in the virtualenv’s bin directory—by default installs new packages and modules in the environment’s site-packages directory.

Those new to virtualenvs should understand that a virtualenv is not tied to any project directory. It’s perfectly possible to work on several projects, each with its own source tree, using the same virtualenv. Activate it, then move around your filestore as necessary to accomplish your programming tasks, with the same libraries available (because the virtualenv determines the Python environment).

When you want to disable the virtualenv and stop using that set of resources, simply issue the command deactivate.

This undoes the changes made on activation, removing the virtualenv’s bin directory from your PATH, so the python command once again runs your usual interpreter. As long as you don’t delete it, the virtualenv remains available for future use by repeating the invocation to activate it.

Managing Dependency Requirements

Since virtualenvs were designed to complement installation with pip, it should come as no surprise that pip is the preferred way to maintain dependencies in a virtualenv. Because pip is already extensively documented, we mention only enough here to demonstrate its advantages in virtual environments. Having created a virtualenv, activated it, and installed dependencies, you can use the pip freeze command to learn the exact versions of those dependencies:

(tempenv) machine:~ user$ pip freeze
appnope==0.1.0
decorator==4.0.10
ipython==5.1.0
ipython-genutils==0.1.0
pexpect==4.2.1
pickleshare==0.7.4
prompt-toolkit==1.0.8
ptyprocess==0.5.1
Pygments==2.1.3
requests==2.11.1
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.1
wcwidth==0.1.7

If you redirect the output of this command to a file called filename, you can re-create the same set of dependencies in a different virtualenv with the command pip install -r filename.

To distribute code for use by others, Python developers conventionally include a requirements.txt file listing the necessary dependencies. When you are installing software from the Python Package Index, pip installs the packages you request along with any indicated dependencies. When you’re developing software it’s convenient to have a requirements file, as you can use it to add the necessary dependencies to the active virtualenv (unless they are already installed) with a simple pip install -r requirements.txt.

To maintain the same set of dependencies in several virtualenvs, use the same requirements file to add dependencies to each one. This is a convenient way to develop projects to run on multiple Python versions: create virtualenvs based on each of your required versions, then install from the same requirements file in each. While the preceding example uses exactly versioned dependency specifications as produced by pip freeze, in practice you can specify dependencies and constrain version requirements in quite complex ways.

Best Practices with virtualenvs

There is remarkably little advice on how best to manage your work with virtualenvs, though there are several sound tutorials: any good search engine gives you access to the most current ones. We can, however, offer a modest amount of advice that we hope will help you to get the most out of them.

When you are working with the same dependencies in multiple Python versions, it is useful to indicate the version in the environment name and use a common prefix. So for project mutex you might maintain environments called mutex_35 and mutex_27 for v3 and v2 development. When it’s obvious which Python is involved (and remember you see the environment name in your shell prompt), there’s less chance of testing with the wrong version. You maintain dependencies using common requirements to control resource installation in both.

Keep the requirements file(s) under source control, not the whole environment. Given the requirements file it’s easy to re-create a virtualenv, which depends only on the Python release and the requirements. You distribute your project, and let your consumers decide which version(s) of Python to run it on and create the appropriate (hopefully virtual) environment(s).

Keep your virtualenvs outside your project directories. This avoids the need to explicitly force source code control systems to ignore them. It really doesn’t matter where you store them—the virtualenvwrapper system keeps them all in a central location.

Your Python environment is independent of your process’s location in the filesystem. You can activate a virtual environment and then switch branches and move around a change-controlled source tree to use it wherever convenient.

To investigate a new module or package, create and activate a new virtualenv and then pip install the resources that interest you. You can play with this new environment to your heart’s content, confident in the knowledge that you won’t be installing rogue dependencies into other projects.

You may find that experiments in a virtualenv require installation of resources that aren’t currently project requirements. Rather than pollute your development environment, fork it: create a new virtualenv from the same requirements plus the testing functionality. Later, to make these changes permanent, use change control to merge your source and requirements changes back in from the fork.

If you are so inclined, you can create virtual environments based on debug builds of Python, giving you access to a wealth of instrumentation information about the performance of your Python code (and, of course, the interpreter).

Developing your virtual environment itself requires change control, and their ease of creation helps here too. Suppose that you recently released version 4.3 of a module, and you want to test your code with new versions of two of its dependencies. You could, with sufficient skill, persuade pip to replace the existing copies of dependencies in your existing virtualenv.

It’s much easier, though, to branch your project, update the requirements, and create an entirely new virtual environment based on the updated requirements. You still have the original virtualenv intact, and you can switch between virtualenvs to investigate specific aspects of any migration issues that might arise. Once you have adjusted your code so that all tests pass with the updated dependencies, you check in your code and requirement changes, and merge into version 4.4 to complete the update, advising your colleagues that your code is now ready for the updated versions of the dependencies.

Virtual environments won’t solve all of a Python programmer’s problems. Tools can always be made more sophisticated, or more general. But, by golly, they work, and we should take all the advantage of those that we can.