Importing modules

One of the catchphrases in the Python world is “batteries included.” This refers to the many TV commercials I saw as a child that would spend their first 29.5 seconds enticing us to buy their exciting, fun-looking, beautiful toys ... only to spend the final half second saying, “batteries not included”--meaning that it wasn’t enough to buy the product to enjoy it, we had to buy batteries as well.

“Batteries included” refers to the fact that when you download and install Python, you have everything you’re going to need to get your work done. This isn’t quite as true as used to be the case, and PyPI (the Python Package Index, described separately in this chapter) provides us with a huge collection of third-party Python modules that we can use to improve our products. But the fact remains that the standard library, meaning the stuff that comes with Python when we install it, includes a huge number of modules that we can use in our programs.

The most commonly used things in the standard library, such as lists and dicts, are built into the language, thanks to a namespace known as builtins. You don’t need to worry about importing things in the builtins module, thanks to the LEGB rule that I discussed back in chapter 6. But anything else in the standard library must be loaded into memory before it can be used.

We load such a module using the import statement. The simplest version of import looks like

import MODULENAME

For example, if I want to use the os module, then I’ll write

import os

Notice a couple of things about this statement:

First, it’s not a function; you don’t say import(os), but rather import os.

Second, we don’t import a filename. Rather, we indicate the variable that we want to define, rather than the file that should be loaded from the disk. So don’t try to import "os" or even import "os.py". Just as def defines a new variable that references a function, so too import defines a new variable that references a module.

When you import os, Python tries to find a file that matches the variable name you’re defining. It’ll typically look for os.py and os.pyc, where the former is the original source code and the latter is the byte-compiled version. (Python uses the filesystem’s timestamp to figure out which one is newer and creates a new byte-compiled version as necessary. So don’t worry about compiling!)

Python looks for matching files in a number of directories, visible to you in sys.path. This is a list of strings representing directories; Python will iterate over each directory name until it finds a matching module name. If more than one directory contains a module with the same name, then the first one Python encounters is loaded, and any subsequent modules will be completely ignored. This can often lead to confusion and conflicts, in my experience, so try to choose unusual and distinct names for your modules.

Now, import has a number of variations that are useful to know, and that you’ll probably see in existing code--as well as use in your own code. That said, the ultimate goal is the same: load a module, and define one or more module-related names in your namespace.

If you’re happy loading a module and using its name as a variable, then import MODULENAME is a great way to go. But sometimes, that name is too long. For that reason, you’ll want to give the module name an alias. You can do that with

import mymod as mm

When you use as, the name mymod will not be defined. However, the name mm will be defined. This is silly and unnecessary if your module name is going to be short. But if the name is long, or you’re going to be referring to it a lot, then you might well want to give it a shorter alias. A classic example is NumPy (https://numpy.org/), which sits at the core of all of Python’s scientific and numeric computing systems, including data science and machine learning. That module is typically imported with an alias of np:

import numpy as np

Once you’ve imported a module, all of the names that were defined in the file’s global scope are available as attributes, via the module object. For example, the os module defines sep, which indicates what string separates elements of a directory path. You can access that value as os.sep. But if you’re going to use it a lot, then it’s a bit of a pain to constantly say os.sep. Wouldn’t it be nice to just call it sep? You can’t do that, of course, because the name sep would be a variable, whereas os.sep is an attribute.

However, you can bridge the gap and get the attribute loaded by using the following syntax:

from os import sep

Note that this won’t define the os variable, but it will define the sep variable. You can use from .. import on more than one variable too:

from os import sep, path

Now, both sep and path will be defined as variables in your global scope.

Worried about one of these imported attributes clashing with an existing variable, method, or module name? Then you can use from .. import .. as:

from os import sep as s

There’s a final version that I often see, and that I generally advise people not to use. It looks like this:

from os import *

This will load the os module into memory, but (more importantly) will take all of the attributes from os and define them as global variables in the current namespace. Given that we generally want to avoid global variables unless necessary, I see it as a problem when we allow the module to decide what variables should be defined.

Note Not all names from a module will be imported with import *. Names starting with _ (underscore) will be ignored. Moreover, if the module defines a list of strings named __all__, only names specified in the module will be loaded with import *. However, from X import Y will always work, regardless of whether __all__ is defined.

At the end of the day, import makes functions, classes, and data available to you in your current namespace. Given the huge number of modules available, both in Python’s standard library and on PyPI, that puts a lot of potential power at your fingertips--and explains why so many Python programs start with several lines of import statements.

The Decimal version

If you’re actually doing calculations involving serious money, you should almost certainly not be using floats. Rather, you should use integers or the Decimal class, both of which are more accurate. (See chapter 1 for some more information on the inaccuracy of floats.) I wanted this exercise to concentrate on the creation of a module, and not the use of the Decimal class, so I didn’t require it.

Here’s how my solution would look using Decimal:

from decimal import Decimal
 
rates = {
    'Chico': Decimal('0.5'),
    'Groucho': Decimal('0.7'),
    'Harpo': Decimal('0.5'),
    'Zeppo': Decimal('0.4')
}
 
def time_percentage(hour):
    return hour / Decimal('24.0')
 
def calculate_tax(amount, state, hour):
    return float(amount + (amount * rates[state] * time_percentage(hour)))

Notice that this code uses Decimal on strings, rather than floats, to ensure maximum accuracy. We then return a floating-point number at the last possible moment. Also note that any Decimal value multiplied or divided by a number remains a Decimal, so we only need to make a conversion at the end.

Error checking the Pythonic way

Since a module will be used by many other programs, it’s important for it to not only be accurate, but also have decent error checking. In our particular case, for example, we would want to check that the hour is between 0 and 24.

Right now, someone who passes an invalid hour to our function will still get an answer, albeit a nonsensical one. A better solution would be to have the function raise an exception if the input is invalid. And while we could raise a built-in Python exception (e.g., ValueError), it’s generally a better idea to create your own exception class and raise it; for example

class HourTooLowError(Exception): pass
class HourTooHighError(Exception): pass
 
def calculate_tax(amount, state, hour):
    if hour < 0:
        raise HourTooLowError(f'Hour of {hour} is < 0')
 
    if hour >= 24:
        raise HourTooHighError(f'Hour of {hour} is >= 24')
 
    return amount + (amount * rates[state] * time_percentage(hour))

Adding such exceptions to your code is considered very Pythonic and helps to ensure that anyone using your module will not accidentally get a bad result.

Loading and reloading modules

When you use import to load a module, what happens? For example, if you say

import mymod

then Python looks for mymod.py in a number of directories, defined in a list of strings called sys.path. If Python encounters a file in one of those directories, it loads the file and stops searching in any other directories.

Note There are a number of ways to modify sys.path, including by setting the environment variable PYTHONPATH and creating files with a .pth suffix in your Python installation’s site-packages directory. For more information on setting sys.path, see the Python documentation, or read this helpful article: http://mng.bz/PAP9.

This means import normally does two distinct things: it loads the module and defines a new variable. But what happens if your program loads two modules, each of which in turn loads modules? For example, let’s say that your program imports both pandas and scipy, both of which load the numpy module. In such a case, Python will load the module the first time, but only define the variable the second time. import only loads a module once, but it will always define the variable that you’ve asked it to create.

This is done via a dict defined in sys called sys.modules. Its keys are the names of modules that have been loaded, and its values are the actual module objects. Thus, when we say import mymod, Python first checks to see if mymod is in sys.modules. If so, then it doesn’t search for or load the module. Rather, it just defines the name.

This is normally a great thing, in that there’s no reason to reload a module once the program has started running. But when you’re debugging a module within an interactive Python session, you want to be able to reload it repeatedly, preferably without exiting from the current Python session.

In such cases, you can use the reload function defined in the importlib module. It takes a module object as an argument, so the module must already have been defined and imported. And it’s the sort of thing that you’ll likely use all the time in development, and almost never in actual production.

Note In previous versions of Python, reload was a built-in function. As of Python 3, it’s in the importlib module, which you must import to use it.

Why do we check __name__?

One of the most famous lines in all of Python reads as follows:

if __name__ == '__main__':

What does this line do? How does it help? This line is the result of a couple different things happening when we load a module:

• First, when a module is loaded, its code is executed from the start of the file until the end. You’re not just defining things; any code in the file is actually executed. That means you can (in theory) invoke print or have for loops. In this case, we’re using if to make some code execute conditionally when it’s loaded.

• Second, the __name__ variable is either defined to be __main__, meaning that things are currently running in the initial, default, and top-level namespace provided by Python, or it’s defined to be the name of the current module. The if statement here is thus checking to see if the module was run directly, or if it was imported by another piece of Python code.

In other words, the line of code says, “Only execute the below code (i.e., inside of the if statement) if this is the top-level program being executed. Ignore the stuff in the if when we import this module.”

You can use this code in a few different ways:

• Many modules run their own tests when invoked directly, rather than imported.

• Some modules can be run interactively, providing user-facing functionality and an interface. This code allows that to happen, without interfering with any function definitions.

• In some odd cases, such as the multiprocessing module in Windows, the code allows you to differentiate between versions of the program that are being loaded and executed in separate processes.

While you can theoretically have as many if __name__ == '__main__' lines in your code as you want, it’s typical for this line to appear only once, at the end of your module file.

You’ll undoubtedly encounter this code, and might even have written it yourself in the past. And now you know how it works!

Modules vs. packages

This chapter is all about modules--how to create, import, and use them. But you might have noticed that we often use another term, package, to discuss Python code. What’s the difference between a module and a package?

A module is a single file, with a “.py” suffix. We can load the module using import, as we’ve seen. But what if your project is large enough that it would make more sense to have several separate files? How can you distribute those files together?

The answer is a package, which basically means a directory containing one or more Python modules. For example, assume you have the modules first.py, second.py, and third.py, and want to keep them together. You can put them all into a directory, mypackage. Assuming that directory is in sys.path, you can then say

from mypackage import first

Python will go into the mypackage directory, look for first.py, and import it. You can then access all of its attributes via first.x, first.y, and so forth.

Alternatively, you could say

import mypackage.first

In this case, Python will still load the module first, but it’ll be available in your program via the long name, mypackage.first. You can then use mypackage.first.x and mypackage.first.y.

Alternatively, you could say

import mypackage

But this will only be useful if, in the mypackage directory, you have a file named __init__.py. In such a case, importing mypackage effectively means that __init__.py is loaded, and thus executed. You can, inside of that file, import one or more of the modules within the package.

What about if you want to distribute your package to others? Then you’ll have to create a package. If this sounds strange, that you need a package to distribute your package, that’s because the same term, package, is used for two different concepts. A PyPI package, or distribution package, is a wrapper around a Python package containing information about the author, compatible versions, and licensing, as well as automated tests, dependencies, and installation instructions.

Even more confusing than the use of “package” to describe two different things is the fact that both the distribution package and the Python package are directories, and that they should have the same name. If your distribution package is called mypackage, you’ll have a directory called mypackage. Inside that directory, among other things, will be a subdirectory called mypackage, which is where the Python package goes.

Creating a distribution package means creating a file called setup.py (documented here: http://mng.bz/wB9q), and I must admit that for many years, I found this to be a real chore. It turns out that I wasn’t alone, and a number of Python developers have come up with ways to create distribution packages with relative ease. One that I’ve been using for a while is called “Poetry” (http://mng.bz/2Xzd), and makes the entire process easy and straightforward.

If you want to distribute packages via PyPI, you’ll need to register for a username and password at https://pypi.org/. Once you have that, here are the minimal steps you’ll need to take an existing package and upload it to PyPI with Poetry, using Unix shell commands:

$ poetry new mypackage                  ❶
$ cd mypackage                          ❷
$ cp -R ~/mypackage-code/* mypackage    ❸
$ poetry build                          ❹
$ poetry publish                        ❺

❶ Creates a new package skeleton called mypackage

❷ Moves into the top-level directory

❸ Copies the contents of the Python package into its subdirectory

❹ Creates the wheelfile and tar.gz versions of your package in the dist directory

❺ Publishes the package to PyPI; to confirm, you enter your username and password when prompted

Note that you can’t upload the specific name mypackage to PyPI. I suggest prefacing your package name with your username or initials, unless you intend to publish it for public consumption.

You could add plenty of other steps to the ones I’ve listed--for example, you can (and should) edit the pyproject.toml configuration file, in which you describe your package’s version, license, and dependencies. But creating a distribution package is no longer difficult. Rather, the hard part will be deciding what code you want to share with the community.