Functions are one of the cornerstones of programming--but not because there’s a technical need for them. We could program without functions, if we really had to. But functions provide a number of great benefits.

First, they allow us to avoid repetition in our code. Many programs have instructions that are repeated: asking a user to log in, reading data from a particular type of configuration file, or calculating the length of an MP3, for example. While the computer won’t mind (or even complain) if the same code appears in multiple places, we--and the people who have to maintain the code after we’re done with it--will suffer and likely complain. Such repetition is hard to remember and keep track of. Moreover, you’ll likely find that the code needs improvement and maintenance; if it occurs multiple times in your program, then you’ll need to find and fix it each of those times.

As mentioned in chapter 2, the maxim “don’t repeat yourself” (DRY) is a good thing to keep in mind when programming. And writing functions is a great way to apply the phrase, “DRY up your code.”

A second benefit of functions is that they let us (as developers) think at a higher level of abstraction. Just as you can’t drive if you’re constantly thinking about what your car’s various parts are doing, you can’t program if you’re constantly thinking about all of the parts of your program and what they’re doing. It helps, semantically and cognitively, to wrap functionality into a named package, and then to use that name to refer to it.

In natural language, we create new verbs all of the time, such as programming and texting. We don’t have to do this; we could describe these actions using many more words, and with much more detail. But doing so becomes tedious and draws attention away from the point that we’re making. Functions are the verbs of programming; they let us define new actions based on old ones, and thus let us think in more sophisticated terms.

For all of these reasons, functions are a useful tool and are available in all programming languages. But Python’s functions add a twist to this: they’re objects, meaning that they can be treated as data. We can store functions in data structures and retrieve them from there as well. Using functions in this way seems odd to many newcomers to Python, but it provides a powerful technique that can reduce how much code we write and increase our flexibility.

Moreover, Python doesn’t allow for multiple definitions of the same function. In some languages, you can define a function multiple times, each time having a different signature. So you could, for example, define the function once as taking a single string argument, a second time as taking a list argument, a third time as taking a dict argument, and a fourth time as taking three float arguments.

In Python, this functionality doesn’t exist; when you define a function, you’re assigning to a variable. And just as you can’t expect that x will simultaneously contain the values 5 and 7, you similarly can’t expect that a function will contain multiple implementations.

The way that we get around this problem in Python is with flexible parameters. Between default values, variable numbers of arguments (*args), and keyword arguments (**kwargs), we can write functions that handle a variety of situations.

You’ve already written a number of functions as you’ve progressed through this book, so the purpose of this chapter isn’t to teach you how to write functions. Rather, the goal is to show you how to use various function-related techniques. This will allow you not only to write code once and use it numerous times, but also to build up a hierarchy of new verbs, describing increasingly complex and higher level tasks.

Table 6.1 What you need to know

def hello(name):
    return f'Hello, {name}!'

>>> hello('world')
'Hello, world!'

>>> hello()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module> TypeError: hello() missing 1 required positional argument: 'name'

>>> hello.__code__.co_argcount
1

def hello(name='world'):
    return f'Hello, {name}!'

Exercise 25 ■ XML generator

Python is often used not just to parse data, but to format it as well. In this exercise, you’ll write a function that uses a combination of different parameters and parameter types to produce a variety of outputs.

Write a function, myxml, that allows you to create simple XML output. The output from the function will always be a string. The function can be invoked in a number of ways, as shown in table 6.3.

Table 6.3 Calling myxml

Notice that in all cases, the first argument is the name of the tag. In the latter two cases, the second argument is the content (text) placed between the opening and closing tags. And in the third case, the name-value pairs will be turned into attributes inside of the opening tag.

Working it out

Let’s start by assuming that we only want our function to take a single argument, the name of the tag. That would be easy to write. We could say

def myxml(tagname):
    return f'<{tagname}></{tagname}>'

If we decide we want to pass a second (optional) argument, this will fail. Some people thus assume that our function should take *args, meaning any number of arguments, all of which will be put in a tuple. But, as a general rule, *args is meant for situations in which you don’t know how many values you’ll be getting and you want to be able to accept any number.

My general rule with *args is that it should be used when you’ll put its value into a for loop, and that if you’re grabbing elements from *args with numeric indexes, then you’re probably doing something wrong.

The other option, though, is to use a default. And that’s what I’ve gone with. The first parameter is mandatory, but the second is optional. If I make the second one (which I call content here) an empty string, then I know that either the user passes content or the content is empty. In either case, the function works. I can thus define it as follows:

def myxml(tagname, content=''):
    return f'<{tagname}>{content}</{tagname}>'

But what about the key-value pairs that we can pass, and which are then placed as attributes in the opening tag?

When we define a function with **kwargs, we’re telling Python that we might pass any name-value pair in the style name=value. These arguments aren’t passed in the normal way but are treated separately, as keyword arguments. They’re used to create a dict, traditionally called kwargs, whose keys are the keyword names and whose values are the keyword values. Thus, we can say

def myxml(tagname, content='', **kwargs):
    attrs = ''.join([f' {key}="{value}"'
                     for key, value in kwargs.items()])
    return f'<{tagname}{attrs}>{content}</{tagname}>'

As you can see, I’m not just taking the key-value pairs from **kwargs and putting them into a string. I first have to take that dict and turn it into name-value pairs in XML format. I do this with a list comprehension, running on the dict. For each key-value pair, I create a string, making sure that the first character in the string is a space, so we don’t bump up against the tagname in the opening tag.

There’s a lot going on in this code, and it uses a few common Python paradigms. Understanding that, it’s probably useful to go through it, step by step, just to make things clearer:

1. In the body of myxml, we know that tagname will be a string (the name of the tag), content will be a string (whatever content should go between the tags), and kwargs will be a dict (with the attribute name-value pairs).

2. Both content and kwargs might be empty, if the user didn’t pass any values for those parameters.

3. We use a list comprehension to iterate over kwargs.items(). This will provide us with one key-value pair in each iteration.

4. We use the key-value pair, assigned to the variables key and value, to create a string of the form key="value". We get one such string for each of the attribute key-value pairs passed by the user.

5. The result of our list comprehension is a list of strings. We join these strings together with str.join, with an empty string between the elements.

6. Finally, we return the combination of the opening tag (with any attributes we might have gotten), the content, and the closing tag.

Solution

def myxml(tagname, content='', **kwargs):                  ❶
    attrs = ''.join([f' {key}="{value}"'
                     for key, value in kwargs.items()])    ❷
    return f'<{tagname}{attrs}>{content}</{tagname}>'      ❸
 
print(myxml('tagname', 'hello', a=1, b=2, c=3))

❶ The function has one mandatory parameter, one with a default, and “**kwargs”.

❷ Uses a list comprehension to create a string from kwargs

❸ Returns the XML-formatted string

You can work through a version of this code in the Python Tutor at http://mng.bz/ OMoK.

Screencast solution

Watch this short video walkthrough of the solution: https://livebook.manning.com/ video/python-workout.

Beyond the exercise

Learning to work with functions, and the types of parameters that you can define, takes some time but is well worthwhile. Here are some exercises you can use to sharpen your thinking when it comes to function parameters:

• Write a copyfile function that takes one mandatory argument--the name of an input file--and any number of additional arguments: the names of files to which the input should be copied. Calling copyfile('myfile.txt', 'copy1 .txt', 'copy2.txt', 'copy3.txt') will create three copies of myfile.txt: one each in copy1.txt, copy2.txt, and copy3.txt.

• Write a “factorial” function that takes any number of numeric arguments and returns the result of multiplying them all by one another.

• Write an anyjoin function that works similarly to str.join, except that the first argument is a sequence of any types (not just of strings), and the second argument is the “glue” that we put between elements, defaulting to " " (a space). So anyjoin([1,2,3]) will return 1 2 3, and anyjoin('abc', pass:'**') will return pass:a**b**c.

sum = 0
for i in range(5):
    sum += i
print(sum)
 
print(sum([10, 20, 30]))
 
TypeError: 'int' object is not callable

x = 100
 
def foo():
    x = 200
    print(x)
 
print(x)
foo()
print(x)

x = 100

def foo():
    global x
    x = 200
    print(x)

print(x)
foo()
print(x)

def foo(x):
    def bar(y):
        return x * y
    return bar

f = foo(10)
print(f(20))

def foo():
    call_counter = 0                                     ❶
    def bar(y):
        nonlocal call_counter                            ❷
        call_counter += 1                                ❸
        return f'y = {y}, call_counter = {call_counter}'
    return bar

b = foo()
for i in range(10, 100, 10):                             ❹
    print(b(i))                                          ❺

y = 10, call_counter = 1
y = 20, call_counter = 2
y = 30, call_counter = 3
y = 40, call_counter = 4
y = 50, call_counter = 5
y = 60, call_counter = 6
y = 70, call_counter = 7
y = 80, call_counter = 8
y = 90, call_counter = 9

Exercise 26 ■ Prefix notation calculator

In Python, as in real life, we normally write mathematics using infix notation, as in 2+3. But there’s also something known as prefix notation, in which the operator precedes the arguments. Using prefix notation, we would write + 2 3. There’s also postfix notation, sometimes known as “reverse Polish notation” (or RPN), which is still in use on HP brand calculators. That would look like 2 3 +. And yes, the numbers must then be separated by spaces.

Prefix and postfix notation are both useful in that they allow us to do sophisticated operations without parentheses. For example, if you write 2 3 4 + * in RPN, you’re telling the system to first add 3+4 and then multiply 2*7. This is why HP calculators have an Enter key but no “=” key, which confuses newcomers greatly. In the Lisp programming language, prefix notation allows you to apply an operator to many numbers (e.g., (+ 1 2 3 4 5)) rather than get caught up with lots of + signs.

For this exercise, I want you to write a function (calc) that expects a single argument--a string containing a simple math expression in prefix notation--with an operator and two numbers. Your program will parse the input and produce the appropriate output. For our purposes, it’s enough to handle the six basic arithmetic operations in Python: addition, subtraction, multiplication, division (/), modulus (%), and exponentiation (**). The normal Python math rules should work, such that division always results in a floating-point number. We’ll assume, for our purposes, that the argument will only contain one of our six operators and two valid numbers.

But wait, there’s a catch--or a hint, if you prefer: you should implement each of the operations as a separate function, and you shouldn’t use an if statement to decide which function should be run. Another hint: look at the operator module, whose functions implement many of Python’s operators.

Working it out

The solution uses a technique known as a dispatch table, along with the operator module that comes with Python. It’s my favorite solution to this problem, but it’s not the only one--and it’s likely not the one that you first thought of.

Let’s start with the simplest solution and work our way up to the solution I wrote. We’ll need a function for each of the operators. But then we’ll somehow need to translate from the operator string (e.g., + or **) to the function we want to run. We could use if statements to make such a decision, but a more common way to do this in Python is with dicts. After all, it’s pretty standard to have keys that are strings, and since we can store anything in the value, that includes functions.

Note Many of my students ask me how to create a switch-case statement in Python. They’re surprised to hear that they already know the answer, namely that Python doesn’t have such a statement, and that we use if instead. This is part of Python’s philosophy of having one, and only one, way to do something. It reduces programmers’ choices but makes the code clearer and easier to maintain.

We can then retrieve the function from the dict and invoke it with parentheses:

def add(a,b):
    return a + b
 
def sub(a,b):
    return a - b
 
def mul(a,b):
    return a * b
 
def div(a,b):
    return a / b

def pow(a,b):
    return a ** b

def mod(a,b):
    return a % b
 
def calc(to_solve):
    operations = {'+' : add,                    ❶
                  '-' : sub,
                  '*' : mul,
                  '/' : div,
                  '**' : pow,
                  '%' : mod}
 
    op, first_s, second_s = to_solve.split()    ❷
    first = int(first_s)                        ❸
    second = int(second_s)
 
    return operations[op](first, second)        ❹

❶ The keys in the operations dict are the operator strings that a user might enter, while the values are our functions associated with those strings.

❷ Breaks the user’s input apart

❸ Turns each of the user’s inputs from strings into integers

❹ Applies the user’s chosen operator as a key in operations, returning a function--which we then invoke, passing it “first” and “second” as arguments

Perhaps my favorite part of the code is the final line. We have a dict in which the functions are the values. We can thus retrieve the function we want with operations [operator], where operator is the first part of the string that we broke apart with str.split. Once we have a function, we can call it with parentheses, passing it our two operands, first and second.

But how do we get first and second? From the user’s input string, in which we assume that there are three elements. We use str.split to break them apart, and immediately use unpacking to assign them to three variables.

>>> s = 'a b c d e'
>>> s.split()
['a', 'b', 'c', 'd', 'e']

>>> s = 'a b c d e'
>>> s.split(maxsplit=3)
['a', 'b', 'c', 'd e']

All of this is fine, but this code doesn’t seem very DRY. The fact that we have to define each of our functions, even when they’re so similar to one another and are reimplementing existing functionality, is a bit frustrating and out of character for Python.

Fortunately, the operator module, which comes with Python, can help us. By importing operator, we get precisely the functions we need: add, sub, mul, truediv/ floordiv, mod, and pow. We no longer need to define our own functions, because we can use the ones that the module provides. The add function in operators does what we would normally expect from the + operator: it looks to its left, determines the type of the first parameter, and uses that to know what to invoke. operator.add, as a function, doesn’t need to look to its left; it checks the type of its first argument and uses that to determine which version of + to run.

In this particular exercise, we restricted the user’s inputs to integers, so we didn’t do any type checking. But you can imagine a version of this exercise in which we could handle a variety of different types, not just integers. In such a case, the various operator functions would know what to do with whatever types we’d hand them.

Solution

import operator                                 ❶
 
def calc(to_solve):
    operations = {'+': operator.add,            ❷
                  '-': operator.sub,
                  '*': operator.mul,
                  '/': operator.truediv,        ❸
                  '**': operator.pow,
                  '%': operator.mod}
 
    op, first_s, second_s = to_solve.split()    ❹
    first = int(first_s)
    second = int(second_s)
 
    return operations[op](first, second)        ❺
 
print(calc('+ 2 3'))

❶ The operator module provides functions that implement all built-in operators.

❷ Yes, functions can be the values in a dict!

❸ You can choose between truediv, which returns a float, as with the “/ ” operator, or floordiv, which returns an integer, as with the “// ” operator.

❹ Splits the line, assigning via unpacking

❺ Calls the function retrieved via operator, passing “first” and “second” as arguments

You can work through a version of this code in the Python Tutor at http://mng.bz/ YrGo.

Screencast solution

Watch this short video walkthrough of the solution: https://livebook.manning.com/ video/python-workout.

Beyond the exercise

Treating functions as data, and storing them in data structures, is odd for many newcomers to Python. But it enables techniques that, although possible, are far more complex in other languages. Here are three more exercises that extend this idea even further:

• Expand the program you wrote, such that the user’s input can contain any number of numbers, not just two. The program will thus handle + 3 5 7 or / 100 5 5, and will apply the operator from left to right--giving the answers 15 and 4, respectively.

• Write a function, apply_to_each, that takes two arguments: a function that takes a single argument, and an iterable. Return a list whose values are the result of applying the function to each element in the iterable. (If this sounds familiar, it might be--this is an implementation of the classic map function, still available in Python. You can find a description of map in chapter 7.)

• Write a function, transform_lines, that takes three arguments: a function that takes a single argument, the name of an input file, and the name of an output file. Calling the function will run the function on each line of the input file, with the results written to the output file. (Hint: the previous exercise and this one are closely related.)

Exercise 27 ■ Password generator

Even today, many people use the same password on many different computers. This means that if someone figures out your password on system A, then they can log into systems B, C, and D where you used the same password. For this reason, many people (including me) use software that creates (and then remembers) long, randomly generated passwords. If you use such a system, then even if system A is compromised, your logins on systems B, C, and D are all safe.

In this exercise, we’re going to create a password-generation function. Actually, we’re going to create a factory for password-generation functions. That is, I might need to generate a large number of passwords, all of which use the same set of characters. (You know how it is. Some applications require a mix of capital letters, lowercase letters, numbers, and symbols; whereas others require that you only use letters; and still others allow both letters and digits.) You’ll thus call the function create_password _generator with a string. That string will return a function, which itself takes an integer argument. Calling this function will return a password of the specified length, using the string from which it was created; for example

alpha_password = create_password_generator('abcdef')
symbol_password = create_password_generator('!@#$%')
 
print(alpha_password(5))    # efeaa
print(alpha_password(10))   # cacdacbada
 
print(symbol_password(5))   # %#@%@
print(symbol_password(10))  # @!%%$%$%%#

A useful function to know about in implementing this function is the random module (http://mng.bz/Z2wj), and more specifically the random.choice function in that module. That function returns one (randomly chosen) element from a sequence.

The point of this exercise is to understand how to work with inner functions: defining them, returning them, and using them to create numerous similar functions.

Working it out

This is an example of where you might want to use an inner function, sometimes known as a closure. The idea is that we’re invoking a function (create_password _generator) that returns a function (create_password). The returned, inner function knows what we did on our initial invocation but also has some functionality of its own. As a result, it needs to be defined as an inner function so that it can access variables from the initial (outer) invocation.

The inner function is defined not when Python first executes the program, but rather when the outer function (create_password_generator) is executed. Indeed, we create a new inner function once for each time that create_password_generator is invoked.

That new inner function is then returned to the caller. From Python’s perspective, there’s nothing special here--we can return any Python object from a function: a list, dict, or even a function. What is special here, though, is that the returned function references a variable in the outer function, where it was originally defined.

After all, we want to end up with a function to which we can pass an integer, and from which we can get a randomly generated password. But the password must contain certain characters, and different programs have different restrictions on what characters can be used for those passwords. Thus, we might want five alphanumeric characters, or 10 numbers, or 15 characters that are either alphanumeric or punctuation.

We thus define our outer function such that it takes a single argument, a string containing the characters from which we want to create a new password. The result of invoking this function is, as was indicated, a function--the dynamically defined create _password. This inner function has access to the original characters variable in the outer function because of Python’s LEGB precedence rule for variable lookup. (See sidebar, “Variable scoping in Python.”) When, inside of create_password, we look for the variable characters, it’s found in the enclosing function’s scope.

Figure 6.3 Python Tutor’s depiction of two password-generating functions

If we invoke create_password_generator twice, as shown in the visualization via the Python Tutor (figure 6.3), each invocation will return a separate version of create_password, with a separate value of characters. Each invocation of the outer function returns a new function, with its own local variables. At the same time, each of the returned inner functions has access to the local variables from its enclosing function. When we invoke one of the inner functions, we thus get a new password based on the combination of the inner function’s local variables and the outer (enclosing) function’s local variables.

Note Working with inner functions and closures can be quite surprising and confusing at first. That’s particularly true because our instinct is to believe that when a function returns, its local variables and state all go away. Indeed, that’s normally true--but remember that in Python, an object isn’t released and garbage-collected if there’s at least one reference to it. And if the inner function is still referring to the stack frame in which it was defined, then the outer function will stick around as long as the inner function exists.

Solution

import random
 
def create_password_generator(characters):               ❶
    def create_password(length):                         ❷
        output = []

        for i in range(length):                          ❸
            output.append(random.choice(characters))     ❹
        return ''.join(output)                           ❺
    return create_password                               ❻
 
alpha_password = create_password_generator('abcdef')
symbol_password = create_password_generator('!@#$%')
 
print(alpha_password(5))
print(alpha_password(10))
 
print(symbol_password(5))
print(symbol_password(10))

❶ Defines the outer function

❷ Defines the inner function, with def running each time we run the outer function

❸ How long do we want the password to be?

❹ Adds a new, random element from characters to output

❺ Returns a string based on the elements of output

❻ Returns the inner function to the caller

You can work through a version of this code in the Python Tutor at http://mng.bz/ GVEM.

Screencast solution

Watch this short video walkthrough of the solution: https://livebook.manning.com/ video/python-workout.

Beyond the exercise

Thinking of functions as data lets you work at even higher levels of abstraction than usual functions, and thus solve even higher level problems without worrying about the low-level details. However, it can take some time to internalize and understand how to pass functions as arguments to other functions, or to return functions from inside other functions. Here are some additional exercises you can try to better understand and work with them:

• Now that you’ve written a function to create passwords, write create_password_checker, which checks that a given password meets the IT staff’s acceptability criteria. In other words, create a function with four parameters: min_ uppercase, min_lowercase, min_punctuation, and min_digits. These represent the minimum number of uppercase letters, lowercase letters, punctuations, and digits for an acceptable password. The output from create_password_ checker is a function that takes a potential password (string) as its input and returns a Boolean value indicating whether the string is an acceptable password.

• Write a function, getitem, that takes a single argument and returns a function f. The returned f can then be invoked on any data structure whose elements can be selected via square brackets, and then returns that item. So if I invoke f = getitem('a'), and if I have a dict d = {'a':1, 'b':2}, then f(d) will return 1. (This is very similar to operator.itemgetter, a very useful function in many circumstances.)

• Write a function, doboth, that takes two functions as arguments (f1 and f2) and returns a single function, g. Invoking g(x) should return the same result as invoking f2(f1(x)).

Summary

Writing simple Python functions isn’t hard. But where Python’s functions really shine is in their flexibility--especially when it comes to parameter interpretation--and in the fact that functions are data too. In this chapter, we explored all of these ideas, which should give you some thoughts about how to take advantage of functions in your own programs.

If you ever find yourself writing similar code multiple times, you should seriously consider generalizing it into a function that you can call from those locations. Moreover, if you find yourself implementing something that you might want to use in the future, implement it as a function. Besides, it’s often easier to understand, maintain, and test code that has been broken into functions, so even if you aren’t worried about reuse or higher levels of abstraction, it might still be beneficial to write your code as functions.