In addition to the list/dict/set comprehensions, Python also has a few (more advanced) functions that can be really convenient when coding functionally. The functools library is a collection of functions that return callable objects. Some of these functions are used as decorators (we'll cover more about that in Chapter 5, Decorators – Enabling Code Reuse by Decorating), but the ones that we are going to talk about are used as straight-up functions to make your life easier.
The partial function is really convenient for adding some default arguments to a function that you use often but can't (or don't want to) redefine. With object-oriented code, you can usually work around cases similar to these, but with procedural code, you will often have to repeat your arguments. Let's take the heapq functions from Chapter 3, Containers and Collections – Storing Data the Right Way, as an example:
>>> import heapq >>> heap = [] >>> heapq.heappush(heap, 1) >>> heapq.heappush(heap, 3) >>> heapq.heappush(heap, 5) >>> heapq.heappush(heap, 2) >>> heapq.heappush(heap, 4) >>> heapq.nsmallest(3, heap) [1, 2, 3]
Almost all the heapq functions require a heap argument, so why not make a shortcut for it? This is where functools.partial comes in:
>>> import functools >>> import heapq >>> heap = [] >>> push = functools.partial(heapq.heappush, heap) >>> smallest = functools.partial(heapq.nsmallest, iterable=heap) >>> push(1) >>> push(3) >>> push(5) >>> push(2) >>> push(4) >>> smallest(3) [1, 2, 3]
Seems a bit cleaner, right? In this case, both versions are fairly short and readable, but it's a convenient function to have.
Why should we use partial instead of writing a lambda argument? Well, it's mostly about convenience, but it also helps solve the late binding problem discussed in Chapter 2, Pythonic Syntax, Common Pitfalls, and Style Guide. Additionally, partial functions can be pickled whereas lambda statements cannot.
The reduce function implements a mathematical technique called fold. It basically applies a function to the first and second elements, uses that result to apply together with the third element, and continues until the list is exhausted.
The reduce function is supported by many languages but in most cases using different names such as curry, fold, accumulate, or aggregate. Python has actually supported reduce for a very long time, but since Python 3, it has been moved from the global scope to the functools library. Some code can be simplified beautifully using the reduce statement; whether it's readable or not is debatable, however.
One of the most used examples of reduce is for calculating factorials, which is indeed quite simple:
>>> import operator >>> import functools >>> functools.reduce(operator.mul, range(1, 6)) 120
Internally, the reduce function will do the following:
>>> import operator >>> f = operator.mul >>> f(f(f(f(1, 2), 3), 4), 5) 120
To clarify this further, let's look at it like this:
>>> iterable = range(1, 6) >>> import operator # The initial values: >>> a, b, *iterable = iterable >>> a, b, iterable (1, 2, [3, 4, 5]) # First run >>> a = operator.mul(a, b) >>> b, *iterable = iterable >>> a, b, iterable (2, 3, [4, 5]) # Second run >>> a = operator.mul(a, b) >>> b, *iterable = iterable >>> a, b, iterable (6, 4, [5]) # Third run >>> a = operator.mul(a, b) >>> b, *iterable = iterable >>> a, b, iterable (24, 5, []) # Fourth and last run >>> a = operator.mul (a, b) >>> a 120
Or with a simple while loop using the deque collection:
>>> import operator >>> import collections >>> iterable = collections.deque(range(1, 6)) >>> value = iterable.popleft() >>> while iterable: ... value = operator.mul(value, iterable.popleft()) >>> value 120
Trees are a case where the reduce function really shines. Remember the one-line tree definition using a defaultdict from Chapter 3, Containers and Collections – Storing Data the Right Way? What would be a good way to access the keys inside of that object? Given a path of a tree item, we can use reduce to easily access the items inside:
>>> import json >>> import functools >>> import collections >>> def tree(): ... return collections.defaultdict(tree) # Build the tree: >>> taxonomy = tree() >>> reptilia = taxonomy['Chordata']['Vertebrata']['Reptilia'] >>> reptilia['Squamata']['Serpentes']['Pythonidae'] = [ ... 'Liasis', 'Morelia', 'Python'] # The actual contents of the tree >>> print(json.dumps(taxonomy, indent=4)) { "Chordata": { "Vertebrata": { "Reptilia": { "Squamata": { "Serpentes": { "Pythonidae": [ "Liasis", "Morelia", "Python" ] } } } } } } # The path we wish to get >>> path = 'Chordata.Vertebrata.Reptilia.Squamata.Serpentes' # Split the path for easier access >>> path = path.split('.') # Now fetch the path using reduce to recursively fetch the items >>> family = functools.reduce(lambda a, b: a[b], path, taxonomy) >>> family.items() dict_items([('Pythonidae', ['Liasis', 'Morelia', 'Python'])]) # The path we wish to get >>> path = 'Chordata.Vertebrata.Reptilia.Squamata'.split('.') >>> suborder = functools.reduce(lambda a, b: a[b], path, taxonomy) >>> suborder.keys() dict_keys(['Serpentes'])
And lastly, some people might be wondering why Python only has fold_left and no fold_right. In my opinion, you don't really need both of them as you can easily reverse the operation.
The regular reduce—the fold left operation:
fold_left = functools.reduce(
lambda x, y: function(x, y),
iterable,
initializer,
)The reverse—the fold right operation:
fold_right = functools.reduce(
lambda x, y: function(y, x),
reversed(iterable),
initializer,
)While this one is definitely very useful in purely functional languages—where these operations are used quite often—initially there were plans to remove the reduce function from Python with the introduction of Python 3. Luckily, that plan was modified, and instead of being removed, it has been moved from reduce to functools.reduce. There may not be many useful cases for reduce, but there are some cool use cases. Especially traversing recursive data structures is far more easily done using reduce, since it would otherwise involve more complicated loops or recursive functions.