Numexpr is a software package for the evaluation of numerical array expressions, which is also installed when you install pandas, and you may have seen it announced in the watermark of other recipes (tested with Numexpr 2.3.1). Numexpr tries to speed up calculations by avoiding the creation of temporary variables because reading the variables can be a potential bottleneck. The largest speedups are expected for arrays that can't fit in the CPU cache.
Numexpr splits large arrays into chunks, which fit in the cache, and it also uses multiple cores in parallel when possible. It has an evaluate()
function, which accepts simple expressions and evaluates them (refer to the documentation for the complete list of supported features).
import numexpr as ne import numpy as np
a = np.random.rand(1e6) b = np.random.rand(1e6)
%timeit 2 * a ** 3 + 3 * b ** 9 %timeit ne.evaluate("2 * a ** 3 +3 * b ** 9 ")
Refer to the following screenshot for the end result:
The code is in the speeding_numexpr.ipynb
file in this book's code bundle.
We generated random data that should not fit in a cache to avoid caching effects and because that is the best use case for Numexpr. The size of the cache differs from one machine to another, so if necessary use a larger or smaller size for the arrays. In the example, we put a string containing a simple arithmetic expression, although we could have used a slightly more complex expression. For more details, refer to the documentation. I tested the code with a machine that has eight cores. The speedup is larger than a factor of eight, so it's clearly due to Numexpr.