The concurrent.futures
module is a Python module with which we can execute callables asynchronously. If you are familiar with Java and go through the module, you will notice some similarities with the equivalent Java API, such as class names and architecture. According to the Python documentation, this is not a coincidence.
A task in this context is an autonomous unit of work. For instance, printing a document can be considered a task, but usually we consider much smaller tasks, such as adding two numbers.
import dautil as dl import ch12util from functools import partial import matplotlib.pyplot as plt import numpy as np from scipy.stats import skew import concurrent.futures from IPython.display import HTML STATS = []
def resample(arr): sample = ch12util.bootstrap(arr) STATS.append((sample.mean(), sample.std(), skew(sample)))
class Bootstrapper(): def __init__(self, data): self.data = data self.log = dl.log_api.conf_logger(__name__) def run(self, index): if index % 10 == 0: self.log.debug('Bootstrap {}'.format( index)) resample(self.data)
def serial(arr, n): for i in range(n): resample(arr)
def parallel(arr, n): executor = concurrent.futures.ThreadPoolExecutor(max_workers=8) bootstrapper = Bootstrapper(arr) for x in executor.map(bootstrapper.run, range(n)): pass executor.shutdown()
rain = dl.data.Weather.load()['RAIN'].dropna().values np.random.seed(33) parallel_times = ch12util.time_many(partial(parallel, rain)) serial_times = ch12util.time_many(partial(serial, rain)) sp = dl.plotting.Subplotter(2, 2, context) ch12util.plot_times(sp.ax, serial_times, parallel_times) STATS = np.array(STATS) ch12util.plot_distro(sp.next_ax(), STATS.T[0], rain.mean()) sp.label() ch12util.plot_distro(sp.next_ax(), STATS.T[1], rain.std()) sp.label() ch12util.plot_distro(sp.next_ax(), STATS.T[2], skew(rain)) sp.label() HTML(sp.exit())
Refer to the following screenshot for the end result:
The code is in the launching_futures.ipynb
file in this book's code bundle.
concurrent.futures
module at https://docs.python.org/3/library/concurrent.futures.html (retrieved January 2016)