Using concurrent.futures thread pools

The concurrent.futures module offers a second kind of executor that we can use in our applications. Instead of creating a concurrent.futures.ProcessPoolExecutor object, we can use the ThreadPoolExecutor object. This will create a pool of threads within a single process.

The syntax for thread pools is almost identical to using a ProcessPoolExecutor object. The performance, however, is remarkably different. In this example of log file analysis, the work is dominated by I/O. Because all of the threads in a process share the same OS scheduling constraints, the overall performance of multithreaded log file analysis is about the same as processing the log files serially.

Using sample log files and a small four-core laptop running macOS X, these are the kinds of results that indicate the difference between threads that share I/O resources and processes:

  • Using the concurrent.futures thread pool, the elapsed time was 168 seconds
  • Using a process pool, the elapsed time was 68 seconds

In both cases, the Pool object's size was 4. The single-process and single-thread baseline time was 150 seconds; adding threads made processing run more slowly. This result is typical of programs doing a great deal of input and output. Multithreading may be more appropriate for user interfaces where threads are idle for long periods of time, or where waiting for the person to move the mouse or touch the screen.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset