A program that does a great deal of calculation and relatively little I/O will not see much benefit from concurrent processing. If a calculation has a budget of 28 minutes of computation, then interleaving the operations in different ways won't have a dramatic impact. Using eight cores may cut the time by approximately one-eighth. The actual time savings depend on the OS and language overheads, which are difficult to predict. Introducing concurrency will not have the kind of performance impact that a better algorithm will have.

When a calculation involves a great deal of I/O, then interleaving CPU processing with I/O requests will dramatically improve performance. The idea is to do computations on some pieces of data while waiting for the OS to complete the I/O of other pieces of data. Because I/O generally involves a great deal of waiting, an eight-core processor can interleave the work from dozens of concurrent I/O requests.

We have two approaches to interleaving computation and I/O. They are as follows:

• We can create a pipeline of processing stages. An individual item must move through all of the stages where it is read, filtered, computed, aggregated, and written. The idea of multiple concurrent stages is to have distinct data objects in each stage. Time slicing among the stages will allow computation and I/O to be interleaved.
• We can create a pool of concurrent workers, each of which we perform all of the processing for a data item. The data items are assigned to workers in the pool and the results are collected from the workers.

The differences between these approaches aren't crisp; there is a blurry middle region that's clearly not one or the other. It's common to create a hybrid mixture where one stage of a pipeline involves a pool of workers to make that stage as fast as the other stages. There are some formalisms that make it somewhat easier to design concurrent programs. The Communicating Sequential Processes (CSP) paradigm can help design message-passing applications. Packages such as pycsp can be used to add CSP formalisms to Python.

I/O-intensive programs often gain the most dramatic benefits from concurrent processing. The idea is to interleave I/O and processing. CPU-intensive programs will see smaller benefits from concurrent processing.