There's more...

The multiprocessing example demonstrates that, while multiprocessing is useful in some situations, considerable effort must be made to optimize the program to best utilize multiprocessing. In addition, multiprocessing can be slower than single thread operations because each process must be started anew, much like the function call overhead.

There is also the problem of each process taking over a CPU core. While this helps with the parallelization of code processing, it means that the core is essentially locked until the processing is done.

For counts under than 1 million, multiprocessing performed well. At 100,000 calls, the total time was just under 4 seconds. At 10,000 calls, the time was less than 0.5 seconds, which is comparable to PyPy's time.

However, when attempting to run this code with the original 1  billion calls, this author's computer (with eight cores) locked up hard. After attempting to kill the process, the computer finally released the lock after 1.5 hours.

There are multiple reasons why the multiprocessing code caused this. The main part is that it isn't well optimized and simply tries to call the function as resources are available. Each process takes up both CPU cycles and memory space, so there eventually comes a point when new processes have to wait for resources to become available.

On the other hand, serial processing, such as Python or PyPy, don't have this overhead problem and can simply plug and chug to process the code. Even on multiple calls, they are still able to process quickly. Of course, this is a more or less artificial test and real-world projects will vary considerably as to which method is best.

Ultimately, this gives a good demonstration of the capabilities of PyPy and how it compares to multiprocessing. Combining PyPy with multiprocessing may work but, based on readings, it looks like the PyPy community isn't interested in improving performance for parallel processing, so your results will vary.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset