Don't rush into rewriting your code in a concurrent fashion just yet. There are plenty of reasons not to write concurrent code. Let's look at a few reasons here:
- First of all, don't do it if you don't need that boost—any type of concurrency adds code complexity and makes debugging exponentially harder.
- Second, the code for many specific computation-demanding tasks is already written. For example, multiple sklearn models support multicore execution—you just need to specify the number of CPUs to use. Some solutions, such as Numba, can release the GIL for specific operations, without large code changes being made.
- Some important packages do not support concurrent operations—asynchronous execution in particular—such as sqlalchemy and most database access tools in general.
All in all, make sure you really need your code to run concurrently or in parallel before investing your time and effort. As cool as it sounds, concurrent code is notoriously difficult and takes significantly more time to develop, optimize, and maintain.
If you want to get a deeper understanding of concurrency in Python (which is a very wide topic), we can recommend the following resources:
- Curious Course on Coroutines and Concurrency, by David Beazley: https://www.youtube.com/watch?v=Z_OAlIhXziw
- Concurrent Execution (https://docs.python.org/3/library/concurrency.html)
Speaking of maintenance, let's talk about the other side of the same performance coin — apart from code performance, there is coding performance. On many occasions, the ability to write code faster, add changes quickly, and introduce fewer bugs could be even more valuable than the speed of code itself. Thus, let's talk about best practices when it comes to coding and tools that will help you be a better developer.