Now that we've introduced black-box evolutionary algorithms and evolution strategies in particular, we are ready to put what we have just learned into practice. The paper called Evolution Strategies as a Scalable Alternative to Reinforcement Learning by OpenAI made a major contribution to the adoption of evolution strategies as an alternative to reinforcement learning algorithms.

The main contribution of this paper is in the approach that scales ES extremely well with a number of CPUs. In particular, the new approach uses a novel communication strategy across CPUs that involves only scalars, and so it is able to scale across thousands of parallel workers.

Generally, ES requires more experience and thus is less efficient than RL. However, by spreading the computation across so many workers (thanks to the adoption of this new strategy), the task can be solved in less wall clock time. As an example, in the paper, the authors solve the 3D Humanoid Walking pattern in just 10 minutes with 1,440 CPUs, with a linear speedup in the number of CPU cores. Because usual RL algorithms cannot reach this level of scalability, they take hours to solve the same task.

Let's look at how they are able to scale so well.