EAs are inspired by biological evolution and implement techniques and mechanisms that simulate biological evolution. This means that EAs go through many trials to create a population of new candidate solutions. These solutions are also called individuals (in RL problems, a candidate solution is a policy) that are better than the previous generation, in a similar way to the process within nature wherein only the strongest survive and have the possibility to procreate. 

One of the advantages of EAs is that they are derivative-free methods, meaning that they don't use the derivative to find the solution. This allows EAs to work very well with all sorts of differentiable and non-differentiable functions, including deep neural networks. This combination is schematized in the following diagram. Note that each individual is a separate deep neural network, and so we'll have as many neural networks as the number of individuals at any given moment. In the following diagram, the population is composed of five individuals:

The specificity of each type of evolutionary algorithm differs from the others, but the underlying cycle is common to all the EAs and works as follows:

1. A population of individuals (also called candidate solutions or phenotypes) is created so that each of them has a set of different properties (called chromosomes or genotypes). The initial population is initialized randomly.
2. Each candidate solution is evaluated independently by a fitness function that determines its quality. The fitness function is usually related to the objective function and, using the terminology we've used so far, the fitness function could be the total reward accumulated by the agent (that is, the candidate solution) throughout its life.
3. Then, the fitter individuals of the population are selected, and their genome is modified in order to produce the new generation. In some cases, the less fit candidate solution can be used as a negative example to generate the next generation. This whole step varies largely, depending on the algorithm. Some algorithms, such as genetic algorithms, breed new individuals through two processes called crossover and mutation, which give birth to new individuals (called offspring). Others, such as evolution strategies, breed new individuals through mutation only. We'll explain crossover and mutation in more depth later in this chapter, but generally speaking, crossover is the process that combines genetic information from two parents, while mutation only alters some gene values in the offspring.
4. Repeat the whole process, going through steps 1-3 until a terminal condition is met. On each iteration, the population that's created is also called a generation.

This iterative process, as shown in the following diagram, terminates when a given fitness level has been reached or a maximum number of generations have been produced. As we can see, the population is created by crossover and mutation, but as we habe already explained, these processes may vary, depending on the specific algorithm:

The main body of a general EA is very simple and can be written in just a few lines of code, as shown here. To summarize this code, on each iteration, and until a fitted generation has been produced, new candidates are generated and evaluated. The candidates are created from the best-fitted individuals of the previous generation:

solver = EvolutionaryAlgortihm()

while best_fitness < required_fitness:
    
    candidates = solver.generate_candidates() # for example from crossover and mutation

    fitness_values = []
    for candidate in candidates:
        fitness_values.append(evaluate(candidate))

    solver.set_fitness_values(fitness_values)

    best_fitness = solver.evaluate_best_candidate()

The applications of EAs are actually spread across many fields and problems, from economy to biology, and from computer program optimization to ant colony optimization.

Since we are mostly interested in the application of evolutionary algorithms for solving sequential decision-making tasks, we will explain the two most common EAs that are used to solve these kinds of jobs. They are known as genetic algorithms (GAs) and evolution strategies (ESes). Later, we'll take a step further with ES by developing a highly scalable version of it.