In this chapter, we approached two different ways in which to solve an RL problem. The first is through the estimation of state-action values that are used to choose the best next action, so-called Q-learning algorithms. The second involves the maximization of the expected reward policy through its gradient. In fact, these methods are called policy gradient methods. In this chapter, we showed the advantages and disadvantages of such approaches, and demonstrated that many of these are complementary. For example, Q-learning algorithms are sample efficient but cannot deal with continuous action. Instead, policy gradient algorithms require more data, but are able to control agents with continuous actions. We then introduced DPG methods that combine Q-learning and policy gradient techniques. In particular, these methods overcome the global maximization of the Q-learning algorithms by predicting a deterministic policy. We also saw how the DPG theorem defines the deterministic policy update through the gradient of the Q-function. 

We learned and implemented two DPG algorithms: DDPG and TD3. Both are off-policy, actor-critic algorithms that can be used in environments with continuous action spaces. TD3 is an upgrade of DDPG that encapsulates a few tricks for the reduction of variance, and to limit the overestimation bias that is common in Q-learning algorithms. 

This chapter concludes the overview of the model-free reinforcement learning algorithms. We took a look at all the best, and most influential algorithms known so far, from SARSA to DQN, and from REINFORCE to PPO, and combined them in algorithms such as DDPG and TD3. These algorithms alone are capable of amazing things, with the right fine-tuning and a large amount of data (see OpenAI Five and AlphaStar). However, this isn't all there is to know about RL. In the next chapter, we move away from model-free algorithms, showing a model-based algorithm whose intent is to reduce the amount of data that is required for learning a task, by learning a model of the environment. In subsequent chapters, we'll also show more advanced techniques, such as imitation learning, new useful RL algorithms such as ESBAS, and non-RL algorithms such as evolutional strategies.