Now you could ask yourself—why can deep learning combined with RL perform so well? Well, the main answer is that deep learning can tackle problems with a high-dimensional state space. Before the advent of deep RL, state spaces had to break down into simpler representations, called features. These were difficult to design and, in some cases, only an expert could do it. Now, using deep neural networks such as a convolutional neural network (CNN) or a recurrent neural network (RNN), RL can learn different levels of abstraction directly from raw pixels or sequential data (such as natural language). This configuration is shown in the following diagram:

Furthermore, deep RL problems can now be solved completely in an end-to-end fashion. Before the deep learning era, an RL algorithm involved two distinct pipelines: one to deal with the perception of the system and one to be responsible for the decision-making. Now, with deep RL algorithms, these processes are joined and are trained end-to-end, from the raw pixels straight to the action. For example, as shown in the preceding diagram, it's possible to train Pacman end-to-end using a CNN to process the visual component and a fully connected neural network (FNN) to translate the output of the CNN into an action.

Nowadays, deep RL is a very hot topic. The principal reason for this is that deep RL is thought to be the type of technology that will enable us to build highly intelligent machines. As proof, two of the more renowned AI companies that are working to solve intelligence problems, namely DeepMind and OpenAI, are heavily researching in RL.

Besides the huge steps achieved with deep RL, there is a long way to go. There are many challenges that still need to be addressed, some of which are listed as follows:

• Deep RL is far too slow to learn compared to humans.
• Transfer learning in RL is still an open problem.
• The reward function is difficult to design and define.
• RL agents struggle to learn in highly complex and dynamic environments such as the physical world.

Nonetheless, the research in this field is growing at a fast rate and companies are starting to adopt RL in their products.