Now that we have talked about the main constraints of tabular algorithms and expressed the need for generalization capabilities in RL algorithms, we have to deal with the tools that allow us to get rid of these tabular constraints and address the generalization problem.

We can now dismiss tables and represent value functions with a function approximator. Function approximation allows us to represent value functions in a constraint domain using only a fixed amount of memory. Resource allocation is only dependent on the function that's used to approximate the problem. The choice of function approximator is, as always, task-dependent. Examples of function approximation are linear functions, decision trees, nearest neighbor algorithms, artificial neural networks, and so on. As you may expect, artificial neural networks are preferred over all the others – it is not a coincidence that it is widespread across all kinds of RL algorithms. In particular, deep artificial neural networks, or for brevity, deep neural networks (DNNs), are used. Their popularity is due to their efficiency and ability to learn features by themselves, creating a hierarchical representation as the hidden layers of the network increase. Also, deep neural networks, and in particular, convolutional neural networks (CNNs), deal incredibly well with images, as demonstrated by recent breakthroughs, especially in supervised tasks. But despite the fact that almost all studies of deep neural networks have been done in supervised learning, their integration in an RL framework has produces very interesting results. However, as we'll see shortly, this is not easy.