The methods seen so far that combine model-based and model-free learning have been designed especially to work with low-dimensional state spaces. So, how do we deal with high-dimensional observation spaces as images?

One choice is to learn in latent space. Latent space is a low-dimensional representation, also called embedding, g(s), of a high-dimensional input, s, such as an image. It can be produced by neural networks such as autoencoders. An example of an autoencoder is shown in the following diagram:

It comprises an encoder that maps the image to a small latent space, g(s), and the decoder that maps the latent space to the reconstructed image. As a result of the autoencoder, the latent space should represent the main features of an image in a constrained space so that two similar images are also similar in latent space.

In RL, the autoencoder may be trained to reconstruct the input, S, or trained to predict the next frame observation, S', (along with the reward, if needed). Then, we can use the latent space to learn both the dynamic model and the policy. The main benefit arising from this approach is the big gain in speed due to the smaller representation of the image. However, the policy learned in the latent space may suffer from severe deficits when the autoencoder isn't able to recover the right representation. 

Model-based learning on high-dimensional spaces is still a very active area of research.

So far, we have covered model-based learning and its combination with model-free learning in a more figurative and theoretical way. Although it's indispensable in terms of understanding these paradigms, we want to put them into practice. So, without further ado, let's focus on the details and implementation of our first model-based algorithm.