Advantages and disadvantages

When developing a reinforcement learning algorithm (all kinds of RL algorithms), there are three basic aspects to consider:

  • Asymptotical performance: This is the maximum performance that an algorithm can achieve if it has infinite resources available in terms of both time and hardware. 
  • Wall clock time: This is the learning time required for an algorithm to reach a given performance with a given computational power.
  • Sample efficiency: This is the number of interactions with the environment to reach a given performance. 

We already explored sample efficiency in both model-free and model-based RL, and we saw how the latter is much more sample efficient. But what about wall clock time and performance? Well, model-based algorithms usually have lower asymptotic performance and are slower to train than model-free algorithms. Generally, higher data efficiency occurs to the detriment of performance and speed.

One of the reasons behind the lower performance of model-based learning can be attributed to model inaccuracies (if it's learned) that introduce additional errors into the policies. The higher learning wall clock time is due to the slowness of the planning algorithm or to the higher number of interactions needed to learn the policy in an inaccurate learned environment. Furthermore, planning model-based algorithms experience slower inference time due to the high computational cost of planning, which still has to be done on each step.

In conclusion, you have to take into account the extra time required to train a model-based algorithm and recognize the lower asymptotical performance of these approaches. However, model-based learning is extremely useful when the model is easier to learn than the policy itself and when interactions with the environment are costly or slow.

From the two sides, we have model-free learning and model-based learning, both with compelling characteristics but distinct disadvantages. Can we take the best from both worlds?

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset