In Chapter 1, Getting Started with Python Machine Learning, we tried to fit polynomials of different complexities controlled by the d dimensionality parameter to fit the data. We realized that a two-dimensional polynomial, a straight line, does not fit the example data very well, because the data was not linear in nature. No matter how elaborate our fitting procedure was, our two-dimensional model saw everything as a straight line. We learned that it was too biased for the data at hand and called it under-fitting.

We played a bit with the dimensions and found out that the 100-dimensional polynomial fits very well to the data on which it was trained (we did not know about train-test splits at that time). However, we quickly found out that it was fitting too well. We realized that it was over-fitting so badly that with different samples of the data points, we would have gotten totally different 100-dimensional polynomials. That's why one can also say that the model has too high a variance for the given data.

These are the extremes between which most of our machine learning problems reside. Ideally, we want to have both low bias and low variance. But, we are in a bad world and have to trade off between them. If we improve on one, the other will likely get worse.