Bias and variance are two of the three major components that comprise a model's error. The third is called the irreducible error and can be attributed to inherent randomness or variability in the data. The total error of a model can be decomposed as follows:

As we saw earlier, bias and variance stem from the same source: model complexity. While bias arises from too little complexity and freedom, variance thrives in complex models. Thus, it is not possible to reduce bias without increasing variance and vice versa. Nevertheless, there is an optimal point of complexity, where the error is minimized as bias and variance are at an optimal trade-off point. When the model's complexity is at this optimal point (the red dotted line in the next figure), then the model performs best both in-sample and out-of-sample. As is evident in the next figure, the error can never be reduced to zero.

Furthermore, although some may think that it is better to reduce the bias, even at the cost of increased variance, it is clear that the model would not perform better, even if it was unbiased, due to the error that variance inevitably induces:

The following figure depicts the perfect model, with a minimum amount of combined bias and variance, or reducible error. Although the model does not fit the data perfectly, this is due to noise that is inherent in the dataset. If we try to fit the training data better, we will induce overfitting (variance). If we try to simplify the model further, we will induce underfitting (bias):