Bias refers to the inability of a method to correctly estimate the target. This does not only apply to machine learning. For example, in statistics, if we want to measure a population's average and do not sample carefully, the estimated average will be biased. In simple terms, the method's (sampling) estimation will not closely match the actual target (average).

In machine learning, bias refers to the difference between the expected prediction and its target. Biased models cannot properly fit the training data, resulting in poor in-sample performance and out-of-sample performance. A good example of a biased model arises when we try to fit a sine function with a simple linear regression. The model cannot fit the sine function, as it lacks the required complexity to do so. Thus, it will not be able to perform well in-sample or out-of-sample. This problem is called underfitting. A graphical example is illustrated in the following figure :

The mathematical formula for bias is the difference between the target value and the expected prediction: