The easiest regression model is called linear regression. The idea behind linear regression is to describe a target variable (such as Boston house pricing—recall the various datasets we studied in Chapter 1, A Taste of Machine Learning) with a linear combination of features.

To keep things simple, let's just focus on two features. Let's say we want to predict tomorrow's stock prices using two features: today's stock price and yesterday's stock price. We will denote today's stock price as the first feature, f₁, and yesterday's stock price as f₂. Then, the goal of linear regression would be to learn two weight coefficients, w₁ and w₂, so that we can predict tomorrow's stock price as follows:

Here,  is the prediction of tomorrow's ground truth stock price .

We could easily extend this to feature more stock price samples from the past. If we had M feature values instead of two, we would extend the preceding equation to a sum of M products, so that every feature is accompanied by a weight coefficient. We can write the resulting equation as follows:

Let's think about this equation geometrically for a second. In the case of a single feature, f₁, the equation for ŷ would become ŷ = w₁ f₁, which is essentially a straight line. In the case of two features,  would describe a plane in the feature space, as illustrated in the following screenshot:

As is evident in the preceding screenshot, all of these lines and planes intersect at the origin. But what if the true y values we are trying to approximate don't go through the origin?

To be able to offset ŷ from the origin, it is customary to add another weight coefficient that does not depend on any feature values and hence acts like a bias term. In a 1D case, this term acts as the ŷ-intercept. In practice, this is often achieved by setting f₀=1 so that w₀ can act as the bias term:

Here, we can keep  .

Finally, the goal of linear regression is to learn a set of weight coefficients that lead to a prediction that approximates the ground truth values as accurately as possible. Rather than explicitly capturing a model's accuracy like we did with classifiers, scoring functions in regression often take the form of so-called cost functions (or loss functions).

As discussed earlier in this chapter, there are several scoring functions we can use to measure the performance of a regression model. The most commonly used cost function is probably the mean squared error, which calculates an error (y_i - ŷ_i)² for every data point i by comparing the prediction ŷ_i to the target output value y_i and then taking the average:

Here, N is the number of data points.

Regression hence becomes an optimization problem—and the task is to find the set of weights that minimizes the cost function.

But enough of all of this theory—let's do some coding!