The third regression algorithm that we want to explore is support vector regression (SVR). As the name implies, SVR is part of the support vector family and sibling of the support vector machine (SVM) for classification (or we can just call it SVC) we learned in Chapter 5, Classifying Newsgroup Topic with Support Vector Machine.

To recap, SVC seeks an optimal hyperplane that best segregates observations from different classes. Suppose a hyperplane is determined by a slope vector w and intercept b, the optimal hyperplane is picked so that the distance (which can be expressed as ) from its nearest points in each of segregated spaces to the hyperplane itself is maximized. Such optimal w and b can be learned and solved by the following optimization problem:

• Minimizing ‖w‖
• Subject to wx⁽ⁱ⁾+b≥1 if y⁽ⁱ⁾=1 and wx⁽ⁱ⁾+b≤1 if y⁽ⁱ⁾=-1, given a training set of (x⁽¹⁾,y⁽¹⁾), (x⁽²⁾,y⁽²⁾),… (x⁽ⁱ⁾,y⁽ⁱ⁾)…, (x^(m),y^(m)).

In SVR, our goal is to find a hyperplane (defined by a slope vector w and intercept b) so that two hyperplanes, wx+b=-ε and wx+b=ε, that are a distance away from itself covers most training data. In other words, most of data points are bounded in the ε bands of the optimal hyperplane. And at the same time, the optimal hyperplane is as flat as possible, which means ‖w‖ is as small as possible, as shown in the following diagram:

This translates into deriving the optimal w and b by solving the following optimization problem:

• Minimizing ‖w‖
• Subject to |y⁽ⁱ⁾-(wx⁽ⁱ⁾+b)| ≤ ε , given a training set of (x⁽¹⁾,y⁽¹⁾), (x⁽²⁾,y⁽²⁾),… (x⁽ⁱ⁾,y⁽ⁱ⁾)…, (x^(m),y^(m))