It is generally a good idea to mix different learning algorithms, in order to capture both linear and non-linear relationships between the features themselves, as well as the target variable. Take, for example, the following dataset, which exhibits both linear and non-linear relationships between the feature (x) and the target variable (y). It is evident that neither a single linear nor a single non-linear regression will be able to fully model the data. A stacking ensemble with a linear and non-linear regression will be able to greatly outperform either of the two models. Even without stacking, by hand-crafting a simple rule, (for example "use the linear model if x is in the spaces [0, 30] or [60, 100], else use the non-linear") we can greatly outperform the two models: