Regression Algorithms Beyond the Straight Line

Linear regression assumes a straight line. When that assumption breaks — the relationship curves, features are correlated, or the model overfits — you reach for one of these.

Every algorithm below uses the same .fit() / .predict() interface, so swapping them is a one-line change.

Polynomial regression

When the data curves, fit a curve. The trick: it’s still linear regression — you just feed it x², x³… as extra features.

$$y = b_0 + b_1x + b_2x^2 + b_3x^3 + \dots$$

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(PolynomialFeatures(degree=2), LinearRegression())
model.fit(X_train, y_train)

The degree is the danger. Degree 2–3 captures gentle curves. Push it to 10 and the curve will thread through every training point perfectly and predict garbage on new data — the textbook example of overfitting.

Regularisation: Ridge, Lasso, ElasticNet

Regularisation adds a penalty on large coefficients to the loss function. Big coefficients mean the model leans hard on particular features, which is how it memorises noise; penalising them forces a simpler, more general model.

Ridge (L2)

$$\text{Loss} = \text{RSS} + \alpha\sum b_j^2$$

Squared penalty — shrinks coefficients toward zero but never to zero. Best when you have many correlated features and want to keep them all.

from sklearn.linear_model import Ridge
model = Ridge(alpha=1.0)     # alpha up = more shrinkage = simpler model

Lasso (L1)

$$\text{Loss} = \text{RSS} + \alpha\sum |b_j|$$

Absolute-value penalty — can drive coefficients exactly to zero, which removes those features entirely. Lasso therefore does automatic feature selection.

from sklearn.linear_model import Lasso
model = Lasso(alpha=0.1)
print(model.coef_)     # some will be exactly 0.0 — those features were dropped

ElasticNet

Both penalties blended — useful when you want Lasso’s selection but have groups of correlated features that Lasso picks between arbitrarily.

from sklearn.linear_model import ElasticNet
model = ElasticNet(alpha=0.1, l1_ratio=0.5)   # 0 = pure Ridge, 1 = pure Lasso

	Penalty	Coefficients	Use when
Ridge	L2 (squared)	shrunk, never zero	many correlated features, keep all
Lasso	L1 (absolute)	can hit exactly zero	you want feature selection
ElasticNet	both	mixed	correlated features and selection

Scaling is mandatory here. The penalty is applied to coefficient size, so a feature measured in thousands is penalised differently from one measured in units. Always StandardScaler before Ridge/Lasso.

Decision tree regressor

Splits the data into regions with yes/no questions and predicts the average of the training values in each region.

from sklearn.tree import DecisionTreeRegressor
model = DecisionTreeRegressor(max_depth=5, random_state=1)

• Good: captures non-linearity automatically, needs no scaling, fully interpretable (you can draw the tree).
• Bad: a single unconstrained tree overfits badly — it will grow until every leaf is one training point. Always set max_depth or min_samples_leaf.
• Note: predictions are stepwise, not smooth — a tree can’t extrapolate beyond the range of its training data.

Random forest regressor

Trains many trees on random subsets of rows and features, then averages their predictions. Individual trees overfit in different directions; averaging cancels the noise out.

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=100, random_state=1)
model.fit(X_train, y_train)

# Which features actually mattered:
print(model.feature_importances_)

The best general-purpose starting point for tabular data: strong accuracy, very little tuning, no scaling needed, and free feature-importance scores.

Gradient boosting / XGBoost

Also many trees, but built sequentially — each new tree is trained to fix the errors the previous ones made.

from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1)

# or the faster, more popular library:
# from xgboost import XGBRegressor
# model = XGBRegressor(n_estimators=100, learning_rate=0.1)

Usually the most accurate option on tabular data — this is what wins Kaggle competitions. The cost: more hyperparameters, slower training, and more sensitivity to bad settings than a random forest.

	Random Forest	Gradient Boosting
Trees built	in parallel, independently	sequentially, each fixing the last
Combines by	averaging	adding weighted corrections
Tuning needed	little	more
Overfitting risk	low	higher (needs learning_rate care)
Typical accuracy	very good	usually best

Support Vector Regression (SVR)

Fits a “tube” of width ε around the data and only penalises points falling outside it. Kernels let it model non-linear shapes.

from sklearn.svm import SVR
model = SVR(kernel='rbf', C=100, epsilon=0.1)

Effective on small, high-dimensional datasets — but requires scaling and scales poorly to large data.

Choosing one

Start:                  LinearRegression      (your baseline)
Data curves:            PolynomialFeatures    (degree 2-3)
Too many features:      Lasso                 (drops the useless ones)
Correlated features:    Ridge
Non-linear, tabular:    RandomForestRegressor (best default)
Squeezing out accuracy: GradientBoosting / XGBoost
Small + high-dim:       SVR

Two rules that matter more than the choice itself:

1. Always benchmark against linear regression. If a random forest can’t beat a straight line, the extra complexity is buying you nothing.
2. Evaluate on the test set, never on training data — a deep tree scores perfectly on data it memorised.

Comparing them properly

from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

models = {
    'Linear': LinearRegression(),
    'Ridge': Ridge(alpha=1.0),
    'Lasso': Lasso(alpha=0.1),
    'RandomForest': RandomForestRegressor(n_estimators=100, random_state=1),
    'GradBoost': GradientBoostingRegressor(random_state=1),
}

for name, model in models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)          # TEST set, always
    rmse = np.sqrt(mean_squared_error(y_test, pred))
    print(f"{name:14} RMSE={rmse:.3f}  R2={r2_score(y_test, pred):.3f}")

For a more trustworthy comparison, use cross-validation rather than a single split — it averages the score over several different splits:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='neg_root_mean_squared_error')
print(-scores.mean())

Takeaways

• Polynomial = linear regression on powers of x; keep the degree low or it overfits.
• Ridge/Lasso/ElasticNet penalise big coefficients — Ridge shrinks, Lasso selects. Both need scaled features.
• Trees handle non-linearity with no scaling; a single tree overfits, so random forests average many, and boosting builds them to correct each other.
• Random Forest is the best default for tabular data; boosting usually wins on accuracy at the cost of tuning.
• Always keep the linear baseline and always score on the test set.