Supervised Learning: Classification vs Regression

Almost every practical machine learning problem is supervised learning: you have historical examples where you already know the answer, and you want a model that predicts the answer for new cases.

Supervised learning splits into exactly two problem types, and which one you have decides everything else — the algorithms available to you, the loss function, and the metrics you evaluate with.

	Regression	Classification
Predicts	A number on a continuous scale	A category from a fixed set
Answers	”How much / how many?"	"Which one / is it?”
Examples	Salary, house price, temperature, demand	Spam or not, disease type, will they churn
Output	72,300 (any real value)	“Yes” / “No” (a label)
Typical metric	RMSE, MAE, R²	Accuracy, precision, recall, F1

How to tell which problem you have

Look at your target column (y) and ask: would averaging two of its values mean anything?

• Salary 50,000 and 70,000 → average 60,000. Meaningful → regression.
• “Spam” and “Not spam” → no average exists. Meaningless → classification.

Two traps worth knowing:

1. Numbers that are really categories. A Rating column of 1–5 stars, or PIN codes, or an encoded Country column of 0/1/2 — these look numeric but averaging them is nonsense. That’s classification (or ordinal) data. This is the same trap covered in the statistics variable-types notes.
2. “Probability of X” is regression-flavoured but still classification. Logistic regression outputs a probability between 0 and 1 — but you ultimately predict a class, so it’s a classification algorithm despite the name.

The full taxonomy

Regression algorithms

Algorithm	Best for	Note
Linear Regression	A straight-line relationship	The baseline — always start here
Multiple Linear Regression	Several predictors at once	Same maths, more columns
Polynomial Regression	Curved relationships	Linear regression on x², x³…
Ridge (L2)	Many correlated features	Shrinks coefficients, fights overfitting
Lasso (L1)	Feature selection needed	Can drive coefficients to exactly zero
ElasticNet	Both of the above	A blend of L1 and L2
Decision Tree Regressor	Non-linear, interpretable	Prone to overfitting alone
Random Forest Regressor	Strong general-purpose	Many trees averaged
Gradient Boosting / XGBoost	Best tabular accuracy	Trees built to fix each other’s errors
SVR	Small, high-dimensional data	Support vector machines for numbers

Classification algorithms

Algorithm	Best for	Note
Logistic Regression	Linearly separable, interpretable	The classification baseline
K-Nearest Neighbours	Small data, irregular boundaries	No training — memorises the data
Naive Bayes	Text, spam filtering	Fast; assumes features are independent
Decision Tree	Rules you need to explain	Human-readable
Random Forest	Strong general-purpose	Robust, little tuning needed
Gradient Boosting / XGBoost	Best tabular accuracy	Usually wins competitions
SVM	Clear margins, high dimensions	Powerful with kernels
Neural Networks	Images, audio, text	Needs lots of data

Practical advice: start with the simple baseline (Linear or Logistic Regression). It’s fast, interpretable, and tells you whether the problem is even learnable. Only reach for Random Forest or boosting once you have that number to beat.

The workflow every supervised model follows

Whichever branch you’re on, the pipeline is identical:

1. Get data            ->  pd.read_csv(...)
2. Split X and y       ->  features vs target
3. Clean               ->  handle missing values
4. Encode              ->  text categories -> numbers
5. Split train/test    ->  train_test_split(...)
6. Scale (if needed)   ->  StandardScaler
7. Fit                 ->  model.fit(X_train, y_train)
8. Predict             ->  model.predict(X_test)
9. Evaluate            ->  RMSE / R²   or   accuracy / F1

Steps 1–6 are data preprocessing and are the same for both problem types. Steps 7–8 are identical in scikit-learn — every model exposes the same .fit() / .predict() interface, which is why swapping algorithms is usually a one-line change:

from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor

model = LinearRegression()          # swap this line...
model = RandomForestRegressor()     # ...for this one, nothing else changes
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

Only step 9 genuinely differs between regression and classification — see evaluating models.

Where to go next

1. Data preprocessing — getting raw data into a shape a model accepts (missing values, encoding, train/test split).
2. Linear regression — the foundational algorithm, worked end to end by hand and in scikit-learn.
3. Regression algorithms — everything beyond the straight line.
4. Classification algorithms — the full set, with how each draws its boundary.
5. Model evaluation — the metrics for both, and why accuracy alone will mislead you.

Takeaways

• Supervised learning = learning from labelled examples; it splits into regression (predict a number) and classification (predict a category).
• The test: would averaging two target values mean anything? Yes → regression.
• Watch for numeric-looking categories — encoded labels and ratings are not regression targets.
• The workflow is the same for both; only the algorithm choice and the evaluation metrics differ.
• Always establish a simple baseline before reaching for complex models.