Training a model is easy. Knowing whether it’s any good is the part that separates working systems from confident nonsense — and it’s where the regression and classification paths finally diverge, because you cannot measure “how far off is this number?” the same way you measure “did it pick the right category?”

The golden rule

Always evaluate on data the model has never seen.

model.fit(X_train, y_train)
y_pred = model.predict(X_test)     # TEST set
score = metric(y_test, y_pred)

Scoring on training data measures memorisation. A deep decision tree can score 100% on data it has memorised and fail completely on anything new.

---

Regression metrics

Using the fit from the linear regression walkthrough (y = 0.6x + 2.2, actual y = [2, 4, 5, 4, 5], predicted = [2.8, 3.4, 4.0, 4.6, 5.2]):

Metric	Formula	Our value	In words
MAE	mean of |y − ŷ|	0.64	Off by 0.64 on average
MSE	mean of (y − ŷ)²	0.48	Squared units — for optimising
RMSE	√MSE	0.693	Off by 0.69, in real units
R²	1 − RSS/TSS	0.60	Explains 60% of the variance

import numpy as np
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

mae = mean_absolute_error(y_test, y_pred)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
r2 = r2_score(y_test, y_pred)

MAE vs RMSE — the one distinction to remember. Both report error in the original units, but RMSE squares before averaging, so it punishes large misses far more:

Model A errors: [1, 1, 1, 1]   -> MAE 1.0,  RMSE 1.0
Model B errors: [0, 0, 0, 4]   -> MAE 1.0,  RMSE 2.0

Identical MAE, very different RMSE. If one catastrophic error is much worse than four small ones (delivery times, medical dosing), optimise RMSE. If all errors cost the same, MAE is the honest summary — and it’s far less swayed by outliers.

R² is the only one comparable across problems, because it’s unitless. 0.6 means 60% of the variance explained; 0 means no better than always guessing the mean; negative means worse than that (yes, that happens).

---

Classification metrics

The confusion matrix

Every classification metric is derived from this table, so learn it first.

from sklearn.metrics import confusion_matrix, classification_report

print(confusion_matrix(y_test, y_pred))
# [[TN FP]
#  [FN TP]]

print(classification_report(y_test, y_pred))   # precision/recall/F1 in one call

Accuracy — and why it lies

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

The share of predictions that were right. Fine when classes are balanced, dangerously misleading when they aren’t.

The fraud example. 1,000 transactions, 10 fraudulent. A model that always predicts “legitimate” gets 99% accuracy and catches zero fraud. The headline number looks excellent; the model is worthless.

That single scenario is why the next two metrics exist.

Precision and recall

$$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}$$

• Precision — “when it says yes, how often is it right?” Punishes false alarms. Denominator is everything predicted positive.
• Recall — “of all the real positives, how many did it catch?” Punishes misses. Denominator is everything actually positive.

They trade off against each other. Lower your threshold and you catch more positives (recall up) but raise more false alarms (precision down).

Which to optimise depends on which error costs more:

Situation	Prioritise	Because
Cancer screening	Recall	A missed case is fatal; a false alarm just means another test
Spam filter	Precision	Junk in the inbox is annoying; a lost job offer is a disaster
Fraud detection	Recall	Missed fraud costs money directly
Recommendations	Precision	Bad suggestions erode trust fast

F1 score

The harmonic mean of precision and recall, when you need one number:

$$F_1 = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Harmonic, not arithmetic, on purpose: it punishes imbalance. Precision 1.0 with recall 0.0 averages to 0.5 the normal way, but F1 = 0 — correctly calling that model useless.

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

accuracy_score(y_test, y_pred)
precision_score(y_test, y_pred)
recall_score(y_test, y_pred)
f1_score(y_test, y_pred)

ROC curve and AUC

Classifiers output probabilities; the 0.5 threshold is arbitrary. The ROC curve plots true-positive rate against false-positive rate across every possible threshold, and AUC is the area beneath it.

from sklearn.metrics import roc_auc_score, roc_curve

y_proba = model.predict_proba(X_test)[:, 1]     # probability of class 1
auc = roc_auc_score(y_test, y_proba)
fpr, tpr, thresholds = roc_curve(y_test, y_proba)

AUC	Meaning
1.0	Perfect separation
0.9+	Excellent
0.7–0.8	Acceptable
0.5	Random guessing — no signal at all

AUC is threshold-independent, so it answers “does this model rank positives above negatives?” rather than “is it right at one particular cut-off”. For heavily imbalanced data, prefer the precision-recall curve — ROC-AUC can look flattering when negatives dominate.

Picking a classification metric

Balanced classes:            accuracy is fine
Imbalanced classes:          F1, or precision/recall separately
Misses are expensive:        recall
False alarms are expensive:  precision
Comparing models overall:    ROC-AUC
Heavily imbalanced:          PR-AUC

---

Diagnosing overfitting

Compare training and test scores. The gap tells you what’s wrong:

Train	Test	Diagnosis	Fix
High	High	Good model	Ship it
High	Low	Overfitting — memorised the data	Simplify, regularise, more data
Low	Low	Underfitting — too simple	More complex model, better features
Low	High	Something’s wrong with your split	Check for leakage or a tiny test set

print("Train:", model.score(X_train, y_train))
print("Test: ", model.score(X_test, y_test))

A tree scoring 1.00 on train and 0.62 on test is the textbook picture of overfitting.

Cross-validation

A single train/test split is one roll of the dice — an unlucky split can flatter or damn a good model. K-fold cross-validation splits the data k ways, trains k times, and averages:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5, scoring='f1')
print(scores)                       # one score per fold
print(scores.mean(), scores.std())  # average, and how stable it is

The standard deviation matters as much as the mean — a model scoring 0.85 ± 0.02 is trustworthy; 0.85 ± 0.15 is a coin flip that happened to average well.

For imbalanced classification use StratifiedKFold so every fold keeps the class balance.

Data leakage — the silent killer

If information from the test set reaches training, your scores become fiction. The usual culprits:

# WRONG — the scaler learns from test data too
X_scaled = StandardScaler().fit_transform(X)
X_train, X_test = train_test_split(X_scaled, ...)

# RIGHT — fit on train only, apply to both
X_train, X_test = train_test_split(X, ...)
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

Same rule for imputers and encoders — the discipline covered in data preprocessing. A Pipeline enforces it automatically:

from sklearn.pipeline import Pipeline

pipe = Pipeline([('scaler', StandardScaler()), ('model', LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)    # scaling now re-fits inside each fold

Suspiciously perfect scores (0.99+) almost always mean leakage, not genius.

Takeaways

• Score on the test set, never on training data.
• Regression: RMSE to report, MAE when outliers shouldn’t dominate, R² to compare across problems.
• Classification: start from the confusion matrix. Accuracy is meaningless on imbalanced data.
• Precision = trust its yes; recall = catches everything. Choose based on which error costs more; F1 balances them; AUC is threshold-independent.
• The train-vs-test gap diagnoses over/underfitting; cross-validation tells you whether a score is stable.
• Guard against leakage — fit transformers on training data only, or use a Pipeline.