Classification Algorithms: Logistic Regression to Boosting

Classification predicts a category: spam or not, which disease, will this customer churn. Every algorithm below is solving the same puzzle — where do I draw the boundary between classes? — and they differ mainly in the shape of boundary they’re able to draw.

KNN — irregular

Tree — rectangles

SVM — smooth curve

Logistic regression

Despite the name it’s a classifier, and it’s the baseline you should always run first.

It computes a linear combination of the features (exactly like linear regression) then squashes the result into a probability between 0 and 1 with the sigmoid function:

$$p = \frac{1}{1 + e^{-(b_0 + b_1x_1 + \dots + b_nx_n)}}$$

If p > 0.5 predict class 1, else class 0.

from sklearn.linear_model import LogisticRegression

model = LogisticRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)           # hard labels: [0 1 1 0]
proba = model.predict_proba(X_test)      # probabilities: [[0.9 0.1] ...]

• Good: fast, interpretable (coefficients show each feature’s influence), gives real probabilities, hard to overfit.
• Bad: only draws a straight boundary.
• Scaling: yes.

predict_proba is underrated. The 0.5 cut-off is a choice, not a law — for fraud or disease detection you might act at 0.2 to catch more positives.

K-Nearest Neighbours (KNN)

No training at all: it memorises the data, then classifies a new point by majority vote of its k closest neighbours.

from sklearn.neighbors import KNeighborsClassifier

model = KNeighborsClassifier(n_neighbors=5)
model.fit(X_train, y_train)

• k too small (1) → jagged boundary, fits noise.
• k too large → over-smoothed, ignores real structure.
• Use an odd k for binary problems to avoid ties.
• Scaling is essential — it’s a distance algorithm, so an unscaled Salary column would swamp Age entirely.
• Slow at prediction time on big data (it compares against every training point).

Naive Bayes

Applies Bayes’ theorem, “naively” assuming every feature is independent of the others:

$$P(\text{class} \mid \text{features}) \propto P(\text{class}) \prod_i P(x_i \mid \text{class})$$

from sklearn.naive_bayes import GaussianNB, MultinomialNB

model = GaussianNB()        # continuous features
# model = MultinomialNB()   # word counts — the classic spam filter

The independence assumption is almost always false (in text, word order and co-occurrence obviously matter) — yet it works remarkably well anyway, especially for text classification and spam filtering. Extremely fast, and fine with very little training data.

Decision tree

Asks a series of yes/no questions, splitting to make each group as pure as possible (measured by Gini impurity or entropy).

from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

model = DecisionTreeClassifier(max_depth=4, random_state=1)
model.fit(X_train, y_train)

plot_tree(model, feature_names=cols, class_names=['No', 'Yes'], filled=True)
plt.show()

• Good: the only model you can literally show a non-technical stakeholder — it’s a flowchart. Handles non-linearity, needs no scaling, mixes numeric and categorical naturally.
• Bad: overfits aggressively if unconstrained. Always set max_depth.
• Boundaries are axis-aligned rectangles, so a diagonal boundary needs many clumsy steps.

Random forest

Hundreds of trees, each on a random subset of rows and features, voting together. The single best default classifier for tabular data.

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(n_estimators=100, random_state=1)
model.fit(X_train, y_train)

print(model.feature_importances_)    # free feature ranking

Strong accuracy out of the box, resistant to overfitting, no scaling required, handles irrelevant features gracefully. The trade-off is interpretability — you can’t read 100 trees the way you read one.

Support Vector Machine (SVM)

Finds the boundary with the widest possible margin between classes. The kernel trick lets it bend that boundary into non-linear shapes.

from sklearn.svm import SVC

model = SVC(kernel='rbf', C=1.0, gamma='scale', probability=True)

Parameter	Effect
kernel	'linear' straight, 'rbf' curved (default choice), 'poly' polynomial
C	low = wider margin, more tolerant; high = fits training data harder
gamma	how far one training point’s influence reaches (rbf only)

Powerful in high dimensions and with clear margins, but requires scaling, gets slow past ~10k rows, and needs probability=True to expose probabilities.

Gradient boosting / XGBoost

Sequential trees, each correcting the previous ones’ mistakes. Typically the highest accuracy available on tabular data.

from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)

# or:  from xgboost import XGBClassifier
#      model = XGBClassifier(n_estimators=100, learning_rate=0.1)

Lower learning_rate with more n_estimators generally generalises better.

Side by side

Algorithm	Boundary	Scaling	Interpretable	Speed	Best for
Logistic Regression	linear	Yes	High	Fast	Baseline, probabilities
KNN	irregular	Essential	Medium	Slow predict	Small data
Naive Bayes	probabilistic	No	Medium	Very fast	Text, spam
Decision Tree	rectangles	No	Highest	Fast	Explainable rules
Random Forest	complex	No	Low	Medium	Best default
SVM	linear or curved	Essential	Low	Slow on big data	High dimensions
Gradient Boosting	complex	No	Low	Slow train	Best accuracy

Multi-class problems

Everything above extends beyond two classes. Scikit-learn handles it for you, via one of two strategies:

• One-vs-Rest (OvR) — one binary classifier per class (“is it A or not?”).
• One-vs-One (OvO) — one classifier per pair of classes, then a vote.

model = LogisticRegression(multi_class='ovr')
model.fit(X_train, y_train)      # y can have 3+ distinct labels

Trees, forests and naive Bayes are natively multi-class — no wrapper needed.

The class imbalance trap

If 99% of transactions are legitimate, a model that predicts “legitimate” every single time is 99% accurate and completely useless. This is the most common way beginners fool themselves.

Defences:

# 1. Tell the model to weight the rare class more heavily
model = LogisticRegression(class_weight='balanced')
model = RandomForestClassifier(class_weight='balanced')

# 2. Keep class proportions when splitting
train_test_split(X, y, test_size=0.2, stratify=y, random_state=1)

# 3. Resample — SMOTE synthesises new minority examples
# from imblearn.over_sampling import SMOTE
# X_res, y_res = SMOTE().fit_resample(X_train, y_train)

And critically: stop using accuracy. Judge imbalanced problems on precision, recall and F1 — see model evaluation.

Choosing one

Start:                  LogisticRegression      (baseline + probabilities)
Need to explain it:     DecisionTree            (max_depth 3-5)
Text / spam:            MultinomialNB
Small, weird boundary:  KNN                     (scale first!)
General tabular:        RandomForest            (best default)
Maximum accuracy:       XGBoost / GradientBoosting
High-dimensional:       SVM                     (scale first!)
Images / audio / text:  Neural networks

Takeaways

• Every classifier draws a decision boundary; the shape it can draw is what really distinguishes them.
• Logistic regression is the baseline — fast, interpretable, real probabilities, straight boundary.
• Scale for KNN and SVM; trees and forests don’t need it.
• Random Forest is the best general default; boosting usually edges it out on accuracy for more tuning effort.
• Never trust accuracy on imbalanced data — use stratify, class_weight='balanced', and precision/recall/F1.