Applying Statistics: Anomaly Detection, Feature Selection & ML

28. Which Measure to Use When

[T]

A student (Ashwini) asked directly: “There are so many methodologies — box plotting, standard deviation, taking the log. Which one is correct? Which is suitable? How do we find which one gives the correct predictable value?”

The instructor reframed it as “which one to use when?” and answered:

Measure	When to use it
IQR	”Whenever we want to identify the outliers in our data.”
Standard Deviation	”How much is the spread in our dataset” — how much the data is spread around the centre
Variance	”Don’t use variance much. We use standard deviation.”

The single most quotable line from this exchange:

“Don’t use variance much. Okay? We use standard deviation.”

Why: variance’s squared units make it uninterpretable in conversation (§16). It exists as the mathematical stepping stone to standard deviation, not as a number you report.

---

29. Outliers, Anomaly Detection & Root Cause Analysis

[T]

29.1 Anomaly detection

Instructor: “Suppose you are doing anomaly detection. How do you identify the anomaly?”

Answer: “If any value is beyond 1.5 IQR, you say this is the anomaly.”

The 1.5 × IQR rule IS the anomaly detection method. Once flagged, you then investigate why the anomaly occurred.

---

29.2 The IT use case — job completion time logs

Raised by a student working in IT and developed by the instructor:

The scenario: you have job completion times recorded in your system logs.

The method:

1. Take all the logs
2. Look at job completion time
3. Are there any outliers in the data? Are there any jobs going beyond the pattern?
4. Note those down
5. Think about optimising those scenarios

“This is regular statistics.”

Why it’s useful: a job that ran unusually long is a candidate for performance optimisation. The IQR rule tells you objectively which jobs count as “unusually long” rather than relying on gut feel.

---

29.3 Outliers in model building

Question from a student: “We are learning AI. Where do these calculations and statistics fit in? Which segment — the analysis part?”

Answer: “Yes. These things are the data analysis part.”

On what to do with an outlier when building a model:

“When you are taking restaurant wait as one of the input features — this occurred only one time. Your model need not learn about this whole pattern. So you can ignore this value. Usually, what we’ll do — we will remove the outliers while building the model.”

---

29.4 The two faces of an outlier — summary

Context	What you do with the outlier
Business analysis / monitoring	Investigate it. It’s a signal — a kitchen failure, a cooling fault, a slow job, a suspicious company
Model building	Remove it. A one-off event isn’t a pattern the model should learn

Both are correct. Which applies depends on whether you’re explaining the past or predicting the future.

---

30. Standard Deviation for Feature Selection

[T+S]

This closes the loop: standard deviation started as a formula, became a churn indicator, and ends as a feature-selection criterion.

The setup

Two candidate input features for a machine learning model — from a temperature sensor:

Feature F₁  →  Mean 30 ,  standard deviation 0.1
Feature F₂  →  Mean 30 ,  standard deviation 5

Identical means. Very different standard deviations.

Question posed: “Which is the most important feature for model building? Which one do you take?”

---

The class’s first instinct — and why it was wrong

Student: “We take the deviation which is less.”

This is the intuitive answer, and it is incorrect.

---

The correct answer

Feature	SD	Decision
F₁	0.1	DROP IT
F₂	5	KEEP IT — pass as input to the model

The reasoning

F₁ has a standard deviation of 0.1 around a mean of 30 — it is almost a constant. Every observation is essentially 30.

“It’s almost constant data. It won’t help you.”

A near-constant feature carries no information. There is nothing in it for the model to learn from — it cannot help the model distinguish one record from another.

“The data has to have a good variation as an input to your model. Then only it will learn.”

---

The real-estate illustration

You’re building a house price model for Chennai. Every house is priced at roughly:

₹1,00,000  ·  ₹1,05,000  ·  ₹1,12,000  ·  ₹1,08,000  ·  ₹98,000

There is barely any deviation. If someone asks “what will be the price of a house in Chennai?” you just answer “about 1 lakh” — and you don’t need a machine learning model to tell you that.

---

The rule

A machine learning model needs a good amount of variation (deviation) in its input features in order to learn.

“If any input feature to the AI model doesn’t have much variation or deviation, we will ignore that feature because it won’t help us. We’ll take a feature which has very good deviation.”

---

31. Where Statistics Fits in an AI/ML Workflow

[T]

Pulling together the answers given across the session:

Stage	What statistics does	Section
Data analysis / EDA	Five-point summary, box plots, spotting outliers	§26, §27
Data cleaning	Detect outliers via 1.5 × IQR; remove them before modelling	§23, §29
Missing value treatment	Mean if no outliers, median if outliers present	§11
Feature engineering	Create predictive features (usage deltas, SDs) from raw data	§3.2
Feature selection	Drop near-constant features (low SD)	§30
Model validation	Test whether an observed lift is statistically significant (A/B)	§3.1
Model governance	Equal opportunity score → bias detection → ship/reject decision	§3.3
Production monitoring	Range and IQR as alarm triggers for anomalies	§15.3, §29

The through-line: statistics is not a preliminary step you finish and leave behind. It appears at every stage of the pipeline, and at the final stage it has veto power over whether the model ships.

---

---