Data Analytics Intermediate Interactive

Model Tuning & Evaluation

How to tell whether a predictive model is actually good — what it really predicts, how to test it honestly, how to tune it, which metric to trust, and how to read the diagnostic output without being fooled.

40–50 min

Estimated time

6 modules

Plus a 4-page poster pack

CSSBB / CQE

Exam-aligned

Regression & ML

Scope

Overview

In this lesson

Evaluation is the discipline of asking, honestly, “would this model still be right on data it has never seen?” Tuning is the discipline of improving it without lying to yourself in the process. This lesson builds both, one layer at a time.

01What your model actually predictsThe conditional mean, residuals, and why a line is a line of averages

02Validate: train, holdout & cross-validationThe three R² values and how overfitting shows up

03Tune: bias, variance & hyperparametersImproving a model without cheating on the test set

04Measure: the metric catalogRegression, classification, clustering & quantile metrics

05Diagnose: reading the regression outputOmnibus, JB, DW, condition number, VIF, AIC/BIC

06Statistics ImplementationExcel functions, use cases, Minitab paths & exam tips

+Interpretation guide & common traps“When you see this, it means this”

✓Practice activity & mini quizWith a full answer key

Learning objectives

By the end of this lesson you should be able to

Explain what a regression model predicts (the conditional mean) and why the residual exists.
Distinguish training, holdout, and cross-validation performance, and read the gap between them as a sign of overfitting.
Tune a model with cross-validation while keeping an untouched test set, understanding the bias–variance trade-off.
Select the correct evaluation metric for a regression, classification, clustering, or quantile problem based on the cost of each kind of error.
Interpret a regression diagnostic block — normality tests, Durbin–Watson, condition number, VIF, AIC/BIC — and know which warnings to act on.
Reproduce the core metrics in Excel and Minitab and connect them to ASQ CSSBB/CQE exam expectations.

Prerequisite You should have met simple linear regression and the idea of a “model” before. No coding is required to follow the lesson, though Python output is shown because that is where these numbers usually appear.

Visual library

Four quick-reference posters

Four full-resolution infographics covering the whole model-building playbook — the workflow, and the regression, classification, quantile & clustering metric families with formulas. Download to print as a desk reference, or open full size on screen.

POSTER PACK

All four posters — one PDF, four pages

A single full-resolution download: the build-the-best-model workflow, plus the regression, classification, and quantile & clustering metric families — every acronym spelled out, with formulas and acceptance ranges. Print the set as a desk reference.

Download PDF4 pages · print Open PackView full size

POSTER 1

Build the Best Model: Workflow, Overfitting & Validation

The end-to-end modelling pipeline, data splitting & leakage, the three R²s, bias–variance, cross-validation schemes, hyperparameter search, and regularization.

Page 1

POSTER 2

Regression Metrics: Meaning, Formula & Acceptance Range

Every continuous-target metric with its equation and acceptance range — MAE, MSE, RMSE, RMSLE, MAPE, R² / adjusted R², AIC / BIC / Mallows’ Cp, and the residual diagnostics.

Page 2

POSTER 3

Classification Metrics & Tuning — with Equations

The confusion matrix and every classification metric with formulas — precision, recall, F1 / Fβ, ROC-AUC, PR-AUC, log loss, Brier, MCC — plus threshold tuning, imbalance, and calibration.

Page 3

POSTER 4

Quantile & Clustering Evaluation — with Equations

Pinball loss, prediction-interval coverage & width, and clustering indices with equations — silhouette, Davies–Bouldin, Calinski–Harabasz, choosing k, and external ARI / NMI.

Page 4

One PDF, four pages — one poster per page.

Module 01 · Foundation

What is your model actually predicting?

Before you can judge a model, you have to know what it is trying to do. Almost everyone’s first misunderstanding is the same one: they think a regression predicts the exact value. It doesn’t — and that gap is where every evaluation metric lives.

The confusion, stated plainly

“When people say OLS regression predicts the average, I get confused, because I thought it predicts the actual value of y.” This is the single most common stumbling block in modeling, and it is worth clearing up completely.

The weight analogy If I ask you to guess the weight of a person you have never met, your safest single guess is the average weight of everyone in the room. OLS regression does exactly this — just with more information. When you give it an input, it doesn’t know the random quirks that made one specific real value what it is; it gives you the average outcome for cases like this one.

It predicts the “conditional mean”
When you plug an x into the model, OLS looks at how the target behaves for that kind of input and returns the average — the mean of y conditional on x. The goal is to predict the actual value; the reality is that the best possible guess for any specific case is the average of similar cases.
The error is the leftover
For any single data point: Actual = Predicted + Error. The error (also called the residual) is the part OLS cannot predict — the random noise it can’t see. The model hands you only the “Predicted” part, which is the average behavior at that x.
Why a straight line still works
Real data has scatter, so a straight line cannot touch every point. By minimizing the squares of the errors, the line settles right in the middle of the cloud of points — which is precisely why it behaves like a line of averages.

The decomposition every metric is built on y_i = ŷ_i + e_i where ŷ_i = predicted (conditional mean) · e_i = residual (what’s left)

The reframe Stop reading “line of best fit” and start reading “line of averages.” OLS predicts what y would be on average for a given x — which is, statistically, your most accurate single guess for the actual value.

Quick check

Your steel-process model predicts 512 kWh for a slab, but the slab actually used 505 kWh. Did the model fail?

No. 512 kWh is the model’s estimate of the average energy for slabs with those settings. The 7 kWh gap is the residual — random, slab-specific variation the model was never trying to capture. You judge the model by the pattern of residuals across many slabs, not by any single miss.

Not every model predicts the mean

This matters because it decides which metrics are even valid. Ordinary regression predicts the mean. Quantile regression deliberately predicts a chosen slice of the distribution instead — say the 90th percentile of energy use, the level you’ll stay under 90% of the time. Because it is not aiming at the mean, mean-based metrics like R² and RMSE measure the wrong target. We return to its special metrics in Module 4.

Why this leads the lesson Every downstream choice — which metric, which diagnostic, what “good” looks like — depends on knowing what the model estimates. Get this wrong and you will happily optimize a number that means nothing.

Module 02 · Validation

Validate: train, holdout & cross-validation

A model that is graded on the same data it studied is like a student who saw the exam in advance. To know if it truly learned, you must test it on material it has never seen. That is the entire idea behind the three R² values you will see printed in modeling code.

The core danger A flexible model can memorize its training data — capturing not just the real signal but the random noise. It then looks brilliant in training and collapses in the real world. This is overfitting, and validation exists to catch it.

The three R² values, in order of trust

In-sample R² — Training R²
r2_train = r2_score(y_train, y_pred_train)
How well the model fits the same data it was trained on. A high value (0.9+) means it fits training data well — but if it is much higher than the other two, that is a red flag for overfitting, not a trophy. Diagnostic only
Holdout R² — Test R²
r2_holdout = r2_score(y_test, y_pred_test)
Performance on a separate ~20% slice the model never saw during training. This is your first real test of generalization. Quick reality check
Cross-validation R² — CV R²
cv_scores = cross_val_score(...)
The model is trained and tested several times on different splits of the data, and the scores are averaged. You get both a central estimate and a spread. The number to trust

How k-fold cross-validation works

The data is cut into k equal folds (5 is common). Each fold takes a turn as the test set while the other four train the model. Average the five scores for a robust estimate, and read the spread as a stability check.

Fold	Train	Test
1–5 (each)	80%	20%

5-fold CV R²: 0.68 ± 0.05 → average predictive power 0.68, split-to-split variation 0.05.

Reading the three together

Metric	Data used	Purpose
In-sample R²	Training data	Fit quality
Holdout R²	20% unseen	Generalization test
CV R²	Many unseen splits	Robust performance

Rule of thumb Use CV R² as your main metric. Training R² is a diagnostic, holdout R² is a quick check, CV R² is the true performance estimate.

Two signatures you must recognize

Healthy What good looks like

Metric	Value
Training R²	0.78
Holdout R²	0.74
CV R²	0.72 ± 0.03

Training slightly higher than the rest; holdout and CV are close; the spread is small. The model learned the signal.

Overfit What trouble looks like

Metric	Value
Training R²	0.96
Holdout R²	0.42
CV R²	0.38 ± 0.10

A huge drop from training to test, and a large, unstable spread. The model memorized noise.

Use case · the steel-process dataset The energy model groups rows by SlabID. Using GroupKFold, every measurement from one slab stays together — all in train or all in test, never split across both. So CV R² answers the question that actually matters on the plant floor: “How well can the model predict energy for completely new slabs it has never seen?” That is the most realistic measure you can get.

The leakage trap If the same slab appears in both train and test, the model can “recognize” it and post an inflated score that evaporates in production. Grouped splitting is not a nicety here — it is the difference between an honest number and a fantasy.

Quick check

A model reports Training R² = 0.94, Holdout R² = 0.51, CV R² = 0.47 ± 0.12. What is your diagnosis, and what is your next move?

Overfitting. The 0.94 → 0.51 collapse plus the wide ±0.12 spread are the classic signature. Next move: reduce model complexity (simpler model, fewer/less-correlated features, or regularization) and re-check with grouped cross-validation — covered in Module 3.

Module 03 · Tuning

Tune: bias, variance & hyperparameters

Validation tells you whether a model generalizes. Tuning is how you make it generalize better — dialing the model’s complexity to the sweet spot between too simple and too clever. The whole art is doing this without secretly cheating on your final test.

The bias–variance trade-off

Every model’s total error on unseen data splits into two competing parts. Push one down and the other tends to rise; good tuning finds the balance.

High bias · Underfit

The model is too simple to capture the real pattern. It is wrong even on the training data.

Signature: low R² everywhere — training, holdout, and CV all mediocre.

High variance · Overfit

The model is too flexible and chases noise. It is brilliant on training data and poor on new data.

Signature: high training R², big drop to holdout/CV, wide spread.

The sweet spot

Enough complexity to catch the signal, not so much that it memorizes noise.

Signature: training a touch higher than holdout/CV, all three close together.

Parameters vs hyperparameters

Aspect	Parameters	Hyperparameters
Who sets them	The learning algorithm, from the data	You, before training
Examples	Regression coefficients (slopes, intercept)	Tree depth, number of trees, regularization strength (λ), k in k-NN, learning rate
How to choose	Automatic — this is “training”	Deliberate search, judged by cross-validation

Tuning is specifically the job of choosing hyperparameters well. And the tool you use to judge each candidate is the one you already met: cross-validation.

Define a search space
List the hyperparameter values worth trying — e.g. tree depth ∈ {3, 5, 8, 12} and λ ∈ {0.01, 0.1, 1, 10}.
Score each candidate with CV
For every combination, run grouped cross-validation and record the mean CV score. Never judge a candidate by its training score.
Search efficiently
Grid search tries every combination — thorough but expensive. Random search samples combinations — usually finds a near-best setting far faster when the space is large.
Lock it in, then test once
Pick the winning hyperparameters, retrain on all your training data, and evaluate a single time on the untouched test set.

The tuning trap — do not skip this If you tune your hyperparameters against the test set, you have quietly turned it into a training set. Your final number becomes optimistic and dishonest. Keep three roles: train (fit), validation / CV (tune), and a held-back test set (report once, at the very end). Nested cross-validation formalizes this when data is scarce.

Regularization — a tuning lever with a bonus Ridge and Lasso regression add a penalty (λ) that shrinks coefficients toward zero. Turning λ up fights overfitting and stabilizes a model suffering from multicollinearity — which is exactly the “condition number” problem you will meet in Module 5. Lasso can also zero out weak predictors, doing feature selection for you.

Quick check · steel model

You are tuning the maximum depth of a tree on the steel data. Depth 3 gives CV R² 0.61, depth 8 gives 0.73, depth 20 gives training R² 0.98 but CV R² 0.55. Which depth do you ship, and why?

Depth 8. It has the best cross-validated score. Depth 3 underfits (0.61 everywhere). Depth 20 overfits — the gorgeous 0.98 training score with a 0.55 CV score is memorization. You always tune to the CV number, not the training number.

Module 04 · Metrics

Measure: choosing the right metric

A metric is a lens. The wrong lens can make a bad model look great. The metric must match three things at once: the type of problem, what the model predicts, and which mistake actually costs you.

Regression metrics — when the target is a number

Metric	What it measures	Behaviour	Use it when
MAE Mean Absolute Error	The average size of the errors, ignoring sign.	Treats all errors equally; robust to outliers.	You want a plain, honest “typical miss” in original units.
MSE Mean Squared Error	The average of the squared errors.	Punishes large mistakes far more heavily than small ones.	Big misses are especially costly and must be discouraged.
RMSE Root Mean Squared Error	The square root of MSE.	Same penalty as MSE but back in the target’s units.	You want MSE’s sensitivity with a number you can read.
R² Coefficient of determination	The share of variance in the target the model explains.	Ranges toward 1.0 for a strong fit; can be misleading alone.	Communicating overall fit; comparing to a baseline.

The three regression errors, side by side MAE = 1n Σ |y − ŷ| · MSE = 1n Σ (y − ŷ)² · RMSE = √MSE

MAE vs RMSE in one line If a few huge errors should hurt a lot, use RMSE. If every error should count the same, use MAE. RMSE ≥ MAE always, and the gap between them tells you how much your errors vary in size.

Classification metrics — when the target is a class

These all come from the confusion matrix: counts of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN).

Metric	Plain meaning	Formula	Use it when
Accuracy	Share of all predictions that are correct.	(TP+TN) / all	Classes are balanced. Misleads on imbalance
Precision	Of the cases flagged positive, how many really are.	TP / (TP+FP)	False positives are costly (false alarms).
Recall (sensitivity)	Of the real positives, how many you caught.	TP / (TP+FN)	Missing a positive is dangerous.
F1-score	The balance point between precision and recall.	harmonic mean	You need both and the data is imbalanced.
ROC-AUC	Ability to separate classes across all thresholds. See the interactive curve ↓	area under ROC	Ranking quality; threshold-free comparison.

Use case · defect detection A vision system flags defective parts. Here a missed defect (a false negative) ships a bad part to a customer — expensive and dangerous — while a false alarm just triggers a re-inspection. So you optimize Recall, accepting some precision loss. Flip the costs (say, a fraud filter that freezes innocent customers’ accounts) and you would optimize Precision instead. The metric follows the cost.

The accuracy illusion If 99% of parts are good, a model that blindly says “good” every time scores 99% accuracy — and catches zero defects. On imbalanced data, accuracy is the metric that lies most confidently. Reach for precision, recall, F1, or ROC-AUC.

Try itMove the threshold — watch precision fight recall

Each dot is a part the model scored 0→1 for “defect.” Slide the decision threshold: everything to its right is flagged. There is no free lunch — catching more defects (recall) means more false alarms (precision).

Decision threshold0.50

Flagged
(pred +)

Passed
(pred −)

Actual
defect

True Positive

False Neg · missed

Actual
good

False Pos · false alarm

True Negative

Precision—

Recall—

F1-score—

Accuracy—

On this defect problem a miss (red, passed) ships a bad part — so you'd push the threshold left to raise recall, accepting more false alarms.

CompareThe ROC curve & AUC — three models, one problem

Same defect-detection problem, three different models. Tap a chip to show or hide its curve — leave all three on to compare them at once, or narrow it down to just one. Drag the threshold and every visible curve gets a marker at that operating point, moving together.

Decision threshold0.50

Strong model0.95Excellent

Fair model0.76Fair

Weak model0.58Poor

The dashed diagonal is a random guesser (AUC = 0.5). The dot at the top-left corner is a perfect classifier (AUC = 1.0) — every real curve lives somewhere between the two.

Probability viewWhat AUC actually means

AUC also equals the probability that a randomly picked defect scores higher than a randomly picked good part — for whichever model you last tapped above. Draw pairs and watch the running estimate close in on its AUC.

Sampling pairs from: Strong model (AUC 0.95)

Draw a pair to see whether the defect part out-scored the good part — D = defect · G = good part.

Correctly ranked

0 / 0

Running estimate

—

Clustering metrics — when there are no labels

Metric	What it measures	Use it when
Silhouette score	How similar each point is to its own cluster versus the nearest other cluster. Ranges −1 to +1; higher means denser, better-separated groups.	You have no ground truth and want to judge how clean the grouping is.
Adjusted Rand Index (ARI)	How well the predicted groups agree with known true labels, corrected for chance.	You do have true groups and want to score the clustering against them.

Quantile-regression metrics — when you predict a percentile

Recall from Module 1 that quantile regression aims at a slice of the distribution, not the mean. So mean-based metrics measure the wrong thing. Two purpose-built metrics take their place.

Pinball loss (quantile loss)

The primary error metric. Unlike MSE, it is asymmetric — it penalizes over- and under-predictions differently depending on the target quantile τ.

τ = 0.5 (median): penalties are equal both ways — this is exactly MAE.
τ = 0.9 (90th pctl): under-predicting is penalized 9× more than over-predicting, which forces the line to sit above ~90% of the points.

Pseudo R²

The goodness-of-fit measure, in the spirit of R². It compares your model’s pinball loss to that of a null model (no predictors — it just guesses the sample quantile). Like R², it runs 0→1; higher means your predictors beat the baseline. This is McFadden’s pseudo R², in the Koenker–Machado form for quantile regression.

Feature	OLS regression (mean)	Quantile regression (τ)
Primary loss	Mean Squared Error	Pinball (quantile) loss
Error metric	RMSE / MAE	Mean pinball loss (= MAE if τ=0.5)
Fit metric	R²	Pseudo R²

Quick check

You’re predicting hospital readmission (rare, ~5% positive). Someone reports 95% accuracy. Are you impressed?

Not yet — maybe not at all. A model that predicts “no readmission” for everyone already scores 95% and helps no one. Ask for recall (are we catching the readmissions?), precision, and ideally ROC-AUC. On rare-event problems, accuracy is close to meaningless.

Module 05 · Diagnostics

Diagnose: reading the regression output

Metrics tell you how accurate a model is. Diagnostics tell you whether you are even allowed to trust it — whether its assumptions hold and its coefficients are stable. A model can predict well and still fail its diagnostics, and vice versa. Read both.

When you fit an OLS model in statsmodels, the summary prints a block of diagnostics beneath the coefficients. Here is that block, decoded, using the real values from the steel-process energy model.

Diagnostic	What it tests for	Benchmark	Steel value	Verdict
Omnibus p	Overall normality of residuals (skew + kurtosis together)	> 0.05 = normal	0.000	✘ ignore (large N)
Jarque–Bera p	Normality of residuals (a second test)	> 0.05 = normal	~0	✘ ignore (same reason)
Skew	Symmetry of the residual distribution	≈ 0	−0.21	✔ good
Kurtosis	Tail heaviness / outliers in residuals	≈ 3	2.61	✔ good
Durbin–Watson	Autocorrelation — are residuals independent?	≈ 2	1.57	⚠ mild autocorrelation
Condition Number	Multicollinearity / numerical stability	< 100 (ideally < 30)	7,500	✘ needs attention
AIC	Fit vs complexity trade-off (prediction focus)	Lower is better (relative only)	~917,400	⚠ compare models only
BIC	Fit vs complexity, with a stronger penalty	Lower is better (relative only)	~917,400+	⚠ compare models only

The large-N trap — the most important lesson here With tens of thousands of rows, the Omnibus and Jarque–Bera normality tests almost always “fail” (p ≈ 0), because they are powerful enough to flag microscopic, harmless departures from perfect normality. That is why both are marked “ignore” above even though the residuals are actually fine. Trust the skew (−0.21 ≈ 0) and kurtosis (2.61 ≈ 3) values and a residual plot instead of the p-value. This single misread sinks a lot of otherwise-good analyses.

Self-checkDiagnostic triage — act or ignore?

Before you read the verdict, decide for yourself: is each number a real problem, or a distraction? Tap to reveal.

Omnibus p = 0.000

tests: normality of residuals

Tap: act or ignore?

With 60k rows this test flags microscopic, harmless departures from normality. Judge normality by skew & kurtosis instead — not this p-value.
✗ Ignore

Jarque–Bera p ≈ 0

tests: normality (second test)

Tap: act or ignore?

Same large-N trap as Omnibus. Two failing normality p-values here mean nothing on their own at this sample size.
✗ Ignore

Skew = −0.21

tests: symmetry of residuals

Tap: act or ignore?

Essentially symmetric (target ≈ 0). This is the value you actually trust for normality — and it says the residuals are fine.
✓ Fine

Kurtosis = 2.61

tests: tail heaviness

Tap: act or ignore?

Close to the normal benchmark of 3, slightly light-tailed. No outlier problem in the residuals.
✓ Fine

Durbin–Watson = 1.57

tests: independence of residuals

Tap: act or ignore?

Below the ideal 2 → mild positive autocorrelation. Check row ordering / grouping; consider robust standard errors.
⚠ Watch

Condition No. = 7,500

tests: multicollinearity

Tap: act or ignore?

Far above 30. Predictors are nearly collinear, so individual coefficients are unstable. Drop/combine features or use ridge.
✗ Act

AIC / BIC ≈ 917k

tests: fit vs complexity

Tap: act or ignore?

Only meaningful compared to another model fit to the same data. The absolute number says nothing on its own.
⚠ Relative

VIF (per predictor)

tests: collinearity per variable

Tap: act or ignore?

VIF > 10 (some say 5) flags a predictor tangled with the others — the per-variable companion to the condition number.
⚠ Check

The other diagnostics you will meet

Beyond the summary block, these appear whenever you evaluate a regression seriously:

Diagnostic	What it tells you	Rule of thumb
VIF Variance Inflation Factor	How much one predictor is explained by the others — collinearity, measured per variable (where the condition number is the whole-model view).	< 5 fine · 5–10 caution · > 10 serious
Standard error	The uncertainty around a coefficient (or around predictions, as “S”, the standard error of the estimate).	Smaller = more precise estimates
PRESS	Predicted residual sum of squares — leave-one-out prediction error. A built-in generalization check.	Lower is better; compare models
MAE / RMSE	Error magnitude in the target’s units — see Module 4.	Lower is better; RMSE punishes big misses
McFadden Pseudo R²	A goodness-of-fit measure for models where ordinary R² doesn’t apply (logistic, quantile).	0.2–0.4 already indicates a good fit

What to do when a diagnostic fails

High condition number / VIF → multicollinearity

Your predictors overlap, so coefficients become unstable and their signs can flip. Fix: drop or combine correlated predictors, standardize the inputs, prune by VIF, or apply ridge regression (Module 3).

Durbin–Watson far from 2 → autocorrelation

Residuals aren’t independent — common with time-ordered or grouped data. Fix: check the ordering, add lag terms, model the group structure, or use autocorrelation-robust standard errors. At 1.57 the steel model shows only mild positive autocorrelation.

Non-normal residuals

With large samples the central limit theorem protects your inference, so mild non-normality is usually fine. Real problems are strong skew or heavy tails driven by outliers — investigate those points, consider a transform.

AIC / BIC look enormous

Their absolute size is meaningless — they only matter relative to another model on the same data. Lower wins; BIC penalizes extra predictors harder, so it prefers simpler models.

The mental model Metrics answer “is it accurate?” Diagnostics answer “is it trustworthy?” The steel model is a perfect example: its residuals are well-shaped (good skew and kurtosis), but a condition number of 7,500 warns that its coefficients are riding on multicollinearity and should not be interpreted one-by-one until that is fixed.

Quick check

A colleague rejects a 60,000-row model because the Jarque–Bera p-value is 0.000, declaring “the residuals aren’t normal, the model is invalid.” Are they right?

Almost certainly not. At 60,000 rows JB will reject on trivial departures from normality. Point them to the skew and kurtosis (near 0 and 3 = fine) and a residual histogram. The p-value is being driven by sample size, not by a real problem.

Module 06 · Statistics Implementation

Doing it in Excel & Minitab

These numbers don’t only live in Python. On the exam and on the job you will build and read them in Excel and Minitab. This section maps every core evaluation metric to the functions and menu paths that produce it.

Excel functions

Function	Syntax	Purpose	When to use it
RSQ	=RSQ(known_y, known_x)	R² of a simple linear regression.	Quick fit measure for one predictor.
LINEST	=LINEST(y, x_range, TRUE, TRUE)	Full regression: coefficients, standard errors, R², F, and residual df (array-entered).	Multiple regression, or when you need SEs and R² together.
SLOPE / INTERCEPT	=SLOPE(y,x) · =INTERCEPT(y,x)	The b and a of the fitted line.	Reporting or hand-computing predictions.
TREND / FORECAST.LINEAR	=FORECAST.LINEAR(new_x, y, x)	Predicted y (ŷ) for new inputs.	Generating the predicted column for error metrics.
STEYX	=STEYX(known_y, known_x)	Standard error of the estimate (Minitab’s “S”).	Typical prediction error in the target’s units.
SUMXMY2	=SUMXMY2(actual, predicted)	Σ(actual − predicted)² — the SSE, the building block of MSE/RMSE.	Manually computing MSE and RMSE.
DEVSQ	=DEVSQ(actual)	Σ(actual − mean)² — the total sum of squares (SST).	Manually computing R² = 1 − SSE/SST.
CORREL / PEARSON	=CORREL(a, b)	Pearson correlation r between two variables.	Screening relationships and collinearity.
SKEW / KURT	=SKEW(residuals) · =KURT(residuals)	Skewness and kurtosis of residuals.	Checking residual shape (see caveat below).
COUNTIFS	=COUNTIFS(actual,1,pred,1)	Counts TP / FP / TN / FN cells of a confusion matrix.	Building precision, recall, accuracy, F1 by hand.

Caveat that trips people up Excel’s KURT returns excess kurtosis, so a normal distribution reads ≈ 0 — whereas the statsmodels output in Module 5 reports raw kurtosis, where normal ≈ 3. Same shape, two conventions. Add 3 to Excel’s value to compare.

Excel use cases

1 · R², MAE and RMSE from two columns

With A2:A100 = actual and B2:B100 = predicted:

# R-squared
=1 - SUMXMY2(A2:A100,B2:B100)/DEVSQ(A2:A100)
# MAE
=SUMPRODUCT(ABS(A2:A100-B2:B100))/COUNT(A2:A100)
# RMSE
=SQRT(SUMXMY2(A2:A100,B2:B100)/COUNT(A2:A100))

2 · Confusion matrix → precision & recall

With actual labels in D and predictions in E (1 = positive):

# TP, FP, FN
TP =COUNTIFS(D:D,1,E:E,1)
FP =COUNTIFS(D:D,0,E:E,1)
FN =COUNTIFS(D:D,1,E:E,0)
# Precision & Recall
Precision =TP/(TP+FP)
Recall    =TP/(TP+FN)
F1        =2*P*R/(P+R)

3 · Standard error of the estimate =STEYX(actual, x) gives the typical distance between the points and the regression line, in the target’s own units — the single most intuitive “how wrong is it, usually?” number, and the direct Excel equivalent of Minitab’s S.

Minitab navigation

Menu path	What it gives you	Which to use & why
Stat → Regression → Regression → Fit Regression Model	Coefficients, R², adjusted R², S, VIF, Durbin–Watson, AIC/BIC, and residual plots in one run.	Your main tool — it produces almost every diagnostic in Module 5 at once.
Stat → Regression → Regression → Predict	Fitted values and prediction intervals for new input rows.	Generate ŷ for a holdout set to compute MAE/RMSE.
Stat → Basic Statistics → Normality Test (alt: Graph → Probability Plot)	Anderson–Darling test and a probability plot of the residuals.	Prefer the plot over the p-value on large samples — same large-N trap as Module 5.
Stat → Basic Statistics → Correlation	Pearson r between predictors.	First-pass multicollinearity screen before trusting coefficients.
Stat → Regression → Binary Logistic Regression	Classification fit with a confusion matrix and pseudo R².	When the target is a class, not a number.
Stat → Multivariate → Cluster K-Means (alt: Cluster Observations)	Cluster assignments and cluster-distance summaries.	Unlabeled grouping; judge tightness the way silhouette does.

Reading Minitab’s regression output Check the diagnostics in this order: S and R²(adj) for accuracy → VIF for multicollinearity → Durbin–Watson for independence → the residual plots for normality and constant variance. Use AIC/BIC only to rank competing models on the same data.

Exam tips — ASQ CSSBB / CQE

R² vs adjusted R²: the exam expects you to know that plain R² only ever rises when you add predictors, while adjusted R² penalizes useless ones — use it to compare models with different numbers of terms.
“S” (standard error of the estimate) is the exam’s favourite “typical prediction error” value — smaller is better, and it is in the response’s units. Know it maps to Excel’s STEYX.
VIF > 10 is the classic multicollinearity flag on the exam; the recommended remedies (remove/combine predictors, collect better data) are frequently tested.
Residual analysis — independence (Durbin–Watson ≈ 2), constant variance (no funnel in the residual plot), and normality (probability plot) — is core CSSBB Analyze/Improve content. Expect to interpret a residual plot, not just compute a number.
Match the metric to the question: regression → MAE/RMSE/R²; classification → precision/recall/F1 and the cost of each error. The exam rewards choosing recall when a miss is dangerous and precision when a false alarm is costly.
The large-N caveat — that normality tests over-reject on big samples — is exactly the kind of interpretation nuance CQE questions use to separate rote memorizers from real analysts.

Reference

When you see this, it means this

Tap any card to flip it — front is the symptom you'd spot in output, back is the diagnosis and the fix.

Training R² ≫ holdout / CV R²

Tap: what it means →

Means: Overfitting — the model memorized noise.
Do: Simplify, cut correlated features, add regularization; re-check with grouped CV.
BAD

All three R² values low

Tap: what it means →

Means: Underfitting — the model is too simple.
Do: Add signal: better features, interactions, or a more flexible model.
WARN

Wide ± spread on CV R²

Tap: what it means →

Means: Instability — performance depends on the split.
Do: Get more data, reduce complexity, confirm the split fits the data's structure.
WARN

95%+ accuracy on a rare event

Tap: what it means →

Means: The accuracy illusion — 'always negative' scores high.
Do: Switch to recall, precision, F1, or ROC-AUC.
BAD

Normality p ≈ 0 on a huge dataset

Tap: what it means →

Means: Usually the large-N trap, not a real problem.
Do: Judge by skew (≈0), kurtosis (≈3), and a residual plot.
GOOD

Condition number in thousands / VIF > 10

Tap: what it means →

Means: Multicollinearity — coefficients unstable.
Do: Drop or combine predictors, standardize, or use ridge regression.
BAD

Durbin–Watson well below 2

Tap: what it means →

Means: Positive autocorrelation — residuals not independent.
Do: Check ordering/grouping, add lag terms, or use robust SEs.
WARN

R² reported for a quantile model

Tap: what it means →

Means: Wrong metric — it assumes a mean prediction.
Do: Use pinball loss and pseudo R² instead.
BAD

Traps

Where people go wrong

Trap Judging on the training score

Why wrong: a model can memorize its training data. Avoid it: report cross-validated performance on unseen data.

Trap Tuning on the test set

Why wrong: it quietly turns the test set into training data and inflates your final number. Avoid it: tune with CV, keep one untouched test set.

Trap Leaking groups across the split

Why wrong: the same slab in train and test lets the model “recognize” it. Avoid it: use GroupKFold to keep whole groups together.

Trap Trusting accuracy on imbalanced data

Why wrong: the majority class dominates the score. Avoid it: use precision, recall, F1, or ROC-AUC.

Trap Chasing a normality p-value on big data

Why wrong: the test over-rejects at large N. Avoid it: read skew, kurtosis, and plots.

Trap Interpreting coefficients under multicollinearity

Why wrong: a high condition number makes individual coefficients unstable and their signs unreliable. Avoid it: fix collinearity first, then interpret.

Trap Comparing AIC/BIC across different datasets

Why wrong: their scale has no absolute meaning. Avoid it: compare only across models fit to the same data.

Trap Trusting a model because the software ran

Why wrong: software executes instructions; it doesn’t validate the question or the assumptions. Avoid it: check inputs, metrics, and diagnostics in context.

Practice

Practice activity — diagnose these three models

For each result, name the problem (healthy, overfit, underfit, or unstable) and give one next step. Answers are on the following page.

Model	Training R²	Holdout R²	CV R²
A · steel energy, depth-2 tree	0.44	0.42	0.41 ± 0.02
B · steel energy, depth-25 tree	0.99	0.55	0.49 ± 0.11
C · steel energy, tuned tree	0.80	0.76	0.75 ± 0.03

Bonus: Model B was validated with a plain random split rather than GroupKFold. How might its holdout R² of 0.55 be even worse than it looks?

Answer key

Practice & quiz answers

Practice activity

A — Underfit. Everything is low (~0.42) and stable. The depth-2 tree is too simple. Next step: allow more depth / add signal.
B — Overfit & unstable. 0.99 training collapses to ~0.49 CV with a wide ±0.11 spread. Next step: reduce depth, prune features, regularize.
C — Healthy. Training slightly above holdout/CV, all close, small spread. Ship it.
Bonus. With a plain random split, measurements from the same slab can land in both train and test, so the model “recognizes” slabs it has partly seen. The true score on genuinely new slabs (via GroupKFold) would likely be below 0.55 — the honest number is worse than it appears.

Wrap-up

The evaluation mindset

A regression predicts the conditional mean — the average outcome for cases like this one. The residual is the part it can’t and isn’t trying to predict.
Judge a model on data it has never seen. Training R² is a diagnostic, holdout R² is a quick check, cross-validation R² is the truth.
A big train→test gap and a wide CV spread mean overfitting; low scores everywhere mean underfitting. Tuning navigates the bias–variance trade-off between them.
Tune with cross-validation, never on the test set. Keep one untouched test set for a single, honest final number.
Match the metric to the problem and the cost of errors: recall when a miss is dangerous, precision when a false alarm is costly, RMSE when big misses hurt, and never accuracy alone on imbalanced data.
Diagnostics test trustworthiness, not accuracy. Beware the large-N normality trap; act on high condition number / VIF and on Durbin–Watson far from 2.
With grouped data, split by the group (SlabID) so the score answers the real question: can it predict something brand new?

Remember this Predict → Validate → Tune → Measure → Diagnose

Decision rule Never trust a single number. Read fit and generalization together, match the metric to the decision, and let the diagnostics tell you whether you’re even allowed to believe the coefficients.

Model Tuning & Evaluation

In this lesson

By the end of this lesson you should be able to

Four quick-reference posters

What is your model actually predicting?

The confusion, stated plainly

It predicts the “conditional mean”

The error is the leftover

Why a straight line still works

Not every model predicts the mean

Validate: train, holdout & cross-validation

The three R² values, in order of trust

In-sample R² — Training R²

Holdout R² — Test R²

Cross-validation R² — CV R²

Two signatures you must recognize

Tune: bias, variance & hyperparameters

The bias–variance trade-off

Parameters vs hyperparameters

Define a search space

Score each candidate with CV

Search efficiently

Lock it in, then test once

Measure: choosing the right metric

Regression metrics — when the target is a number

Classification metrics — when the target is a class

Clustering metrics — when there are no labels

Quantile-regression metrics — when you predict a percentile

Diagnose: reading the regression output

The other diagnostics you will meet

What to do when a diagnostic fails

Doing it in Excel & Minitab

Excel functions

Excel use cases

Minitab navigation

Exam tips — ASQ CSSBB / CQE

When you see this, it means this

Where people go wrong

Practice activity — diagnose these three models

Check your understanding

Comprehension quiz

Practice & quiz answers

Practice activity

The evaluation mindset