Model Evaluation & Metrics: Precision, Recall, AUC & CV

Q: Precision or recall — which should I optimize?

It depends on which error is costlier. Optimize precision when false positives are expensive (flagging legitimate email as spam, blocking a valid transaction) — you want predicted positives to be trustworthy. Optimize recall when false negatives are dangerous (missing a tumor, failing to catch fraud) — you want to catch every real positive. When both matter, use F1 (their harmonic mean) or tune the decision threshold to the cost tradeoff.

Model Evaluation & Metrics

Proper evaluation is what separates a model that looks good from one that is good. The core risk is fooling yourself: a single lucky train/test split, an accuracy score on imbalanced data, or test information leaking into training can all produce numbers that collapse in production. Reliable evaluation is mostly about disciplined methodology and choosing metrics that match the actual goal.

The first principle: pick metrics aligned with the business objective, not whatever's convenient. 95% accuracy is worthless if 95% of cases are negative and you miss every positive one — what matters is the cost of each kind of error.

TL;DR

Choose metrics aligned with business goals.
Use cross-validation, not a single train/test split.
Watch for data leakage.
Monitor performance on production data, not just at training time.

Quick Example

Cross-validation gives a more honest estimate than one split — report mean and spread:

Splits & Cross-Validation

Hold out a test set you touch once, and tune on validation:

Prefer K-fold (or StratifiedKFold to preserve class balance) over a single split — it uses data efficiently and reports variance, not just a point estimate.

Classification Metrics

The confusion matrix (TN/FP/FN/TP) underlies them all:

Choosing between them is about error cost:

Spam detection → high precision (a false positive hides a real email).
Cancer screening → high recall (a false negative misses a sick patient).
General/balanced → F1; for ranking quality independent of threshold → AUC.

Regression Metrics

Use MAE when all errors count equally, RMSE when large errors are disproportionately bad, R² to communicate variance explained.

Imbalanced Classes

⚠️ With 95% negatives, a model that predicts "negative" for everything scores 95% accuracy and is useless. Accuracy lies on imbalanced data.

Hyperparameter Tuning

Best Practices

Match the metric to the cost of each error type.
Cross-validate and report mean ± std, not a single number.
Keep the test set sacred — evaluate on it once, at the end.
Stratify and address imbalance with weights/resampling and proper metrics.
Fit all preprocessing inside CV folds to prevent leakage.

Common Mistakes

Leakage from preprocessing before the split

Accuracy on imbalanced data

FAQ

Precision or recall — which should I optimize?

It depends on which error is costlier. Optimize precision when false positives are expensive (flagging legitimate email as spam, blocking a valid transaction) — you want predicted positives to be trustworthy. Optimize recall when false negatives are dangerous (missing a tumor, failing to catch fraud) — you want to catch every real positive. When both matter, use F1 (their harmonic mean) or tune the decision threshold to the cost tradeoff.

Why use cross-validation instead of a single train/test split?

A single split gives one estimate that can be lucky or unlucky depending on which rows landed in test. Cross-validation rotates the test fold across the data, so every point is evaluated once, and you get a mean and a variance — telling you both expected performance and how stable it is. It uses limited data more efficiently and is far more trustworthy, especially on smaller datasets.

What's the difference between ROC-AUC and PR-AUC?

ROC-AUC measures ranking quality across all thresholds using true/false positive rates; it's threshold-independent and intuitive but can look optimistic on heavily imbalanced data. PR-AUC (precision-recall AUC, AUPRC) focuses on the positive class and is more informative when positives are rare and you care about them — fraud, disease, anomalies. For imbalanced problems, prefer PR-AUC.

How do I know if my great validation score will hold in production?

You don't fully, until you monitor production. Guard against the usual culprits: ensure no leakage, use a held-out test set evaluated once, and check that your validation distribution matches serving reality (no sampling bias, no train/serving skew). Then monitor live performance and data drift — offline metrics estimate, production confirms. See MLOps.