Data Leakage & Validation Splits in Machine Learning

Data Leakage and Validation Splits

Data leakage happens when information that would not be available at prediction time finds its way into training or evaluation. The model looks excellent in the notebook — sometimes impossibly excellent — and then performs far worse in production, because the evaluation measured something other than the real task. Leakage is one of the most common reasons machine-learning projects fail after a promising start, and it is rarely caused by carelessness: it hides in preprocessing steps, in how rows were split, and in features that quietly encode the answer.

The defense is to make validation imitate deployment as closely as possible: preprocessing fitted only on training data, splits that respect how entities and time work in the real problem, and a final test set that nobody touches until the end.

TL;DR

Fit every preprocessing step inside each training fold — use pipelines, not transformations applied to the whole dataset.
Use grouped splits when the same patient, user, device, or document appears in several rows.
Use time-based splits for anything predicting the future: train on the past, validate on later data, often with a gap.
Remove features unavailable at prediction time, including ones recorded after the outcome.
Check for duplicates and near-duplicates across training and test data.
Keep a held-out test set used once, and treat suspiciously good results as a bug until proven otherwise.

Quick Example

Selecting features before cross-validation leaks the labels. On pure noise — 200 rows, 5,000 random features, random labels — the leaky version reports 79% accuracy; the correct pipeline reports a chance-level 51%:

The leaky version picked the 20 features that happened to correlate with the labels across all rows, including the rows later used for validation. The validation folds were no longer unseen data.

Core Concepts

Kinds of Leakage

Why Pipelines Prevent Preprocessing Leakage

A scikit-learn Pipeline bundles preprocessing and the model, so cross-validation refits everything on each training fold and only applies it to the validation fold. Scalers, imputers, encoders, feature selectors, and resamplers such as SMOTE all belong inside the pipeline. See scikit-learn and Feature Engineering.

Target Leakage Is a Question About Time

For every feature, ask: would this value be known at the moment the prediction is made? Features computed from later events — status fields updated after the outcome, aggregates over the whole history, columns filled in by the process being predicted — make the model look prescient in evaluation and useless in production. Feature stores with point-in-time correct joins exist largely to prevent this.

Choosing a Validation Split

Grouped Splits

When one entity contributes many rows, a random split lets the model memorize entity-specific quirks and recognize them in validation. In the simulation below, 100 patients each have ten visits and a label that is random per patient — there is nothing real to learn. A random split still reports 69% accuracy; grouping by patient gives the honest 50%:

Choose the group as the unit the model will meet as new in production: a new patient, a new customer, a new store.

Temporal Splits

For time-ordered data, every validation fold should come after its training data. A gap between them mirrors the delay between when data is collected and when a prediction is used, and prevents leakage through features built from recent windows:

Rolling statistics, lags, and other time features must be computed using only data available before each prediction time. See Pandas Time Series.

The Untouched Test Set

Cross-validation is for choosing models and settings. A final test set, set aside at the start and evaluated once, estimates real-world performance. Every time results on the test set influence a decision, it becomes a little more like validation data and a little less trustworthy.

Catching Leakage

Too good to be true: accuracy far above published results or domain expectations deserves suspicion, not celebration.
Feature importance: one feature dominating the model often turns out to be a proxy for the label.
Adversarial validation: train a classifier to distinguish training rows from test rows. If it succeeds easily, the sets differ in ways the split did not intend — sometimes because of leakage, sometimes because of drift.
Duplicate checks: hash rows or use near-duplicate detection across splits.
Shadow deployment: run the model on live data without acting on it, and compare with offline results before trusting it. See MLOps.

Best Practices

Split First, Then Explore

Split off the test set before exploratory analysis, so findings about the data do not quietly shape features using test information.

Encode the Prediction Time

Store, for each training example, the moment the prediction would be made, and build features with point-in-time logic relative to it. This turns target leakage from a matter of intuition into something you can check.

Match Production Conditions

If production predicts for new customers, validate on unseen customers; if it predicts next week, validate on later weeks; if inputs arrive with a delay, simulate the delay. Validation is a simulation of deployment.

Document the Split

Record how data was split, grouped, and ordered, and why. Reviewers can then judge whether the reported metrics answer the right question. See Model Evaluation & Metrics.

Common Mistakes

Scaling or Imputing Before the Split

Oversampling Before Cross-Validation

Random Splits for Forecasting

Features Recorded After the Outcome

FAQ

What is data leakage in machine learning?

Information reaching the model during training or evaluation that would not be available when predictions are made in production. It inflates evaluation metrics and leads to models that underperform badly once deployed.

How do I know if my model has leakage?

Warning signs include results that seem too good, a single dominant feature, a large gap between cross-validation and live performance, and training and test sets that are easy to tell apart. Review every feature for availability at prediction time and check how the data was split.

Should I use random or time-based splits?

Use time-based splits whenever the model predicts future events or the data drifts over time. Random splits are appropriate only when rows are genuinely independent and time order does not matter.

What is group k-fold cross-validation?

A cross-validation scheme that keeps all rows from the same group — a patient, user, or device — in the same fold. It measures how well the model generalizes to new groups, which is usually what production requires.

Is it leakage to tune hyperparameters with cross-validation?

Not if the final performance estimate comes from data not used for tuning — a separate test set or the outer loop of nested cross-validation. Reporting the best cross-validation score after extensive tuning overstates performance.