Data Leakage in Machine Learning: Why High Accuracy Can’t Save Your Model

Data leakage in machine learning can cause models to achieve 98% accuracy during evaluation but fail in production, according to a technical explainer. The article warns that preprocessing steps like mean imputation and feature scaling must be fit on training data only, and the test set must remain unseen until evaluation. It illustrates correct workflows, such as splitting data before fitting a scaler, to prevent leakage.

You train a machine learning model. It hits 98% accuracy . You feel great — the model looks almost perfect. Then you deploy it. And the performance falls off a cliff. What happened? More often than not, the answer is data leakage — one of the most common and most quietly destructive mistakes in machine learning. It’s the kind of problem that makes a model look brilliant during evaluation and fall apart the moment it meets the real world. The scary part? Your accuracy can look excellent while your model is fundamentally unreliable. And nothing in the metrics will warn you. Data leakage happens when information from outside the training data — especially from the test set — sneaks into the training process, directly or indirectly. Put simply: the model gets access to information it shouldn’t have had at that stage. This shows up in a lot of places, but one of the most common is data preprocessing and feature engineering . The general rule to hold onto is: The test set must remain completely unseen until evaluation. That doesn’t just mean “don’t train on test rows.” It also means statistics calculated from the test set should never influence training — not even indirectly. Here’s a workflow that looks perfectly reasonable at first glance: Dataset → Preprocessing → Train/Test Split → Model Training → Evaluation The problem: if preprocessing calculates anything using the entire dataset, the test set has already influenced the process — before the split even happens. The safer version flips the order: Dataset → Train/Test Split → Fit preprocessing on Train → Transform Train & Test → Train Model The rule to remember: fit on training data, transform both training and test data. Let’s see why this matters with a couple of examples. Say a feature has missing values, and you decide to fill them using the mean. Common enough. Your data: 1, 2, 3, 4, 5 After splitting: Train → 1, 3, 4Test → 2, 5 If you calculate the mean before splitting, you’re using all five values — including the two that belong to your test set. That mean now carries information from data your model was never supposed to see. The fix: split first, then compute the mean from the training data only. Train mean = 1 + 3 + 4 / 3 Use that training-derived mean to fill missing values in both the training and test sets. The test set never contributes to the calculation — it only receives the result. The test set can be transformed using information learned from the training set — but it should never be used to learn that information. Feature scaling is another classic leakage trap. Standardization, for instance: z = x - μ / σ If μ and σ are computed from the whole dataset, the test set has once again leaked into your preprocessing. Wrong: scaler.fit transform X X train, X test = train test split X Better: X train, X test = train test split X scaler.fit X train X train = scaler.transform X train X test = scaler.transform X test The scaler learns only from X train. The test set just gets transformed using what was already learned. Training data: fit + transform Test data: transform only Fitting means learning from data — a scaler learns mean and standard deviation, an imputer learns replacement values, an encoder may learn category mappings and feature selection may learn which features matter. The moment any of that learning touches the test set, the test set stops being truly unseen. It also shows up in feature engineering. Imagine predicting whether a customer will cancel their subscription, and one of your features accidentally encodes information recorded after the cancellation happened. Your model will look incredible in evaluation — because it’s essentially been handed the answer. In production, that information won’t exist at prediction time, and the model’s real performance will collapse. The model isn’t smart. It’s just been given a peek into the future. With time-dependent data — stock prices, weather, sales, sensor readings — leakage takes a different shape. If you split randomly, your test points end up scattered across the timeline, with training data sitting on both sides of them — before and after: Timeline: Jan Feb Mar Apr May Jun Jul Aug Sep ...Train: ● ● ● ● ● ●Test: ● ● ● The model ends up training on months that come after some of the months it’s being tested on. In effect, it’s learning from the future to predict the past — and no model in production will ever get that luxury. The evaluation looks great, but it’s measuring something that can’t happen in the real world. That’s not how forecasting works. In production, you only ever know the past. Your evaluation should reflect that constraint. Instead, preserve the temporal order and give the model a clean cutoff: Train the past : Jan → JunTest the future : Jul → Sep Now the model only ever trains on data that came before the point it’s predicting, and gets evaluated purely on what comes next — giving you a realistic sense of how it’ll actually behave once it’s forecasting data it has genuinely never seen. Picture two models: Model Validation Accuracy Model A 98% Model B 91% Most people would grab Model A without a second thought. But if Model A’s number came from leaked data and Model B’s came from a clean evaluation, Model B is very likely the better model. Its 91% is an honest estimate of real-world performance; Model A’s 98% might just be an inflated illusion. High accuracy doesn’t automatically mean a good model. The real question is: how was that accuracy obtained? A simple mental check catches most leakage before it happens: Would this information actually be available when the model makes a prediction in production? If the answer is no, it has no business influencing training or feature creation. Anything that learns from data should learn from the training set only — full stop. Split first, fit on train, transform everything else. python from sklearn.pipeline import Pipeline pipeline = Pipeline "scaler", StandardScaler , "model", LogisticRegression pipeline.fit X train, y train predictions = pipeline.predict X test Here, the scaler is fitted as part of the training process itself — never separately on the full dataset. Data leakage rarely looks like someone literally copying test rows into training data. It’s usually subtler than that: A model boasting 99% accuracy isn’t automatically something to celebrate. Before you do, ask yourself: Did my model really learn — or did I accidentally hand it information it shouldn’t have had? Because in machine learning, a trustworthy 90% will always beat a leaked 99%. The goal was never to build a model that shines on data it’s already seen. It’s to build one that holds up when real, unseen data walks through the door. Data Leakage in Machine Learning: Why High Accuracy Can’t Save Your Model https://pub.towardsai.net/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model-9d9a6bfdf01e was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.