{"slug": "data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model", "title": "Data Leakage in Machine Learning: Why High Accuracy Can’t Save Your Model", "summary": "Data leakage in machine learning can cause models to achieve 98% accuracy during evaluation but fail in production, according to a technical explainer. The article warns that preprocessing steps like mean imputation and feature scaling must be fit on training data only, and the test set must remain unseen until evaluation. It illustrates correct workflows, such as splitting data before fitting a scaler, to prevent leakage.", "body_md": "You train a machine learning model. It hits **98% accuracy**. You feel great — the model looks almost perfect.\n\nThen you deploy it.\n\nAnd the performance falls off a cliff.\n\nWhat happened?\n\nMore often than not, the answer is **data leakage** — one of the most common (and most quietly destructive) mistakes in machine learning. It’s the kind of problem that makes a model look brilliant during evaluation and fall apart the moment it meets the real world.\n\nThe scary part? Your accuracy can look excellent while your model is fundamentally unreliable. And nothing in the metrics will warn you.\n\nData leakage happens when information from outside the training data — especially from the test set — sneaks into the training process, directly or indirectly.\n\nPut simply: **the model gets access to information it shouldn’t have had at that stage.**\n\nThis shows up in a lot of places, but one of the most common is **data preprocessing and feature engineering**. The general rule to hold onto is:\n\nThe test set must remain completely unseen until evaluation.\n\nThat doesn’t just mean “don’t train on test rows.” It also means **statistics calculated from the test set should never influence training** — not even indirectly.\n\nHere’s a workflow that looks perfectly reasonable at first glance:\n\n```\nDataset → Preprocessing → Train/Test Split → Model Training → Evaluation\n```\n\nThe problem: if preprocessing calculates anything using the *entire* dataset, the test set has already influenced the process — before the split even happens.\n\nThe safer version flips the order:\n\n```\nDataset → Train/Test Split → Fit preprocessing on Train → Transform Train & Test → Train Model\n```\n\nThe rule to remember: **fit on training data, transform both training and test data.** Let’s see why this matters with a couple of examples.\n\nSay a feature has missing values, and you decide to fill them using the mean. Common enough. Your data:\n\n```\n1, 2, 3, 4, 5\n```\n\nAfter splitting:\n\n```\nTrain → 1, 3, 4Test  → 2, 5\n```\n\nIf you calculate the mean *before* splitting, you’re using all five values — including the two that belong to your test set. That mean now carries information from data your model was never supposed to see.\n\n**The fix:** split first, then compute the mean from the training data only.\n\n```\nTrain mean = (1 + 3 + 4) / 3\n```\n\nUse that training-derived mean to fill missing values in both the training and test sets. The test set never contributes to the calculation — it only receives the result.\n\nThe test set can be transformed using information learned from the training set — but it should never be used to learn that information.\n\nFeature scaling is another classic leakage trap. Standardization, for instance:\n\n```\nz = (x - μ) / σ\n```\n\nIf μ and σ are computed from the whole dataset, the test set has once again leaked into your preprocessing.\n\n**Wrong:**\n\n```\nscaler.fit_transform(X)X_train, X_test = train_test_split(X)\n```\n\n**Better:**\n\n```\nX_train, X_test = train_test_split(X)\nscaler.fit(X_train)X_train = scaler.transform(X_train)X_test = scaler.transform(X_test)\n```\n\nThe scaler learns only from X_train. The test set just gets transformed using what was already learned.\n\n**Training data:** fit + transform **Test data:** transform only\n\nFitting means *learning* from data — a scaler learns mean and standard deviation, an imputer learns replacement values, an encoder may learn category mappings and feature selection may learn which features matter. The moment any of that learning touches the test set, the test set stops being truly unseen.\n\nIt also shows up in feature engineering. Imagine predicting whether a customer will cancel their subscription, and one of your features accidentally encodes information recorded *after* the cancellation happened. Your model will look incredible in evaluation — because it’s essentially been handed the answer. In production, that information won’t exist at prediction time, and the model’s real performance will collapse.\n\nThe model isn’t smart. It’s just been given a peek into the future.\n\nWith time-dependent data — stock prices, weather, sales, sensor readings — leakage takes a different shape. If you split randomly, your test points end up scattered across the timeline, with training data sitting on *both sides* of them — before and after:\n\n```\nTimeline:  Jan  Feb  Mar  Apr  May  Jun  Jul  Aug  Sep  ...Train:      ●         ●    ●              ●    ●    ●Test:            ●              ●    ●\n```\n\nThe model ends up training on months that come *after* some of the months it’s being tested on. In effect, it’s learning from the future to predict the past — and no model in production will ever get that luxury. The evaluation looks great, but it’s measuring something that can’t happen in the real world.\n\nThat’s not how forecasting works. In production, you only ever know the past. Your evaluation should reflect that constraint. Instead, preserve the temporal order and give the model a clean cutoff:\n\n```\nTrain (the past):    Jan → JunTest (the future):        Jul → Sep\n```\n\nNow the model only ever trains on data that came before the point it’s predicting, and gets evaluated purely on what comes next — giving you a realistic sense of how it’ll actually behave once it’s forecasting data it has genuinely never seen.\n\nPicture two models:\n\nModel Validation Accuracy Model A 98% Model B 91%\n\nMost people would grab Model A without a second thought. But if Model A’s number came from leaked data and Model B’s came from a clean evaluation, Model B is very likely the better model. Its 91% is an honest estimate of real-world performance; Model A’s 98% might just be an inflated illusion.\n\nHigh accuracy doesn’t automatically mean a good model. The real question is: **how was that accuracy obtained?**\n\nA simple mental check catches most leakage before it happens:\n\nWould this information actually be available when the model makes a prediction in production?\n\nIf the answer is no, it has no business influencing training or feature creation.\n\nAnything that *learns* from data should learn from the training set only — full stop. Split first, fit on train, transform everything else.\n\n``` python\nfrom sklearn.pipeline import Pipeline\npipeline = Pipeline([    (\"scaler\", StandardScaler()),    (\"model\", LogisticRegression())])pipeline.fit(X_train, y_train)predictions = pipeline.predict(X_test)\n```\n\nHere, the scaler is fitted as part of the training process itself — never separately on the full dataset.\n\nData leakage rarely looks like someone literally copying test rows into training data. It’s usually subtler than that:\n\nA model boasting 99% accuracy isn’t automatically something to celebrate. Before you do, ask yourself:\n\nDid my model really learn — or did I accidentally hand it information it shouldn’t have had?\n\nBecause in machine learning, **a trustworthy 90% will always beat a leaked 99%.** The goal was never to build a model that shines on data it’s already seen. It’s to build one that holds up when real, unseen data walks through the door.\n\n[Data Leakage in Machine Learning: Why High Accuracy Can’t Save Your Model](https://pub.towardsai.net/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model-9d9a6bfdf01e) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model", "canonical_source": "https://pub.towardsai.net/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model-9d9a6bfdf01e?source=rss----98111c9905da---4", "published_at": "2026-08-19 13:31:01+00:00", "updated_at": "2026-08-19 14:11:23.560855+00:00", "lang": "en", "topics": ["machine-learning"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model", "markdown": "https://wpnews.pro/news/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model.md", "text": "https://wpnews.pro/news/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model.txt", "jsonld": "https://wpnews.pro/news/data-leakage-in-machine-learning-why-high-accuracy-cant-save-your-model.jsonld"}}