Your Time-Series Validation Score Is Inflated, and Your Test Suite Will Never Tell You

A developer warns that time-series validation scores are inflated due to overlapping windows, which reduce the effective sample size to roughly the row count divided by the window length. The post argues that standard leakage checks and time-based splits do not catch this issue, and that uncertainty estimates and model comparisons are therefore misleading. It recommends reporting effective sample size alongside row count and computing uncertainty on non-overlapping subsets.

Every machine learning engineer learns early that leakage inflates validation scores. We check for target leakage. We check for train-test contamination. We use time-based splits instead of random ones on temporal data. Then we build a feature over a rolling window, evaluate the model, and quietly reintroduce a related problem that no leakage check is designed to catch. It is not leakage. The trouble is a sample size you never actually had. Temporal modelling almost always involves windows. You want a label describing what happens over the next thirty days, so you compute it at every timestep. You want a feature summarising the trailing thirty days, so you compute that too. Standard practice, and correct as far as it goes. Take a daily price series with about 3,300 rows and generate a thirty-day forward label at each step. Your dataset now has roughly 3,275 labelled examples. Your training loop sees 3,275. Your metrics are computed over 3,275. Every confidence estimate you produce inherits that figure. Now consider two consecutive rows. Row one carries a label describing days 1 through 30. Row two carries a label describing days 2 through 31. Twenty-nine of the thirty days feeding those labels are shared. These are not two independent examples. They are one example with a small perturbation, and your dataset contains twenty-nine more just like it before you reach a genuinely new observation. Leakage means information from the future has contaminated the past. Time-based splits fix it, and most teams handle this correctly now. This is different. There is no contamination across the split boundary. Every row is causally valid. The problem is that your effective sample size is roughly your row count divided by the window length, and every uncertainty estimate in your pipeline assumes otherwise. Divide instead of sliding: 3,300 rows with a thirty-day window yields about 110 genuinely independent examples. Not 3,275. The ratio equals the window length exactly. A seven-day window inflates sevenfold. A ninety-day window inflates ninetyfold. Uncertainty estimates scale with the root of how many truly separate examples you hold, so overstating by thirty compresses everything by a factor near 5.5. Confidence intervals on your metrics land at roughly a fifth of their warranted width. Bootstrap distributions over your validation scores are far tighter than reality. Statistical tests comparing model A against model B will declare significance on differences that are noise. Hyperparameter selection will confidently pick a configuration that simply got a favourable draw from your hundred-odd real examples. Worst of all, cross-validation does not save you. K-fold on overlapping windows spreads near-duplicate rows across folds, so your held-out fold contains examples that share twenty-nine days with something the model trained on. The fold boundary looks clean. The information boundary is not. Nothing errors. Every assertion passes. Your data is valid, your split is temporally correct, your code is right. The defect lives in an assumption underneath the metric, and assumptions raise no exceptions. Push the window to 365 days on the same series and you can generate roughly 2,940 labelled examples from nine years of history. Nine. That is the number of independent observations available for a model predicting annual behaviour. A confidence estimate spanning 2,940 rows that describe nine underlying events is neither cautious nor bold. It carries no information at all. And those nine are not nine draws from a stationary process either - across nine years of any real-world series, the data generating process itself has usually changed more than once. Report effective sample size alongside row count in every experiment log. Row count over window length is a crude estimator and vastly better than nothing. Put it in the same table as your metrics so nobody reads the metrics without it. Compute uncertainty on non-overlapping subsets. Train on everything if you like - more correlated examples still help the fit. But derive intervals, error bars and significance tests from the independent subset only. Use blocked cross-validation with purging and embargo. Blocked splits keep contiguous segments together. Purging removes examples whose windows straddle the boundary. An embargo gap prevents the fold edge from sharing information at all. This is standard in financial ML and underused everywhere else that windows appear. Prefer block bootstrap over the standard variety. Resampling individual rows destroys the autocorrelation that created the dependence, which quietly restores the original error while looking rigorous. Treat window length as a modelling constraint, not a free parameter. A window that leaves you a handful of independent examples is not a modelling choice awaiting better regularisation. It is a problem your dataset cannot support, and the correct output is a scoped-down claim rather than a wider error bar. The pattern generalises well past finance. Any domain with sliding windows over correlated sequences carries it: sensor streams, clinical monitoring, demand forecasting, telemetry, anything with a rolling aggregate. Wherever consecutive examples share most of their underlying observations, your row count is a measure of computation rather than a measure of evidence. More rows from the same underlying history do not add information. They add duplicates with slightly different noise, and every statistical procedure downstream will thank you for them by becoming more confident about less. Count separately. Then decide what you are entitled to claim. We publish our own forecasts under this constraint - committed before the outcome, scored afterwards in the open with failures retained - at neuportal.ai/experiment Divide your row count by your window length before your next standup. Educational content - not financial advice.