Purged and Embargoed Cross-Validation for Options ML

A developer detailed how plain k-fold cross-validation silently overfits options trading models due to sequential data and overlapping triple-barrier labels, and proposed a purged and embargoed split to make validation honest. The fix drops training rows near test windows to prevent leakage, and the developer advises against tuning when decisive validation rows are below 30-50, emphasizing separate calibration sets and buyer metrics for model promotion.

Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it. Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap . A label at bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future. The model leaks. V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain TimeSeriesSplit is only marginally better; it still lets adjacent windows bleed into each other. For each test window t0, t1 : max training horizon bars after the test window — drop those too.Overlapping labels are not i.i.d. Purging + embargoing makes the split honest. python def purged embargo split n, n splits=5, embargo frac=0.02 : idx = np.arange n fold = np.array split idx, n splits splits = for i in range n splits : test = fold i emb = int len test embargo frac lo, hi = max 0, test 0 -emb , min n, test -1 +emb+1 train mask = np.ones n, bool ; train mask lo:hi = False splits.append idx train mask , test return splits Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless. Rule: never tune when the decisive non-abstained validation rows are below ~30–50. Widen the date range or symbol basket first; don't trust the trial. train fit → validation early stop + HP select → disjoint calibration set sigmoid/ isotonic → test untouched, final score only . V1 sometimes conflated validation and calibration. Keep them separate. Log every trial's train/val/test gap, not just the winner's test score. Promote only if replay AND shadow ≥1 live session both beat baseline on buyer metrics : 1.5x/2.0x hit rate, MAE-before-hit, time-to-hit, wrong-side rate. Research only. Not investment advice.