Why plain k-fold silently overfits your trading model — and the 4-line fix that stops it.
Financial data is sequential. k-fold shuffles rows, so a training row from 2 PM Tuesday sits
next to a test row from 10 AM Monday. Worse: triple-barrier labels overlap. A label at
bar t looks 6 bars into the future; a training row at t+2 "knows" part of that future.
The model leaks.
V1's history is full of "HIGH overfit" verdicts — train AUC high, test AUC flat. Plain
TimeSeriesSplit
is only marginally better; it still lets adjacent windows bleed into each
other.
For each test window [t0, t1]
:
max_training_horizon
bars after the test window — drop those too.Overlapping labels are not i.i.d. Purging + embargoing makes the split honest.
def purged_embargo_split(n, n_splits=5, embargo_frac=0.02):
idx = np.arange(n)
fold = np.array_split(idx, n_splits)
splits = []
for i in range(n_splits):
test = fold[i]
emb = int(len(test) * embargo_frac)
lo, hi = max(0, test[0]-emb), min(n, test[-1]+emb+1)
train_mask = np.ones(n, bool); train_mask[lo:hi] = False
splits.append((idx[train_mask], test))
return splits
Optuna once "won" a validation set with only 4 decisive rows — statistically meaningless.
Rule: never tune when the decisive (non-abstained) validation rows are below ~30–50. Widen the
date range or symbol basket first; don't trust the trial.
train (fit) → validation (early stop + HP select) → disjoint calibration set (sigmoid/
isotonic) → test (untouched, final score only). V1 sometimes conflated validation and
calibration. Keep them separate.
Log every trial's train/val/test gap, not just the winner's test score. Promote only if
replay AND shadow (≥1 live session) both beat baseline on buyer metrics: 1.5x/2.0x hit
rate, MAE-before-hit, time-to-hit, wrong-side rate.
Research only. Not investment advice.