Run 51 in an HMM sweep never reached training.
The data pull had 306 bars. The walk-forward planner left 172 for training. The HMM needed at least 362. There was no parameter I could move in the UI to make the run valid.
At least it failed loudly.
I checked the HMM first. Nothing was wrong with it. The data puller, fold planner, and refusal check each had a different idea of how much history was enough.
Training was one function call. Getting those three pieces to agree was the actual job. That is fairly typical of the ML work inside StratCraft.
Say two assets produce almost the same momentum score.
The first got there with steady volume, stable volatility, and broad participation across its sector. The second jumped on extreme turnover, became crowded, and moved far away from its recent range.
A plain momentum feature gives them roughly the same number. I would not want the system to treat them as the same setup. One might continue. The other might reverse violently.
That is about where ML becomes useful to me. I do not need it to name tomorrow's winner. I need it to notice interactions that are awkward to encode by hand. Momentum can mean something different when liquidity, volatility, crowding, and the broader market state change around it.
It can also learn an interaction that exists only in the sample. The model does not tell me which one it learned.
The first target people reach for is tempting:
What will this stock return tomorrow?
The answer comes back as +0.73%
, which looks scientific because it has two decimal places.
That precision is mostly decorative. Financial labels are noisy. A model can miss the exact return badly while still being useful at separating stronger candidates from weaker ones.
Most factor research in StratCraft ends up working with scores and ranks. Give every asset a score, sort the universe, and check whether higher-ranked assets actually do better than lower-ranked ones. Rank IC is one of the basic tests.
A stripped-down version looks like this:
import numpy as np
from scipy.stats import spearmanr
def rank_ic(scores: np.ndarray, forward_returns: np.ndarray) -> float:
valid = np.isfinite(scores) & np.isfinite(forward_returns)
if valid.sum() < 20:
return float("nan")
correlation, _ = spearmanr(scores[valid], forward_returns[valid])
return float(correlation)
This does not make the signal tradable. It only asks whether the ordering contains information. I still need to check stability across folds, turnover, costs, monotonicity across buckets, and whether the result survives another market period.
Rank IC is one row in the result. Bad candidates can score well on it.
There is a common shortcut in ML trading research: calculate every factor available, drop the columns into XGBoost, and let feature importance sort it out.
I have built versions of this. It produces a very wide table and a backtest that is much easier to trust than it should be.
XGBoost has no objection to a feature that was not known at decision time, twelve copies of the same exposure, or a label that overlaps the validation boundary. They are all valid columns as far as the model is concerned. You just get a nicer chart.
The fix cannot live inside XGBoost because XGBoost does not know what "known at the time" means. That belongs to the research layer.
A discovery run carries its validation geometry as data rather than leaving it in a notebook comment. The shortened contract looks like this:
type CvScheme = 'walk_forward' | 'cpcv';
interface DataSnapshotSpec {
walk_forward_folds: number;
walk_forward_scheme: 'expanding' | null;
embargo_bars: number | 'auto';
cv_scheme?: CvScheme;
cpcv_total_segments?: number;
cpcv_test_segments?: number;
}
That object becomes part of the snapshot identity. Change the embargo or CV scheme and it is a different research run, not an accidental overwrite of the old result.
The detail I care about most is the embargo. Suppose a label uses the next 24 bars of returns. A training row close to the validation boundary may already contain information from the validation period. A normal chronological split still leaks. Purging removes overlapping observations. The embargo leaves additional space around the boundary.
This is boring code. It is also the difference between an out-of-sample test and an in-sample test wearing a fake moustache.
Run 51 had three owners for sizing. Each component was locally reasonable. Together they produced an impossible experiment.
Lowering the HMM minimum would have made the demo run and hidden the bug. Instead, pull sizing, fold splitting, and refusal now derive from one CV sizing contract. The model declares its training floor. The planner works backwards from there and calculates how much market history to request.
Roughly:
model training floor
+ warmup
+ purge / embargo loss
+ out-of-sample segments
= minimum data pull
If the provider cannot supply enough history, the run should refuse before training. I would rather explain a refusal than publish a score from a malformed experiment.
Basically the old code looked fine in review. Each piece was reasonable on its own. They just had three different definitions of enough data
living in three places, and nobody had asked whether they agreed.
Cheap signal sweeps create a different problem. Researchers run lots of them.
Try enough templates, parameters, universes, and windows, and something will look brilliant by accident. Keeping only the best Sharpe ratio does not remove the failed attempts from history. Statistically, they are still in the room.
Failed candidates stay in the database with their snapshot and lineage. That gives the family-level multiple-testing correction the full family instead of a cleaned-up list of winners.
The evaluation path also computes Probabilistic Sharpe Ratio and Deflated Sharpe Ratio. CPCV is available when one walk-forward path feels too dependent on a particular split. None of these fixes a badly chosen universe.
I do not think any of these statistics are magic. A bad universe definition can still poison the whole exercise. Costs can be wrong. Regimes can change immediately after the test ends. Maybe the cleanest result is still luck.
I keep the failed candidates because the correction needs them. Deleting them changes the question after seeing the answer.
A usable trading system rarely stops at one model. It may have a momentum score, a mean-reversion score, an HMM regime estimate, a collection of mined factors, and a few rules that exist because someone got hurt before.
The current path looks roughly like this:
idea or factor
-> frozen as-of snapshot
-> purged validation
-> statistical gates
-> signal catalog
-> combinator
-> backtest with costs and constraints
-> promote, monitor, or reject
The Signal Factory and Combinator are separate. Signals can be tested and versioned on their own. The combinator deals with admitted signals instead of letting one large model quietly absorb every responsibility.
The simple baseline matters here. Equal weighting is not glamorous, which is exactly why it is useful. A learned combination method should have to beat a stable, understandable baseline out of sample. If it cannot, adding more parameters did not buy anything.
Correlation matters too. A large catalog can still be one momentum bet duplicated under a pile of names.
The phrase "the model is 20% and the system is 80%" is a decent shorthand. I would not treat those numbers as measured fact. Some projects spend nearly all their effort on data. Others really do have difficult model research.
Even without the percentages, most of my work lands outside training. model.fit()
is often the cleanest line in the project. The surrounding system has to decide what data existed and how labels overlap. It also owns fair comparisons, selection bias, signal combination, and promotion.
On Run 51, the correct output was refused: insufficient training bars
. Other candidates stop at promotion. If a learned combination method loses to equal weight out of sample, there is no good reason to ship the extra complexity.
A validation stack does not guarantee profitable trading. People can still overfit manually, obviously. The software does not need to make it easier.
Find me on StratCraft | GitHub