# My AI Trading Bot Looked Profitable. Every Number That Said So Was the Wrong Number. > Source: > Published: 2026-11-06 02:41:32+00:00 For five months I ran an LLM-powered forecaster against real prediction markets: Kalshi and Polymarket. It researched a market, estimated a probability, compared it with the price, and logged a "shadow" bet whenever it disagreed enough. Every resolved bet went into a SQLite database. The dashboards looked great. Directional accuracy was above 60%. Brier scores cleared the bar I'd set. One strategy beat the market on its first 45 trades. Another showed a statistically significant edge. None of it held up. That isn't because the model was secretly brilliant and unlucky. Each of those numbers was measuring something other than "will this make money." This post is about how I found that out, one metric at a time. It's the most useful engineering lesson I've learned this year, and it applies well beyond betting. If you evaluate any AI system, whether a classifier, a ranker, an agent or a forecaster, you probably have one of these five problems too. The headline number was **60.9% directional accuracy**. That was over every market the model looked at. But most of those were SKIPs: markets where the model's estimate sat close to the price, so no bet would ever be placed. On those, "pick the favourite" is easy and the model agrees with the crowd. That pads the score. The bets that matter are the ones where the model *disagrees* with the market by 5 points or more. That's the only subset where money moves. On that subset: **61 of 152 correct. 40.1%.** The corpus number and the actionable number pointed in **opposite directions**. The model was right when it agreed with everyone and wrong 60% of the time when it had an opinion. **The lesson:** score the decisions your system actually makes, not everything it touches. If your agent only acts on 15% of inputs, the other 85% are noise in your eval. My go-live gate said "Brier score under 0.25." Soccer came in at 0.2177, which passed. The market's own price scored 0.2118 on the same games. The model "passed" while losing to the thing it was betting against. A fixed threshold told me nothing. The only bar that matters for a forecaster is the **price**, because the price is your opponent. When I re-scored every football category against the market, the market won every time. Over 517 markets it was 0.2376 for the model and 0.2166 for the price. I killed football forecasting. **The lesson:** a threshold is a guess. A baseline is a fact. Compare against the best simple thing you could do instead, whether that's the price, the heuristic, the last model or the `if/else`. Cross-venue "arbitrage" (the same event priced differently on two exchanges) looked like the cleanest win. At **n=45**, its Brier score was 0.0906 against the market's 0.0928. It was a small edge, but a real one, and it was enough that I wired it up to place real orders. At **n=170**, the edge was gone: 0.1951 for the model, 0.1951 for the market. On the 21 bets it actually took, it went 9 for 21. A negative or zero edge doesn't turn positive with more data. More volume only makes you more certain it's not there. **The lesson:** never promote a system on its first good sample. Decide the checkpoint size *before* you look, and don't let an early number buy production access. This one hurt the most because the bug was a single constant. The "favourites" strategy bought heavily favoured outcomes priced 0.65–0.80. I pre-registered it: run until 100 resolved out-of-sample bets, then judge. At 135 resolved bets: That was already negative. Then I found my fee model was wrong. I had Polymarket US's taker fee as `0.003 × p`. The real formula is `0.0695 × p × (1 − p)`, about 1.5 points at these prices, or **five times what I'd modelled**. Net edge was **-4.97pp**. Even the one *statistically significant* positive result had the same problem. Closing-line value on favourites was +1.15pp, with a 95% CI of [+0.37, +1.97] that stayed above zero. But that was measured at the *mid*-price. Scored at the ask, the price you actually pay, after fees, it came to **-1.93pp**. The spread plus the fee ate it. **The lesson:** measure at the point of execution, with the real costs, from the real docs. "Edge at mid" is a backtest fantasy. For AI systems the equivalent is quality measured without latency, token cost or the retry rate. The economics markets ask things like "Will CPI come in above 3.2%?" The model's estimates were wildly off, and for weeks I treated that as a modelling problem. It was a parsing bug. Any strike with a decimal point got cut off, so the question the model received read **"more than -0"**. 251 rows were estimated from questions with the threshold missing. I was grading a model on a test it could not read. The opposite leak showed up in the backtest. The walk-forward calibration was quietly using information from after the forecast time (look-ahead), and it counted some markets more than once. Fixing both made the numbers worse, and they were also the first honest ones. **The lesson:** before you blame the model, read the exact input it received. Then check that your evaluation can't see the future. After all five fixes, one number decides whether real money is allowed: **Realised edge, net of fees, at the executable price, one row per market, with a bootstrapped 95% confidence interval whose lower bound must clear zero.** The all-time figure today: **-0.61pp, CI [-8.01, +6.88].** Not cleared. Then there's the part nobody puts on a dashboard: statistical power. My earlier gate said I needed "about 400 more bets." That was a 50%-power figure built on an effect size that turned out to be inflated. At 80% power, proving a real **3-point edge takes about 2,000 bets.** At my volume, that's a long time. So the bot doesn't trade. Specifically: Every "I built an AI trading bot" post I've read ends with an upward-sloping equity curve. Mine ends with a confidence interval that straddles zero and an honest power calculation. That's the more useful post. The bot didn't fail because LLMs can't reason about probabilities. The *evaluation* failed five times, and each failure flattered the model. That pattern is everywhere in AI engineering right now: a leaderboard number that doesn't match your traffic, a demo scored on the easy cases, a cost model off by 5×, a benchmark leaking into the training data. The forecaster was the easy part. Building a measurement system that couldn't lie to me took months, and that's the part I'd hire for. *If you're building evals for agents or forecasters and want to compare notes on baselines, power and pre-registration, I'd love to hear how you're doing it.*