# AI forecasters have caught the superforecasters, and the harder questions start now

> Source: <https://rews.cc/a/ai-forecasters-have-caught-the-superforecasters-and-the-hard-a877a9>
> Published: 2026-11-20 12:38:46+00:00

**Recommendation:** Evaluate forecasting models separately on short-horizon, data-rich questions and on market questions, since Good Judgment measured humans at 0.40 versus 0.59 for the best AI on the latter.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

Here is a claim that would have sounded ambitious a year ago: AI forecasters now match or beat the best humans at ordinary forecasting questions, the will-X-happen-by-date-Y kind scored on benchmarks like ForecastBench and the Metaculus Cup. That is the opening premise of a [post on Astral Codex Ten](https://astralcodexten.com/p/the-ai-forecasting-agenda), Scott Alexander’s blog. The leaders are mostly specialised systems wrapped in forecasting harnesses and built by startups such as FutureSearch, Mantic and Preseen. General foundation models like ChatGPT and Claude are, on his account, roughly six months behind them, so he expects them to reach top-human level by the end of the year too.

It helps to remember what the alternatives used to be. If you wanted a forecast as good as the best humans could make, you could hire a team of superforecasters, which costs thousands of dollars and takes weeks. Or you could look at a prediction market or Metaculus, which only cover a small set of general-interest questions and, in Alexander’s phrasing, are “constantly trying to screw you over by gaming resolution criteria.” (That’s a market-structure problem. The traders make money on how a question resolves, which isn’t always the same as what the question was supposed to be asking.) Now the field’s problem, he writes, is closer to working out “what kind of magnifying glass should we use to inspect our giant novelty lottery check as we confirm that it’s real?”

So, is it real? The Forecasting Research Institute (FRI) runs ForecastBench, and [on January 29 it reported](https://forecastingresearch.substack.com/p/llms-are-closing-the-gap-on-human) that superforecasters were still in first place, ahead of the best language models by 0.017 Brier points. FRI reckoned that gap was worth about a year of model progress. It then drew a straight line and wrote: “we extrapolate LLM-superforecaster parity in November 2026 (95% CI: January 2026 – November 2027).” By July, the startup Lightning Rod [was reporting](https://blog.lightningrod.ai/p/ai-reaches-human-superforecaster-range) that its Foresight-v3 system had reached fourth place on the leaderboard with an overall Brier Index of 67.7. The human superforecaster median, at 69.3, was still first, but several AI systems now sat inside the humans’ confidence interval. The superforecasters’ own firm isn’t fully convinced. Good Judgment [argues](https://goodjudgment.substack.com/p/what-forecastbench-doesnt-measure) that the questions resolved so far lean toward short-horizon, data-rich topics where machines have structural advantages, and that on the benchmark’s “market questions” superforecasters remain far more accurate, scoring 0.40 against 0.59 for the nearest AI entrant.

Still, the broad picture holds. On the standard questions, the machines have mostly caught up. The interesting part is what comes next, and Alexander, after talking to people at FRI, sets out three open problems.

## If this, then what?

A traditional question asks whether the Democrats will win the next election. A conditional question asks: if the Democrats win, what happens to the odds of a stock market crash? Policymakers want conditionals, because policy is a conditional. If you mandate vaccines, what happens to mortality compared with the world where you don’t?

Prediction markets have been slow to offer these, and the reason is funny once you look at the plumbing. If the condition never happens, the market has to give everyone their money back, which is a nuisance. And, as Alexander notes, conditional markets attract fewer customers than “new forms of degenerate sports gambling.” An AI forecaster doesn’t need customers or refunds. You just ask it, and FutureSearch, Preseen and Mantic all offer conditional questions as an explicit feature.

Whether the answers are any good is another matter. About a year ago FRI found that the AIs of the time did significantly worse on conditional questions than on ordinary ones, apparently because they didn’t quite grasp the concept. The experts Alexander spoke with expect gradual improvement rather than a sudden jump. They wouldn’t say whether today’s models beat the average smart human, but they think the models still trail top superforecasters, and a benchmark should give a better answer in a few months. In the meantime FRI is already using the method: in a [working paper by Jason Abaluck and Ezra Karger](https://forecastingresearch.org/research/airo), an ensemble of models puts the median probability of an AI-caused catastrophe by 2050 at 7.2% if no substantial new AI policy is adopted, against 6.0% unconditionally.

## If this, because of that

Conditionals have a known flaw. Suppose that in 2025 you ask whether the economy is more likely to be bad in 2027 if the Democrats win in 2026. The answer could be yes simply because a bad economy is what got the Democrats elected, and the bad economy then carried on. That tells you nothing about whether voting Democratic hurts the economy, which is the question you actually cared about. Prediction markets can’t fix this, because a market needs an observable event to pay out on, and “what would have happened otherwise” never resolves.

The FRI people pointed out that a causal forecast is really just a large pile of conditional forecasts. Add enough variables (the previous economy, a war with Iran) and check whether the economy still gets worse when the only thing you change is who wins. Ten binary variables give 2^10, or 1,024, branches. No human will forecast that many, but for an AI it is a somewhat larger API bill. At least one company already tries something along these lines, though nobody has studied how well it works.

To test it, FRI is using Freeciv, a free clone of *Civilization*. In a game you can reload an old save, change one thing and replay it as often as you like, so the full distribution of outcomes, and the causal effect of the change, can actually be measured. Real life offers no reload button.

## Forecasting the forecasters

The third problem matters most to Alexander, who would like AI forecasters to tell him which strategies are most likely to reduce existential risk. The trouble is that AI is the one field where good forecasting seems to break down. Every year, top human and AI forecasters have underestimated how fast AI advances. Earlier this summer, for example, they put the chance of AI solving a Millennium Problem this year at only 5%. FRI’s own record shows the same pattern: in its biorisk work, [the median superforecaster expected](https://x.com/Research_FRI/status/1940017616023704037) AI to match a top virologist team on troubleshooting tests by 2034, and models did it as of April 2025.

Some experts think this is one big correlated miss rather than a systematic flaw. A well-calibrated 95% forecast should be wrong 5% of the time, every AI capability moves together, and things may revert to the mean. Others suspect AI is a field where the usual winning method (start from historical base rates and adjust slightly) loses to the usual losing method (reason from big ideas such as scaling laws). If that’s true, a machine trained on the winning method may carry the same blind spot into every question about itself.

Then again, in January FRI fitted a trend line to AI forecasting skill and it pointed to November 2026. If AI forecasters really have caught the humans this month, that is one forecast about AI that came in on schedule, and it was a forecast about AI forecasting.
