cd /news/ai-research/forecastgym-a-live-controlled-and-re… · home › topics › ai-research › article
[ARTICLE · art-172419] src=machinebrief.com ↗ pub= topic=ai-research verified=true sentiment=· neutral

ForecastGym: A Live, Controlled, and Replayable Benchmark for Forecasting Agents

A new arXiv paper (2611.23045v1) introduces ForecastGym, a live, controlled and replayable benchmark that records what four commercial web search APIs return for open prediction-market questions every two days and replays that frozen evidence offline once outcomes are known. The first release covers 150 questions, 614 evidence states and 131k search results; rerunning the same queries after settlement returned results that all but state the outcome for up to 92% of questions despite tested date filters, and on the frozen evidence all tested LLMs trailed the market in forecasting accuracy. Switching the search API changed a model's Brier score by up to 0.043 — more than the spread across models — while persistent notes scored similarly to rereading all evidence at half the input tokens, leading the authors to conclude the evidence a forecaster sees can matter as much as the model that reads it.

recommendation

Freeze and replay the exact search evidence when evaluating forecasting agents, since API choice alone moved Brier scores by up to 0.043, and use persistent notes instead of rereading all evidence to halve input tokens.

more in playbook: Evals and testing for LLM apps
by read1 min views2 publishedNov 19, 2026

arXiv:2611.23045v1 Announce Type: cross Abstract: Agents built on large language models (LLMs) increasingly forecast real-world events by searching the Web and reasoning over what they find. How well such an agent forecasts depends on both the retrieved evidence and the model, making control over that evidence essential for evaluation. Live evaluations can mix retrieval with reasoning when each system searches on its own, while retrospective search can leak later information through date filters or today's ranking of historical pages. Comparing models, search sources, and memory strategies requires both fixed evidence and a record of how it evolved before resolution. We introduce ForecastGym, which records what four commercial web search APIs return for open prediction-market questions every two days and replays this frozen evidence offline once outcomes are known, comparing models, memory, and captured search sources under a fixed evidence-acquisition policy. Its first release covers 150 questions, 614 evidence states, one per question and collection time, and 131k search results. Rerunning the same queries after settlement returns results that all but state the outcome for up to 92% of the questions, despite the tested date filters. On the frozen evidence, all tested LLMs trail the market in forecasting accuracy. Our diagnostic analysis suggests that missed public information contributes more to the gap in high-gap states than misreading does. Switching the search API changes a model's Brier score by up to 0.043, more than the spread across models, while persistent notes score similarly to rereading all evidence at half the input tokens. The evidence a forecaster sees can matter as much as the model that reads it.

── more in #ai-research 4 stories · sorted by recency
── more on @forecastgym 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/forecastgym-a-live-c…] indexed:0 read:1min 2026-11-19 · —