Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents, was introduced in arXiv paper 2609.28876v1, combining 1,568 Polymarket events split by time into training and evaluation periods with 18.8M dated news articles. In an evaluation of 12 models, research tools lowered Brier score for all 12, and forecasts improved as events unfolded, with the largest gains at steps where more newly dated evidence was recorded, though every model still trailed historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowered research cost but did not consistently improve forecast quality, and the environment supplies interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.

arXiv:2609.28876v1 Announce Type: new Abstract: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questions with dated news, allowing agents to research an event and revisit their predictions at successive historical dates. The same tasks and tools support repeated evaluation, collection of training interactions, and feedback from recorded outcomes without waiting for new events to resolve. Forecast-Dojo contains 1,568 Polymarket events, split by time into training and evaluation periods, and 18.8M dated news articles. In an evaluation of 12 models, research tools lower Brier score for all 12. Forecasts also improve as events unfold, with the largest gains at steps where more newly dated evidence is recorded. Every model still trails historical market forecasts in both Brier score and accuracy. A belief notebook carried between dates lowers research cost but does not consistently improve forecast quality. Beyond evaluation, Forecast-Dojo provides interaction trajectories and outcome feedback for agent learning, with supervised fine-tuning as a proof of concept.