How Proper Scoring Rules Shape LLM Forecasting

A new arXiv paper (2608.28482v1) finds that the choice of proper scoring rule as a training objective significantly shapes the calibration, probability use, and error structure of LLM forecasters, despite identical theoretical incentives for truthful reporting. Comparing five scoring rules on binary forecasts of resolved real-world events, the Brier-trained model achieved the lowest Brier score and highest AUC-ROC, while the log-trained model had the highest log score and lowest calibration error. The authors caution that single-seed training means some differences may reflect stochasticity.

arXiv:2608.28482v1 Announce Type: new Abstract: This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training objectives for binary forecasts of resolved real-world events. Although the rules share the same theoretical incentive for truthful probability reporting, the resulting models differ in calibration, probability use, and estimated profiles of bias, information, and noise, with smaller differences in aggregate accuracy and discrimination. The Brier-trained model has the lowest observed Brier score and highest AUC-ROC, while the log-trained model has the highest observed log score and lowest calibration error. Models with similar aggregate performance also reach that performance through different combinations of bias, information, and noise. Proper scoring rules therefore need not behave interchangeably as training objectives. Reward choice may shape not only how well an LLM forecasts, but how its forecasting errors are structured. Each condition uses a single seed, so some differences may reflect training stochasticity.