For ML practitioners working on prediction, forecasting, or decision-support systems: the central methodological claim in this post is worth understanding even as a proposal. Current AI forecasting benchmarks face a well-documented data-leakage problem -- if a model's training data includes web content from after the question was resolved, benchmark scores are inflated. The Metal Ivy post proposes using a static, timestamped cache of the internet as the training environment, so the RL agent can only "see" information available before each question's resolution date, cleanly separating training signal from future leakage.
What the post proposes
A Metal Ivy blog post, crossposted to LessWrong, argues that applying reinforcement learning to a large, historical cached internet could produce a superhuman forecaster. The proposed setup: train an RL agent to predict future events using only the subset of a web cache dated before each question's resolution, then reward it based on calibration and accuracy. The author argues this setup would allow a clean RL loop similar to those that produced superhuman performance in Go and chess -- applied to open-ended world-event forecasting rather than a constrained game.
Context -- the broader debate
The AI forecasting space has a contested track record. Several 2024-2025 papers claimed LLM-based forecasters rivaled or exceeded human forecasters; a prominent LessWrong critique ("Contra papers claiming superhuman AI forecasting") argues those claims rely on methodological problems including data leakage, non-representative question sets, and comparisons to weak human baselines. The cached-internet proposal attempts to address the leakage critique specifically, which is its main contribution relative to prior work.
Limitations and what to watch
The post is a conceptual proposal, not a research paper with empirical results. Building and maintaining a high-quality, timestamped web cache at the required scale is a substantial engineering challenge. Whether RL reward signals from forecasting are rich enough to drive the kind of capability gains seen in game-playing agents remains an open question. Practitioners interested in this direction should watch for follow-up empirical work testing whether the training loop produces the claimed generalization.
Key Points #
- 1What: A blog post proposes using reinforcement learning on a timestamped internet cache to train a superhuman forecaster, sidestepping the data-leakage problem in prior AI forecasting benchmarks.
- 2Why: Existing LLM forecasters face a well-documented leakage problem -- models trained on web data may have seen future outcomes; a static cache with date cutoffs would isolate training signal from resolved events.
- 3So what: The proposal is speculative and lacks empirical results, but it engages a real methodological gap; practitioners should track whether follow-up work tests the RL-on-cache approach.
Scoring Rationale #
A non-peer-reviewed blog proposal addressing a real methodological gap in AI forecasting (data leakage). The cached-internet RL framing is coherent and engages the LessWrong forecasting debate directly, but there are no empirical results. Appropriate for a speculative but substantive community post on a topic relevant to practitioners.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.