# Reinforcement Learning Creates a Superhuman Forecaster > Source: > Published: 2026-06-28 19:37:48+00:00 For ML practitioners working on prediction, forecasting, or decision-support systems: the central methodological claim in this post is worth understanding even as a proposal. Current AI forecasting benchmarks face a well-documented data-leakage problem -- if a model's training data includes web content from after the question was resolved, benchmark scores are inflated. The Metal Ivy post proposes using a static, timestamped cache of the internet as the training environment, so the RL agent can only "see" information available before each question's resolution date, cleanly separating training signal from future leakage. ### What the post proposes A Metal Ivy blog post, crossposted to LessWrong, argues that applying reinforcement learning to a large, historical cached internet could produce a superhuman forecaster. The proposed setup: train an RL agent to predict future events using only the subset of a web cache dated before each question's resolution, then reward it based on calibration and accuracy. The author argues this setup would allow a clean RL loop similar to those that produced superhuman performance in Go and chess -- applied to open-ended world-event forecasting rather than a constrained game. ### Context -- the broader debate The AI forecasting space has a contested track record. Several 2024-2025 papers claimed LLM-based forecasters rivaled or exceeded human forecasters; a prominent LessWrong critique ("Contra papers claiming superhuman AI forecasting") argues those claims rely on methodological problems including data leakage, non-representative question sets, and comparisons to weak human baselines. The cached-internet proposal attempts to address the leakage critique specifically, which is its main contribution relative to prior work. ### Limitations and what to watch The post is a conceptual proposal, not a research paper with empirical results. Building and maintaining a high-quality, timestamped web cache at the required scale is a substantial engineering challenge. Whether RL reward signals from forecasting are rich enough to drive the kind of capability gains seen in game-playing agents remains an open question. Practitioners interested in this direction should watch for follow-up empirical work testing whether the training loop produces the claimed generalization. ## Key Points - 1What: A blog post proposes using reinforcement learning on a timestamped internet cache to train a superhuman forecaster, sidestepping the data-leakage problem in prior AI forecasting benchmarks. - 2Why: Existing LLM forecasters face a well-documented leakage problem -- models trained on web data may have seen future outcomes; a static cache with date cutoffs would isolate training signal from resolved events. - 3So what: The proposal is speculative and lacks empirical results, but it engages a real methodological gap; practitioners should track whether follow-up work tests the RL-on-cache approach. ## Scoring Rationale A non-peer-reviewed blog proposal addressing a real methodological gap in AI forecasting (data leakage). The cached-internet RL framing is coherent and engages the LessWrong forecasting debate directly, but there are no empirical results. Appropriate for a speculative but substantive community post on a topic relevant to practitioners. Practice interview problems based on real data 1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with. [Try 250 free problems](/problems)