{"slug": "ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state", "title": "Ai2's StateBench shows frontier models losing track after thousands of state changes", "summary": "Researchers Jackson Petty, William Merrill and Ashish Sabharwal at the Allen Institute for AI (Ai2) released StateBench on November 3, a benchmark testing whether language models can track information as it changes across long documents. Frontier models scored 13% or lower on the longest general finite tasks, while directly trained Gated DeltaNet and hybrid models generalized far beyond their training sequences. The benchmark treats retrieval as only the easiest of four levels of state tracking, and the results bear on architecture choices, pretraining data and products that rely on models following code, logs or agent state over time.", "body_md": "# Ai2's StateBench shows frontier models losing track after thousands of state changes\n\n**Ai2 researchers Jackson Petty, William Merrill and Ashish Sabharwal tested four levels of state tracking; frontier models scored 13% or lower on the longest general finite tasks.**\n\n        By [RuntimeWire Staff](https://runtimewire.com/author/runtimewire-staff)\n        · Published \n\nPrimary source: [Hugging Face Newsroom](https://huggingface.co/blog/allenai/statebench)\n\n## Why it matters\n\nLong context claims usually rest on retrieval, which StateBench treats as only the easiest level of tracking. The benchmark shows frontier models failing dependent updates at long lengths, while directly trained Gated DeltaNet and hybrid models generalized far beyond their training sequences. That distinction affects architecture choices, pretraining data and any product that depends on a model following code, logs or agent state over time.\n\nJackson Petty, [William Merrill (@lambdaviking)](https://x.com/lambdaviking?ref=runtimewire) and Ashish Sabharwal released [StateBench](https://github.com/allenai/state-bench?ref=runtimewire) on November 3rd. The benchmark tests whether a language model can follow information as it changes across a long document and where it starts losing the thread.…\n\nWhen you purchase through links in our articles, [we may earn a small commission](https://runtimewire.com/affiliate-monetization-standards). This doesn’t affect our editorial independence.", "url": "https://wpnews.pro/news/ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state", "canonical_source": "https://runtimewire.com/article/ai2-statebench-frontier-models-state-tracking", "published_at": "2026-11-03 17:38:29+00:00", "updated_at": "2026-11-03 20:49:24.892245+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "machine-learning", "artificial-intelligence"], "entities": ["Allen Institute for AI", "StateBench", "Jackson Petty", "William Merrill", "Ashish Sabharwal", "Gated DeltaNet", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state", "markdown": "https://wpnews.pro/news/ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state.md", "text": "https://wpnews.pro/news/ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state.txt", "jsonld": "https://wpnews.pro/news/ai2-s-statebench-shows-frontier-models-losing-track-after-thousands-of-state.jsonld"}}