{"slug": "flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed", "title": "Flawed benchmarks (epoch.ai benchmark registry filtered by flawed)", "summary": "Epoch AI maintains a registry of 85 AI benchmarks spanning mathematics, software engineering, agentic workflows, and games, and has flagged a subset as flawed. The registry includes benchmarks such as a long-horizon coding test in which models reimplement entire programs without access to the original source code, a 100-puzzle chess set generated programmatically with a chess engine, and 1,000 factoid questions covering politics, science and technology, art, sports, geography, and music. The registry also aggregates many benchmarks into a single general capability scale and lists collections of unsolved research mathematics problems, including problems posed by mathematician Paul Erdős that remained open as of August 2026.", "body_md": "A registry of 85 AI benchmarks covering mathematics, software engineering, agentic workflows, games, and more.\n\nAn index aggregating many different benchmarks into a single, general capability scale.\n\nA collection of unsolved problems from research mathematics, curated to be especially interesting and difficult.\n\nA collection of problems posed or studied by mathematician Paul Erdős, open as of August 2026 and curated to be especially interesting and difficult.\n\nA long-horizon coding benchmark. Models reimplement entire programs end-to-end, without access to the original source code.\n\nA test of AI systems' learning capabilities, measuring whether their scores improve across repeated playthroughs of the relatively obscure game Earthborne Rangers.\n\n100 positions from a puzzle-oriented variant of a well-known game, the details of which are deliberately left undisclosed.\n\n100 novel puzzles, generated programmatically with a chess engine. Each puzzle has a single best next move.\n\n1,000 factoid questions about politics, science and technology, art, sports, geography, music, and more.\n\nExpert-written math problems covering advanced undergrad through early career research.", "url": "https://wpnews.pro/news/flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed", "canonical_source": "https://epoch.ai/benchmarks/search", "published_at": "2026-09-18 18:16:18+00:00", "updated_at": "2026-09-18 18:25:23.667194+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-agents", "ai-tools"], "entities": ["Epoch AI", "Paul Erdős", "Earthborne Rangers"], "alternates": {"html": "https://wpnews.pro/news/flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed", "markdown": "https://wpnews.pro/news/flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed.md", "text": "https://wpnews.pro/news/flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed.txt", "jsonld": "https://wpnews.pro/news/flawed-benchmarks-epoch-ai-benchmark-registry-filtered-by-flawed.jsonld"}}