{"slug": "openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken", "title": "OpenAI finds roughly 30 percent of popular AI coding test is broken", "summary": "OpenAI reviewed the SWE-Bench Pro AI coding test and found roughly 30% of its tasks are flawed, leading the company to withdraw its endorsement. The problems include overly strict or vague task requirements that skew AI performance assessments. OpenAI is calling for more reliable benchmarks after discovering that some models could cheat by copying solutions from project histories.", "body_md": "# OpenAI finds roughly 30 percent of popular AI coding test is broken\n\n## Key Points\n\n- OpenAI is pulling its endorsement of the AI coding test SWE-Bench Pro after a review found roughly 30 percent of its tasks are flawed.\n- The problems stem from the tasks being pulled from real software projects, making them too strict, too vague, or misleading for AI models. That skews the assessment of what AI can actually do.\n- OpenAI is calling for more reliable benchmarks. Artificial Analysis had already dropped the test from its rankings after finding that some models copied solutions from project commit histories instead of solving the tasks.\n\n**OpenAI reviewed SWE-Bench Pro, a widely used test for measuring AI models' programming skills, and found roughly 30 percent of its tasks are broken. The company is pulling its earlier endorsement of the benchmark.**\n\nResults from tests like these feed into decisions about whether and how to release a model, including safety assessments under OpenAI's Preparedness Framework. When a test contains errors, it can paint a misleading picture of what an AI can actually do.\n\nTo run the review, OpenAI first deployed an automated screening tool that flagged 286 suspicious tasks. AI agents built on Codex then examined each case in detail before a human researcher made the final call. That process labeled 200 tasks (27.4 percent) as flawed. In a parallel review, five experienced software developers evaluated the same cases and flagged even more, 249 tasks (34.1 percent). The human reviewers were stricter than the AI agents, though both sides agreed in 74 percent of cases.\n\n## A single whitespace character can mean pass or fail\n\nOpenAI breaks the problems into four categories. Some tests are too strict, rejecting solutions that actually work. Others are too vague, expecting the AI to meet requirements buried in hidden test cases. Some tests are too shallow, letting incomplete solutions pass. And some task descriptions simply point in the wrong direction. One example from the OpenLibrary project: the task description called for a single space, but the hidden test expected two. An AI that correctly followed the instructions would fail.\n\nThe tasks were pulled from the commit histories of real software projects, originally written for human collaboration, not designed as clean evaluation tasks for AI models. According to OpenAI, tests from those projects tend to be too strict because they were built to verify one specific change, not to serve as general-purpose requirements. On the public version of the test with 731 tasks, top models had jumped from 23.3 to 80.3 percent accuracy in just eight months. SWE-Bench Pro was meant to replace the older SWE-bench Verified, which [OpenAI had already dismissed for similar reasons](https://the-decoder.com/openai-wants-to-retire-the-ai-coding-benchmark-that-everyone-has-been-competing-on/).\n\nThis time, OpenAI doesn't recommend a specific replacement. The company simply calls on the industry to build new benchmarks using experienced developers, ones that are hard to game, trustworthy, and actually meaningful.\n\nIn mid-June, the analytics firm Artificial Analysis had already [removed SWE-Bench Pro from its Coding Agent Index](https://x.com/ArtificialAnlys/status/2065328920514515037) and swapped in DeepSWE, a test from Datacurve. The reason: SWE-Bench Pro was gameable. Some models had copied the correct solution from a project's commit history instead of actually solving the task.\n\nThe switch reshuffled the leaderboard. Codex with GPT-5.5 (xhigh) climbed from 65 to 76 points and passed Claude Code with Opus 4.8 (max) at 73, while Claude Code with Fable 5 (max) took the top spot at 77 points. On SWE-Bench Pro, Codex with GPT-5.5 had scored just 31 points, compared to 64 to 84 on other tests.\n\n```\nAI News Without the Hype – Curated by Humans\n\n\t\t\t\t\tSubscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive \"AI Radar\" frontier report six times a year, full archive access, and access to our comment section.\t\t\t\t\n\n\t\t\t\t\tSubscribe now\n```\n\n[OpenAI](https://openai.com/index/separating-signal-from-noise-coding-evaluations/)|\n\n[Artificial Analysis / Code-Ranking](https://x.com/ArtificialAnlys/status/2065328920514515037)", "url": "https://wpnews.pro/news/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken", "canonical_source": "https://the-decoder.com/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken/", "published_at": "2026-07-09 13:23:57+00:00", "updated_at": "2026-07-09 13:40:38.553009+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-safety", "ai-policy"], "entities": ["OpenAI", "SWE-Bench Pro", "Codex", "Artificial Analysis", "DeepSWE", "Datacurve", "GPT-5.5", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken", "markdown": "https://wpnews.pro/news/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken.md", "text": "https://wpnews.pro/news/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken.txt", "jsonld": "https://wpnews.pro/news/openai-finds-roughly-30-percent-of-popular-ai-coding-test-is-broken.jsonld"}}