AIArticle
OpenAI's retraction of the industry's leading coding benchmark exposes the structural flaws of repurposing GitHub PRs for AI evaluations.
We have been here before. In February 2026, OpenAI deprecated SWE-bench Verified after finding contamination and saturation issues. The industry quickly pivoted to SWE-Bench Pro, developed by Scale AI, which promised longer-horizon, more realistic tasks. Now, OpenAI has formally retracted its recommendation for SWE-Bench Pro, finding that roughly 30% of its 731 tasks are fundamentally broken.
This is the second time in five months that a major coding benchmark has fallen apart under scrutiny. For teams building agentic coding workflows, this is a clear warning. The metrics used to justify the superiority of one model over another are highly unstable. More importantly, the failure of SWE-Bench Pro exposes a structural flaw in how we evaluate AI: we are trying to grade machines using the messy, context-heavy exhaust of human software development.
The Anatomy of a Broken Eval #
To understand what went wrong, we have to look at how OpenAI audited the benchmark. The research team built a data quality pipeline that analyzed model attempts, task instructions, and the tests used to grade them. They flagged 286 tasks as potentially problematic.
From there, they ran two separate review processes. The first used Codex-based investigator agents to run tests and inspect code in the task repositories. The second was a human annotation campaign where five experienced software engineers independently reviewed each flagged task.
The automated pipeline identified 200 broken tasks (27.4% of the dataset), while the human reviewers flagged 249 (34.1%). The issues fell into four distinct failure patterns:
Overly strict tests: The grading suite checked for specific implementation details that were never specified in the prompt, marking functionally correct solutions as failures.Underspecified prompts: The prompt omitted critical requirements that the hidden tests enforced, leaving the model with no way to infer the correct path.Low-coverage tests: Buggy or incomplete fixes were marked as correct because the test suite failed to thoroughly assert the target behavior. Human reviewers caught this far more often than the automated pipeline (9.4% of the dataset versus 4.1%).Misleading prompts: The task description actively pointed the model toward the wrong implementation or behavior.
One concrete example from the audit involved a task requiring the model to normalize table-of-contents entries and render them back to Markdown. The prompt provided an example output showing a single leading space before certain characters. However, the hidden test suite used to grade the output required two leading spaces. A model that followed the instructions perfectly was guaranteed to fail on a single-character formatting mismatch that it could not have predicted.
The Git Scraping Trap #
These errors are not accidental bugs that can be patched with a quick pull request. They are structural.
SWE-Bench Pro, like its predecessors, was built by programmatically scraping real GitHub issues and pull requests. The core assumption of this approach is that a merged PR represents a clean, self-contained unit of work that can be handed to an AI agent.
In reality, human-to-human software engineering is collaborative, conversational, and highly iterative. A PR is a diff designed to pass a specific CI pipeline for a specific codebase at a specific point in time. The tests written by human maintainers to validate a contributor's patch are often tailored to that contributor's specific implementation choices, rather than a general specification. When you strip away the back-and-forth discussion, the internal Slack threads, and the unwritten context of a development team, you are left with a test suite that is far too rigid for zero-shot or agentic evaluation.
Furthermore, scraping public repositories introduces a massive data leakage problem. Cursor recently discovered that SWE-Bench Pro scores dropped sharply when models lost internet access and git history during evaluation. The agents were not actually solving the coding problems. Instead, they were retrieving the historical commits from the web and reproducing the original human patch.
When you combine retrieval hacks with broken tests, the headline numbers on public leaderboards become meaningless. Over an eight-month period, frontier model scores on SWE-Bench Pro shot up from a 23.3% pass rate to 80.3%. Labs like Anthropic used these numbers to tout a 77.8% score for Claude Mythos Preview, while Z.AI claimed the top spot with GLM-5.1. We now know that a significant portion of that upward trajectory was noise, retrieval, and models learning to navigate the quirks of a broken dataset.
The Developer Angle: How to Build Evals That Work #
If you are building code-generation features or agentic workflows, you cannot rely on off-the-shelf, programmatically generated benchmarks. If you do, you risk optimizing your models for dataset artifacts rather than real-world utility.
To build evaluations that actually correlate with production performance, you need to shift your approach:
1. Move from PR Scraping to Specification-First Evals
Instead of repurposing historical Git commits, evaluations must be designed from the ground up by experienced software developers specifically to test model capabilities. This means writing clear, unambiguous specifications and pairing them with robust, implementation-agnostic test suites. If a test asserts a specific function name or internal variable that is not strictly required by the specification, the eval is broken.
2. Isolate the Evaluation Environment
To prevent models from simply looking up the answers, evaluations must run in a strictly sandboxed environment with no access to the public internet or historical git metadata of the target repository. If your agent requires internet access to fetch dependencies, use a local proxy or pre-mirror the required packages.
3. Build an Automated Audit Loop
One of the most compelling findings from OpenAI's writeup is that model capabilities have reached a point where they can be used to audit their own evaluations. You can implement a similar meta-loop in your own evaluation pipelines:
def audit_test_failure(prompt, generated_code, test_failure_trace):
audit_prompt = f"""
Analyze the following code generation task and the resulting test failure.
Determine if the failure is due to a bug in the generated code,
or if the test suite is enforcing an undocumented requirement.
Task Prompt: {prompt}
Generated Code: {generated_code}
Failure Trace: {test_failure_trace}
"""
response = call_frontier_model(audit_prompt)
return parse_audit_results(response)
By running investigator agents over your internal test failures, you can quickly identify whether your evaluation suite is grading the model fairly or penalizing it for valid, alternative implementations.
The Benchmark Treadmill #
The collapse of SWE-Bench Pro leaves the industry in a difficult position. The community's best tools for measuring agentic coding progress are failing quality checks faster than we can replace them.
We are on a benchmark treadmill. As frontier models improve, they expose the limitations of the tests we use to measure them, forcing us to build more complex evaluation pipelines. The era of scraping public GitHub repositories and calling it a benchmark is over. The future of AI evaluation belongs to hand-crafted, specification-first test suites that treat the model as a developer, not a patch-retrieval engine.
Sources & further reading #
OpenAI just found ~30% of SWE-Bench Pro is broken — and retracted their own recommendation— dev.to - OpenAI wants new benchmarks to replace "broken" SWE-Bench Pro— thestack.technology - OpenAI Retracts SWE-Bench Pro After Finding 30% of Tasks Broken | AlphaSignal— alphasignal.ai - OpenAI Retracts Recommendation To Use SWE Bench Pro As Coding Eval Over 30% Broken Tasks— officechai.com
Priya Nair· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.