OpenAI pulled the plug on SWE-bench Verified earlier this year after finding contamination and design issues. Their replacement recommendation: SWE-Bench Pro. That one just failed its own audit.
In a new writeup, OpenAI's research team reports that roughly 30% of SWE-Bench Pro's 731 tasks are broken. The evaluation — designed to test agentic coding on realistic, longer-horizon tasks — has flaws severe enough that OpenAI is now retracting their earlier endorsement.
"Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro."
The audit combined AI investigator agents and human reviewers (five engineers per flagged task). They identified 200–249 broken tasks depending on method. Four failure patterns dominated:
The structural cause: these benchmarks are built from real GitHub pull requests. Human-to-human PR collaboration doesn't produce clean, isolated tasks. Tests written to validate a specific contributor's PR aren't the same as tests designed to measure model capability.
OpenAI used Codex-based investigator agents to run this audit — inspecting repo history, executing tests, analysing failure traces at scale.
"Evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, and edge cases with much greater depth and consistency."
The benchmarks used to measure model progress are now being audited by those same models. That loop is new. It matters.
The coding eval landscape is in a rough patch. The community's best tools for measuring agentic coding progress keep failing quality checks — and the replacement cycle is accelerating. Worth watching whether Scale AI (who runs SWE-Bench Pro) responds with a revised dataset.
Source: Separating signal from noise in coding evaluations — OpenAI ✏️ Drafted with KewBot (AI), edited and approved by Drew.