OpenAI just found ~30% of SWE-Bench Pro is broken — and retracted their own recommendation

OpenAI retracted its recommendation of SWE-Bench Pro after an audit found roughly 30% of its 731 tasks are broken. The audit, combining AI investigator agents and human reviewers, identified 200–249 broken tasks with four dominant failure patterns. OpenAI used Codex-based agents to inspect repo history, execute tests, and analyze failure traces, highlighting that evaluation flaws are becoming easier to detect as model capabilities improve.

OpenAI pulled the plug on SWE-bench Verified earlier this year after finding contamination and design issues. Their replacement recommendation: SWE-Bench Pro. That one just failed its own audit. In a new writeup, OpenAI's research team reports that roughly 30% of SWE-Bench Pro's 731 tasks are broken. The evaluation — designed to test agentic coding on realistic, longer-horizon tasks — has flaws severe enough that OpenAI is now retracting their earlier endorsement. "Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro." The audit combined AI investigator agents and human reviewers five engineers per flagged task . They identified 200–249 broken tasks depending on method. Four failure patterns dominated: The structural cause: these benchmarks are built from real GitHub pull requests. Human-to-human PR collaboration doesn't produce clean, isolated tasks. Tests written to validate a specific contributor's PR aren't the same as tests designed to measure model capability. OpenAI used Codex-based investigator agents to run this audit — inspecting repo history, executing tests, analysing failure traces at scale. "Evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, and edge cases with much greater depth and consistency." The benchmarks used to measure model progress are now being audited by those same models. That loop is new. It matters. The coding eval landscape is in a rough patch. The community's best tools for measuring agentic coding progress keep failing quality checks — and the replacement cycle is accelerating. Worth watching whether Scale AI who runs SWE-Bench Pro responds with a revised dataset. Source: Separating signal from noise in coding evaluations — OpenAI https://openai.com/index/separating-signal-from-noise-coding-evaluations/ ✏️ Drafted with KewBot AI , edited and approved by Drew.