OpenAI no longer recommends SWE-Bench Pro

OpenAI announced it no longer recommends SWE-Bench Pro as a benchmark for evaluating coding models, citing concerns that the metric may not accurately reflect real-world software engineering performance. The decision follows internal analysis showing the benchmark's limitations in capturing meaningful improvements in AI-assisted coding.

Article URL: https://openai.com/index/separating-signal-from-noise-coding-evaluations/ Comments URL: https://news.ycombinator.com/item?id=48837396 Points: 4 Comments: 0