Real-SWE ran frontier models against licensed, private enterprise codebases (billing, tax, multi-service work) and one number jumped out at me: rollout duration barely moves resolution.
71.4% of rollouts that finished in under 10 minutes FAILED. 73.4% of rollouts that ran 10 minutes or longer also FAILED. Pass rate sits flat at 27-29% either way. Extending runtime from minutes to long rollouts shifts the outcome by about two percentage points, which is noise.
The leader, Fable 5.1 on Claude Code, only lands 38.8% resolution. GPT-6 Astra on Codex CLI gets 33.8%. The top model still fails roughly six out of ten private enterprise tasks.
This is the part vendor demos skip. The easy stuff gets solved fast, so on a short rollout you see high apparent throughput. But the tasks that matter, the ones buried in real payroll and tax and integration code, hit a structural wall. The agent doesn't run out of compute on those. It runs out of understanding, or context, or the harness doesn't give it the right entry point. Another ten minutes of looped retries doesn't fix any of that.
The other thing Real-SWE does right is treat each score as model+harness, not model alone. Fable 5.1 is only 38.8% paired with Claude Code's scaffold. That's a harness result on private code, not a statement about the model in a vacuum. So many leaderboards still publish model names with no harness pinned, and then people compare them across totally different scaffolding and draw nonsense conclusions.
Takeaway for anyone buying an agent: when a vendor shows you a pass rate, ask which slice that number came from. A model that looks great because it clears the fast, shallow tasks is hiding the exact set you actually need it to solve.
Benchmark source: withspecific.com/benchmarks/real-swe