Higher leaderboard scores on coding benchmarks do not automatically mean your team merges more pull requests. That is the blunt conclusion of Higher-Scoring Coding Models Did Not Get More Work Done by Yining Hua, posted to arXiv on November 17, 2026.
The study treats a real policy shock — a 19-day U.S. export ban on Claude Fable 5 starting June 12, 2026 — as a natural experiment. Developers kept the same repos and habits; only the default model changed. Merge rates barely moved.
What people search: does SWE-bench predict velocity? #
Searchers ask whether SWE-bench scores correlate with shipping speed. Hua's answer, from GitHub data rather than vendor tables: not detectably, for the Fable 5 suspension window and for five other 2026 upgrades in the same panel.
| Metric | Fable 5 vs Opus 4.8 estimate | 95% interval | Significant? |
|---|---|---|---|
| 7-day PR merge rate | +1.4 pp | -1.4 to +4.3 | No |
| Merged PRs per week | +2.4% | -4.9% to +10.2% | No |
| Five upgrades (pooled) | -0.9 pp mean | -1.9 to +0.2 | No (all ≤ 0 point estimates) | Those intervals come from the paper abstract and methods summary on arXiv; reproduce them from the PDF before you cite them in a compliance doc.
Why this paper lands after a year of benchmark marketing #
Review and merge decisions on agent-assisted pull requests
Model cards and launch posts still lead with SWE-Bench and SWE-Bench Pro deltas. explainx.ai tracked the organizational gap in SWE-CoWork, where top models exceed 62% fail-to-pass tests yet finish under 8% of full assignments (SWE-CoWork paper).
Hua states plainly in the abstract that no study had tested whether a higher-scoring coding model gets more real work done before this one. That is an empirical hole, not a rhetorical one.
The export ban provides a rare exogenous switch:
- Treatment group:**297 developers who had been onClaude Fable 5** and switched when access ended.
- Control group:**8,908 developers who stayed onOpus 4.8** throughout.
- Benchmark gap: Fable 5 was about11 points higher on SWE-Bench Pro than Opus 4.8 per the paper's framing.
Because people and projects were stable, the design aims to isolate model capability as marketed on benchmarks from team skill, repo difficulty, and seasonality — threats that plague casual before/after comparisons on X.
Methods in plain language #
Issue-to-patch flow for SWE-style coding benchmarks versus merged pull requests
Hua fits a difference-in-differences model with developer fixed effects and date fixed effects. Intuitively: compare how each developer's merge rate changed around the ban against how a similar calendar window behaved for developers who never lost Fable 5.
The outcome is whether a pull request merges within seven days, not raw commit counts. That matches how many eng orgs define throughput for agent-assisted work, and it pairs with review norms we covered in Claude Code team review risk.
The paper also runs five model upgrades before and after the suspension on 1,208 developers and 15,442 pull requests. All five point estimates for merge-rate lifts are negative or zero, with overlapping confidence intervals — the opposite of a clean "new model = faster shipping" story.
What this does not prove #
Reasonable limits belong in the same paragraph as the headline:
- Observational GitHub data may miss private forks, force-pushes, or squash merges that look slow but ship fast.
- 19 days may be too short for large refactors that benchmarks reward but calendars delay.
- Opus 4.8 as the fallback is one counterfactual; other routes (GPT-6 Astra, local Qwen) could behave differently.
- Export bans correlate withnews and anxiety , which might distract developers independent of model IQ.
None of those caveats rescue the claim that +11 SWE-Bench Pro points automatically bought measurable merge acceleration in this sample.
How this fits the November 2026 agent eval stack #
Read Hua alongside three threads explainx.ai already covers:
- Organizational completeness:SWE-CoWork shows high test pass rates with** single-digit full-task** completion.
- Harness effects:Together agent evals found** 97%** on narrow API-doc tasks — a reminder thattask design moves scores.
- Policy shocks: Fable-tier access already intersectedexport control posts and vendorfrontier gating ; bans changewhich model runs, not justwhich badge marketing shows.
If you run an agent pilot, the actionable shift is to pre-register production metrics — merge rate, rework, review hours, incident counts — before you let procurement pick on leaderboard rank alone.
What developers should do this week #
For team leads: Add a merge-rate dashboard tagged by model and harness version. Compare against your SWE-bench internal slice, not the public leaderboard.
For individual builders: When a model upgrades, run two weeks parallel on non-critical repos before you retire the old tier — mirror how loop engineering teams stage harness changes.
For eval authors: Publish confidence intervals on business outcomes, not only point estimates on synthetic tasks. Hua's design is a template: find exogenous model switches (bans, price hikes, enterprise defaults) and measure GitHub or Jira, not just pass@k.
Interpreting the Fable 5 counterfactual for procurement #
Procurement teams often treat +10 SWE-Bench Pro as a capacity contract — implicitly promising capacity planning that Hua's study does not support. During the June 2026 suspension, most displaced users landed on Opus 4.8, a model the paper frames as roughly eleven points lower on that benchmark.
If benchmark gaps translated linearly into merge probability, you would expect a visible jump when Fable 5 returned. The estimated +1.4 percentage point move on seven-day merge rate sits inside an interval that crosses zero. Weekly volume shows the same story: +2.4% merged PRs with an interval spanning -4.9% to +10.2%. That pattern fits harness-limited agent work more than raw reasoning-limited work. An agent that aces synthetic patches may still stall on:
- Ambiguous acceptance criteria in real tickets
- Cross-service failures that never appear in benchmark containers
- Review latency when human owners disagree with generated architecture
- CI flakes that burn the seven-day merge window
Each failure mode appears in explainx.ai's organizational benchmarks and customer dispute reporting this quarter, independent of Hua's GitHub sample.
Designing your own merge-rate study #
You do not need a federal export ban to replicate the spirit of this paper:
- Pick an exogenous switch: enterprise default model changes, sudden price hikes, or security blocks on a tier.
- Freeze repo cohorts: compare developers on the same services before and after, not unrelated teams.
- Pre-register outcomes: seven-day merge rate, time-to-first-review, revert rate, and escaped defect counts.
- Pair with benchmark slices: run the same internal SWE-bench subset monthly so you can seedivergence between leaderboard deltas and delivery metrics.
Document the harness version beside the model name. Revolut's Codex fincrime story shows enterprise wins come from workflow integration, not leaderboard rank alone.
Connection to Claude export politics #
The June 12, 2026 Fable suspension sits in the same policy summer as intensified chip export debates covered in Nvidia Megaspeed threads. For builders, the lesson is dual: geopolitics can remove your best model overnight, and the fallback may not slow you down if bottlenecks live in process, not logits.
Plan fallback tiers and merge dashboards together — not only API routing tables.
FAQ alignment with search wording #
Is SWE bench good for picking a coding model? It is good for regression testing on isolated issues. Hua's evidence says it is insufficient alone for predicting merged PR velocity across hundreds of developers.
Why did my team not speed up after a benchmark winner? Plausible reasons include review bottlenecks, ambiguous tickets, flaky CI, and agents that patch tests without finishing requirements — the same failure modes SWE-CoWork names explicitly.
Reading the five upgrade panel #
Beyond the Fable ban, Hua repeats the diff-in-diff design across five unrelated model upgrades in 2026 on 1,208 developers and 15,442 pull requests. Every upgrade-specific merge-rate estimate is negative or zero in the abstract, with overlapping confidence intervals.
That uniformity matters: a single weird fortnight around export politics could be noise, but a panel of upgrades failing to show merge lifts pushes responsibility back to process metrics. Teams should log review rounds per PR, percent AI-authored lines reverted, and escapes to production beside SWE-bench internal slices.
Vendor marketing will keep citing point deltas. This paper supplies the counter-question: did merged output move? If not, negotiate pilots on merge SLAs, not leaderboard slides.
For finance and platform owners: attach agent spend reports to the same dashboard. Microsoft Agent 365 analytics at Ignite and Perplexity orchestrator pricing show the industry moving toward outcome-based AI bills. Merge-rate flatlines mean token invoices can rise without delivery gains — exactly the risk enterprise FinOps teams warned about in November.
Related reading #
Hua releases four figures and twenty pages in the arXiv PDF; cite effect sizes from there before you forward screenshots. The abstract alone is enough for the headline, not for board-level ROI math.
Reading gets you the idea. Building it with feedback gets you the skill.