Higher SWE-Bench Scores Did Not Ship More Code: Fable 5 Ban Natural Experiment A 19-day U.S. export ban on Claude Fable 5 starting June 12, 2026 produced no detectable change in developer shipping output, according to an arXiv paper by Yining Hua posted November 17, 2026. Hua's difference-in-differences analysis of 297 developers who switched off Fable 5 and 8,908 who stayed on Opus 4.8 found a 7-day PR merge rate change of +1.4 percentage points (95% interval -1.4 to +4.3) and +2.4% merged PRs per week (-4.9% to +10.2%), neither significant, despite Fable 5 scoring roughly 11 points higher on SWE-Bench Pro. Five other 2026 model upgrades across 1,208 developers and 15,442 pull requests pooled to a -0.9 percentage point mean merge-rate change (-1.9 to +0.2), with all point estimates at or below zero. Higher leaderboard scores on coding benchmarks do not automatically mean your team merges more pull requests. That is the blunt conclusion of Higher-Scoring Coding Models Did Not Get More Work Done https://arxiv.org/abs/2611.23216v1?ref=explainx by Yining Hua , posted to arXiv on November 17, 2026 . The study treats a real policy shock — a 19-day U.S. export ban on Claude Fable 5 starting June 12, 2026 — as a natural experiment . Developers kept the same repos and habits; only the default model changed. Merge rates barely moved. What people search: does SWE-bench predict velocity? Searchers ask whether SWE-bench scores correlate with shipping speed . Hua's answer, from GitHub data rather than vendor tables: not detectably , for the Fable 5 suspension window and for five other 2026 upgrades in the same panel. | Metric | Fable 5 vs Opus 4.8 estimate | 95% interval | Significant? | |---|---|---|---| | 7-day PR merge rate | +1.4 pp | -1.4 to +4.3 | No | | Merged PRs per week | +2.4% | -4.9% to +10.2% | No | | Five upgrades pooled | -0.9 pp mean | -1.9 to +0.2 | No all ≤ 0 point estimates | Those intervals come from the paper abstract and methods summary on arXiv; reproduce them from the PDF before you cite them in a compliance doc. Why this paper lands after a year of benchmark marketing Review and merge decisions on agent-assisted pull requests Model cards and launch posts still lead with SWE-Bench and SWE-Bench Pro deltas. explainx.ai tracked the organizational gap in SWE-CoWork https://explainx.ai/blog/swe-cowork-coding-agents-digital-employees-benchmark-2026 , where top models exceed 62% fail-to-pass tests yet finish under 8% of full assignments SWE-CoWork paper https://arxiv.org/abs/2611.20218?ref=explainx . Hua states plainly in the abstract that no study had tested whether a higher-scoring coding model gets more real work done before this one. That is an empirical hole, not a rhetorical one. The export ban provides a rare exogenous switch: - Treatment group: 297 developers who had been on Claude Fable 5 and switched when access ended. - Control group: 8,908 developers who stayed on Opus 4.8 throughout. - Benchmark gap: Fable 5 was about 11 points higher on SWE-Bench Pro than Opus 4.8 per the paper's framing. Because people and projects were stable, the design aims to isolate model capability as marketed on benchmarks from team skill, repo difficulty, and seasonality — threats that plague casual before/after comparisons on X. Methods in plain language Issue-to-patch flow for SWE-style coding benchmarks versus merged pull requests Hua fits a difference-in-differences model with developer fixed effects and date fixed effects . Intuitively: compare how each developer's merge rate changed around the ban against how a similar calendar window behaved for developers who never lost Fable 5. The outcome is whether a pull request merges within seven days , not raw commit counts. That matches how many eng orgs define throughput for agent-assisted work, and it pairs with review norms we covered in Claude Code team review risk https://explainx.ai/blog/claude-code-team-code-review-process-ai-reviewer-risk-tiers-reverts-down-40-percent-2026 . The paper also runs five model upgrades before and after the suspension on 1,208 developers and 15,442 pull requests . All five point estimates for merge-rate lifts are negative or zero , with overlapping confidence intervals — the opposite of a clean "new model = faster shipping" story. What this does not prove Reasonable limits belong in the same paragraph as the headline: - Observational GitHub data may miss private forks, force-pushes, or squash merges that look slow but ship fast. - 19 days may be too short for large refactors that benchmarks reward but calendars delay. - Opus 4.8 as the fallback is one counterfactual; other routes GPT-6 Astra, local Qwen could behave differently. - Export bans correlate with news and anxiety , which might distract developers independent of model IQ. None of those caveats rescue the claim that +11 SWE-Bench Pro points automatically bought measurable merge acceleration in this sample. How this fits the November 2026 agent eval stack Read Hua alongside three threads explainx.ai already covers: 1. Organizational completeness: SWE-CoWork https://explainx.ai/blog/swe-cowork-coding-agents-digital-employees-benchmark-2026 shows high test pass rates with single-digit full-task completion. 2. Harness effects: Together agent evals https://explainx.ai/blog/together-ai-agent-evals-97-percent-coding-agents-november-2026 found 97% on narrow API-doc tasks — a reminder that task design moves scores. 3. Policy shocks: Fable-tier access already intersected export control posts https://explainx.ai/blog/congress-bipartisan-nvidia-china-chip-curbs-ndaa-cia-briefing-november-2026 and vendor frontier gating ; bans change which model runs, not just which badge marketing shows. If you run an agent pilot, the actionable shift is to pre-register production metrics — merge rate, rework, review hours, incident counts — before you let procurement pick on leaderboard rank alone. What developers should do this week For team leads: Add a merge-rate dashboard tagged by model and harness version. Compare against your SWE-bench internal slice , not the public leaderboard. For individual builders: When a model upgrades, run two weeks parallel on non-critical repos before you retire the old tier — mirror how loop engineering https://explainx.ai/blog/loop-engineering-coding-agents-claude-code-guide-2026 teams stage harness changes. For eval authors: Publish confidence intervals on business outcomes, not only point estimates on synthetic tasks. Hua's design is a template: find exogenous model switches bans, price hikes, enterprise defaults and measure GitHub or Jira , not just pass@k. Interpreting the Fable 5 counterfactual for procurement Procurement teams often treat +10 SWE-Bench Pro as a capacity contract — implicitly promising capacity planning that Hua's study does not support. During the June 2026 suspension , most displaced users landed on Opus 4.8 , a model the paper frames as roughly eleven points lower on that benchmark. If benchmark gaps translated linearly into merge probability , you would expect a visible jump when Fable 5 returned. The estimated +1.4 percentage point move on seven-day merge rate sits inside an interval that crosses zero. Weekly volume shows the same story: +2.4% merged PRs with an interval spanning -4.9% to +10.2% . That pattern fits harness-limited agent work more than raw reasoning-limited work. An agent that aces synthetic patches may still stall on: - Ambiguous acceptance criteria in real tickets - Cross-service failures that never appear in benchmark containers - Review latency when human owners disagree with generated architecture - CI flakes that burn the seven-day merge window Each failure mode appears in explainx.ai's organizational benchmarks and customer dispute reporting this quarter, independent of Hua's GitHub sample. Designing your own merge-rate study You do not need a federal export ban to replicate the spirit of this paper: 1. Pick an exogenous switch: enterprise default model changes, sudden price hikes, or security blocks on a tier. 2. Freeze repo cohorts: compare developers on the same services before and after, not unrelated teams. 3. Pre-register outcomes: seven-day merge rate, time-to-first-review, revert rate, and escaped defect counts. 4. Pair with benchmark slices: run the same internal SWE-bench subset monthly so you can see divergence between leaderboard deltas and delivery metrics. Document the harness version beside the model name. Revolut's Codex fincrime story https://explainx.ai/blog/revolut-openai-codex-fincrime-gpt-5-5-enterprise-story-2026 shows enterprise wins come from workflow integration , not leaderboard rank alone. Connection to Claude export politics The June 12, 2026 Fable suspension sits in the same policy summer as intensified chip export debates covered in Nvidia Megaspeed threads https://explainx.ai/blog/nvidia-huang-megaspeed-alice-huang-full-story-timeline-what-happened-2026 . For builders, the lesson is dual: geopolitics can remove your best model overnight , and the fallback may not slow you down if bottlenecks live in process, not logits. Plan fallback tiers and merge dashboards together — not only API routing tables. FAQ alignment with search wording Is SWE bench good for picking a coding model? It is good for regression testing on isolated issues. Hua's evidence says it is insufficient alone for predicting merged PR velocity across hundreds of developers. Why did my team not speed up after a benchmark winner? Plausible reasons include review bottlenecks , ambiguous tickets , flaky CI , and agents that patch tests without finishing requirements — the same failure modes SWE-CoWork names explicitly. Reading the five upgrade panel Beyond the Fable ban, Hua repeats the diff-in-diff design across five unrelated model upgrades in 2026 on 1,208 developers and 15,442 pull requests . Every upgrade-specific merge-rate estimate is negative or zero in the abstract, with overlapping confidence intervals . That uniformity matters: a single weird fortnight around export politics could be noise, but a panel of upgrades failing to show merge lifts pushes responsibility back to process metrics . Teams should log review rounds per PR , percent AI-authored lines reverted , and escapes to production beside SWE-bench internal slices. Vendor marketing will keep citing point deltas . This paper supplies the counter-question: did merged output move? If not, negotiate pilots on merge SLAs , not leaderboard slides. For finance and platform owners: attach agent spend reports to the same dashboard. Microsoft Agent 365 analytics at Ignite https://explainx.ai/blog/microsoft-ignite-2026-complete-coverage-every-announcement-november-2026 and Perplexity orchestrator pricing https://explainx.ai/blog/perplexity-orchestrator-api-45-models-november-2026 show the industry moving toward outcome-based AI bills . Merge-rate flatlines mean token invoices can rise without delivery gains — exactly the risk enterprise FinOps teams warned about in November. Related reading Hua releases four figures and twenty pages in the arXiv PDF; cite effect sizes from there before you forward screenshots. The abstract alone is enough for the headline, not for board-level ROI math. Reading gets you the idea. Building it with feedback gets you the skill.