# Higher SWE-Bench Scores Did Not Ship More Code: Fable 5 Ban Natural Experiment

> Source: <https://explainx.ai/blog/higher-scoring-coding-models-swe-bench-merge-rate-natural-experiment-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-18 00:00:00+00:00

**Recommendation:** Validate coding-model upgrades against your own merge-rate or task-completion data rather than SWE-Bench deltas; this study found an 11-point benchmark gap produced no significant throughput change.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

**Higher leaderboard scores on coding benchmarks do not automatically mean your team merges more pull requests.** That is the blunt conclusion of [Higher-Scoring Coding Models Did Not Get More Work Done](https://arxiv.org/abs/2611.23216v1?ref=explainx) by **Yining Hua**, posted to arXiv on **November 17, 2026**.

The study treats a real policy shock — a **19-day U.S. export ban on Claude Fable 5** starting **June 12, 2026** — as a **natural experiment**. Developers kept the same repos and habits; only the default model changed. **Merge rates barely moved.**

## What people search: does SWE-bench predict velocity?

Searchers ask whether **SWE-bench scores** correlate with **shipping speed**. Hua's answer, from GitHub data rather than vendor tables: **not detectably**, for the Fable 5 suspension window and for five other 2026 upgrades in the same panel.

| Metric | Fable 5 vs Opus 4.8 estimate | 95% interval | Significant? | 
|---|---|---|---|
| 7-day PR merge rate | +1.4 pp | -1.4 to +4.3 | No | 
| Merged PRs per week | +2.4% | -4.9% to +10.2% | No | 
| Five upgrades (pooled) | -0.9 pp mean | -1.9 to +0.2 | No (all ≤ 0 point estimates) | 

Those intervals come from the paper abstract and methods summary on arXiv; reproduce them from the PDF before you cite them in a compliance doc.

## Why this paper lands after a year of benchmark marketing

Review and merge decisions on agent-assisted pull requests

Model cards and launch posts still lead with **SWE-Bench** and **SWE-Bench Pro** deltas. explainx.ai tracked the organizational gap in [SWE-CoWork](https://explainx.ai/blog/swe-cowork-coding-agents-digital-employees-benchmark-2026), where top models exceed **62% fail-to-pass** tests yet finish **under 8%** of full assignments ([SWE-CoWork paper](https://arxiv.org/abs/2611.20218?ref=explainx)).

Hua states plainly in the abstract that **no study had tested** whether a higher-scoring coding model gets **more real work done** before this one. That is an empirical hole, not a rhetorical one.

The export ban provides a rare **exogenous** switch:

- **Treatment group:****297** developers who had been on**Claude Fable 5** and switched when access ended.
- **Control group:****8,908** developers who stayed on**Opus 4.8** throughout.
- **Benchmark gap:** Fable 5 was about**11 points higher on SWE-Bench Pro** than Opus 4.8 per the paper's framing.

Because people and projects were stable, the design aims to isolate **model capability as marketed on benchmarks** from **team skill, repo difficulty, and seasonality** — threats that plague casual before/after comparisons on X.

## Methods in plain language

Issue-to-patch flow for SWE-style coding benchmarks versus merged pull requests

Hua fits a **difference-in-differences** model with **developer fixed effects** and **date fixed effects**. Intuitively: compare how each developer's merge rate changed around the ban against how a similar calendar window behaved for developers who never lost Fable 5.

The outcome is **whether a pull request merges within seven days**, not raw commit counts. That matches how many eng orgs define **throughput** for agent-assisted work, and it pairs with review norms we covered in [Claude Code team review risk](https://explainx.ai/blog/claude-code-team-code-review-process-ai-reviewer-risk-tiers-reverts-down-40-percent-2026).

The paper also runs **five model upgrades** before and after the suspension on **1,208 developers** and **15,442 pull requests**. All five point estimates for merge-rate lifts are **negative or zero**, with overlapping confidence intervals — the opposite of a clean "new model = faster shipping" story.

## What this does not prove

Reasonable limits belong in the same paragraph as the headline:

- **Observational GitHub data** may miss private forks, force-pushes, or squash merges that look slow but ship fast.
- **19 days** may be too short for large refactors that benchmarks reward but calendars delay.
- **Opus 4.8** as the fallback is one counterfactual; other routes (GPT-6 Astra, local Qwen) could behave differently.
- **Export bans** correlate with**news and anxiety** , which might distract developers independent of model IQ.

None of those caveats rescue the claim that **+11 SWE-Bench Pro points** automatically bought **measurable merge acceleration** in this sample.

## How this fits the November 2026 agent eval stack

Read Hua alongside three threads explainx.ai already covers:

1. **Organizational completeness:**[SWE-CoWork](https://explainx.ai/blog/swe-cowork-coding-agents-digital-employees-benchmark-2026) shows high test pass rates with** single-digit full-task** completion.
2. **Harness effects:**[Together agent evals](https://explainx.ai/blog/together-ai-agent-evals-97-percent-coding-agents-november-2026) found** 97%** on narrow API-doc tasks — a reminder that**task design** moves scores.
3. **Policy shocks:** Fable-tier access already intersected[export control posts](https://explainx.ai/blog/congress-bipartisan-nvidia-china-chip-curbs-ndaa-cia-briefing-november-2026) and vendor**frontier gating** ; bans change**which model** runs, not just**which badge** marketing shows.

If you run an agent pilot, the actionable shift is to **pre-register production metrics** — merge rate, rework, review hours, incident counts — before you let procurement pick on leaderboard rank alone.

## What developers should do this week

**For team leads:** Add a **merge-rate dashboard** tagged by model and harness version. Compare against your **SWE-bench internal slice**, not the public leaderboard.

**For individual builders:** When a model upgrades, run **two weeks parallel** on non-critical repos before you retire the old tier — mirror how [loop engineering](https://explainx.ai/blog/loop-engineering-coding-agents-claude-code-guide-2026) teams stage harness changes.

**For eval authors:** Publish **confidence intervals** on business outcomes, not only point estimates on synthetic tasks. Hua's design is a template: find **exogenous** model switches (bans, price hikes, enterprise defaults) and measure **GitHub or Jira**, not just pass@k.

## Interpreting the Fable 5 counterfactual for procurement

Procurement teams often treat **+10 SWE-Bench Pro** as a **capacity contract** — implicitly promising capacity planning that Hua's study does not support. During the **June 2026 suspension**, most displaced users landed on **Opus 4.8**, a model the paper frames as roughly **eleven points lower** on that benchmark.

If benchmark gaps translated linearly into **merge probability**, you would expect a visible jump when Fable 5 returned. The estimated **+1.4 percentage point** move on **seven-day merge rate** sits inside an interval that crosses zero. Weekly volume shows the same story: **+2.4%** merged PRs with an interval spanning **-4.9% to +10.2%**.

That pattern fits **harness-limited** agent work more than **raw reasoning-limited** work. An agent that aces synthetic patches may still stall on:

- **Ambiguous acceptance criteria** in real tickets
- **Cross-service failures** that never appear in benchmark containers
- **Review latency** when human owners disagree with generated architecture
- **CI flakes** that burn the seven-day merge window

Each failure mode appears in explainx.ai's organizational benchmarks and customer dispute reporting this quarter, independent of Hua's GitHub sample.

## Designing your own merge-rate study

You do not need a federal export ban to replicate the spirit of this paper:

1. **Pick an exogenous switch:** enterprise default model changes, sudden price hikes, or security blocks on a tier.
2. **Freeze repo cohorts:** compare developers on the same services before and after, not unrelated teams.
3. **Pre-register outcomes:** seven-day merge rate, time-to-first-review, revert rate, and escaped defect counts.
4. **Pair with benchmark slices:** run the same internal SWE-bench subset monthly so you can see**divergence** between leaderboard deltas and delivery metrics.

Document the **harness version** beside the model name. [Revolut's Codex fincrime story](https://explainx.ai/blog/revolut-openai-codex-fincrime-gpt-5-5-enterprise-story-2026) shows enterprise wins come from **workflow integration**, not leaderboard rank alone.

## Connection to Claude export politics

The **June 12, 2026** Fable suspension sits in the same policy summer as intensified **chip export** debates covered in [Nvidia Megaspeed threads](https://explainx.ai/blog/nvidia-huang-megaspeed-alice-huang-full-story-timeline-what-happened-2026). For builders, the lesson is dual: geopolitics can **remove your best model overnight**, and the **fallback may not slow you down** if bottlenecks live in process, not logits.

Plan **fallback tiers** and **merge dashboards** together — not only API routing tables.

## FAQ alignment with search wording

**Is SWE bench good for picking a coding model?** It is good for **regression testing** on isolated issues. Hua's evidence says it is **insufficient alone** for predicting **merged PR velocity** across hundreds of developers.

**Why did my team not speed up after a benchmark winner?** Plausible reasons include **review bottlenecks**, **ambiguous tickets**, **flaky CI**, and **agents that patch tests without finishing requirements** — the same failure modes SWE-CoWork names explicitly.

## Reading the five upgrade panel

Beyond the Fable ban, Hua repeats the diff-in-diff design across **five unrelated model upgrades** in 2026 on **1,208 developers** and **15,442 pull requests**. Every upgrade-specific merge-rate estimate is **negative or zero** in the abstract, with **overlapping confidence intervals**.

That uniformity matters: a single weird fortnight around export politics could be noise, but a **panel of upgrades** failing to show merge lifts pushes responsibility back to **process metrics**. Teams should log **review rounds per PR**, **percent AI-authored lines reverted**, and **escapes to production** beside SWE-bench internal slices.

Vendor marketing will keep citing **point deltas**. This paper supplies the counter-question: **did merged output move?** If not, negotiate pilots on **merge SLAs**, not leaderboard slides.

**For finance and platform owners:** attach agent spend reports to the same dashboard. [Microsoft Agent 365 analytics at Ignite](https://explainx.ai/blog/microsoft-ignite-2026-complete-coverage-every-announcement-november-2026) and [Perplexity orchestrator pricing](https://explainx.ai/blog/perplexity-orchestrator-api-45-models-november-2026) show the industry moving toward **outcome-based AI bills**. Merge-rate flatlines mean **token invoices can rise without delivery gains** — exactly the risk enterprise FinOps teams warned about in November.

## Related reading

Hua releases **four figures and twenty pages** in the arXiv PDF; cite effect sizes from there before you forward screenshots. The abstract alone is enough for the headline, not for board-level ROI math.

Reading gets you the idea. Building it with feedback gets you the skill.
