Microsoft ran the largest real-world field study of AI coding agents to date. Tens of thousands of engineers. Two tools — Claude Code and GitHub Copilot CLI. Four months of telemetry. The headline finding is a 24% lift in merged pull requests. But if you stop at the headline, you miss the more useful parts: how these tools actually spread inside a large engineering organization, who stuck with them, and what eventually killed Claude Code’s budget despite solid productivity gains.
The 24% Number: What It Actually Means #
Researchers from Microsoft — Emerson Murphy-Hill, Jenna Butler, and Alexandra Savelieva — published the study on arxiv (2607.01418). The methodology is straightforward: developer-level telemetry tracking how many pull requests engineers merged before and after adopting agentic CLI tools, controlling for what they would have produced otherwise.
The result: adopters merged roughly 24% more PRs. The likely range is +14.5% to +33.7%, and the lift held steady across the full four-month window — it wasn’t a novelty spike that faded after a few weeks. Engineers who used the tools five or more days per week saw lifts above 50%. Those using them around three days a week landed closer to 15%.
The caveat the study’s own authors flag: a merged pull request is not the same as the value it delivers. The research didn’t measure code quality, defect rate, reviewer burden, or whether the PRs being merged were meaningful or trivial. A 24% increase in PR volume could mean 24% more useful code shipped — or 24% more AI-generated noise landing in review queues. The study doesn’t resolve that question. You’ll need to track your own revert rates and review times to know which category your team falls into.
Adoption Spread Through Colleagues, Not Calendars #
This is the finding that deserves more attention than it’s getting. Among tens of thousands of Microsoft engineers, the biggest predictor of whether someone first tried an AI coding agent wasn’t a training mandate, a product announcement, or a calendar invite. It was whether the people around them had already started using one.
First use spread primarily through social networks — colleagues, teammates, people in adjacent offices. Adoption climbed to between 84 and 95 percent of the engineering cohort by April 2026. That’s a remarkable adoption rate for any tool in any organization, achieved organically rather than by mandate.
The implication for engineering managers is direct: if you want your team to actually use these tools, the announcement email isn’t the lever. The lever is identifying the engineers already enthusiastic about AI tools and giving them early access and visible success. The rest follows through proximity.
Who Stayed vs. Who Quit #
Retention data showed one clear pattern: engineers who write the most code were the most likely to keep using AI agents. Demographics — role, seniority, team — were weaker predictors than raw coding volume.
This tracks with how these tools actually work. A developer who opens a dozen files a day gets to see the agent’s leverage compound continuously. Someone who codes occasionally sees smaller absolute gains and has less motivation to build the habit. If your goal is team-wide adoption, prioritize your most active code contributors first — retention will be higher and the productivity signal will be clearer.
Claude Code vs. Copilot CLI: Capability Is Not the Whole Equation #
The study covered both tools in the same rollout. The capability difference is real: Claude Code triggered 2.3 times more file edits per session than Copilot CLI. It operates with a larger context window and more autonomous repo-level reasoning. Copilot CLI has caught up considerably — it now handles agent-loop execution, multi-step planning, file editing, and PR creation — but Claude Code still goes deeper into the codebase per session.
Microsoft ultimately transitioned most engineers to Copilot CLI by June 30, 2026, and the reason was not capability. Token costs from Claude Code consumed the Experiences and Devices division’s entire annual AI budget ahead of schedule — an AI bill that ended up exceeding employee expenses. Copilot CLI offered roughly 80% of the capability at substantially lower cost, with tighter integration into Microsoft’s existing Azure DevOps and GitHub Enterprise stack.
The lesson isn’t that Claude Code is wrong for your team. It’s that token cost is a real budget line you need to model before rollout, not after. TechRepublic’s coverage of the study notes this is the first time a company of this scale has publicly documented the cost tradeoff at production volume.
What You Should Actually Be Tracking #
PR count is a start, not a destination. If you’re rolling out AI coding agents and want to know whether they’re working, here’s what the study’s limitations suggest you measure alongside volume:
PR size— Are agents producing larger or smaller PRs? Larger PRs often mean slower reviews.** Review time per PR**— Is reviewer burden going up or down?** Revert rate**— How often are AI-assisted PRs rolled back?** Defect rate post-merge**— Does AI code introduce more bugs per feature?** Cost per accepted change**— What does each merged PR actually cost in token spend?
Bottom Line #
The data from Microsoft’s rollout is the most credible evidence yet that AI coding agents produce meaningful, durable productivity gains at scale. A 24% increase in merged PRs across four months and tens of thousands of engineers is not noise. But the study also shows that success depends on frequency of use, and that even impressive productivity gains don’t automatically justify costs at enterprise token volumes. Read LeadDev’s breakdown of the adoption patterns if you’re planning how to get your team to actually use these tools rather than just have access to them.