[AI](https://sourcefeed.dev/c/ai)Article
New research from Microsoft and academia pins down why AI coding gains keep vanishing between the demo and the dashboard.
[Priya Nair](https://sourcefeed.dev/u/priya_nair)
Most myth-busting listicles are content marketing wearing a lab coat. "Eight Myths on Software Engineering and GenAI," published in ACM Queue by Jenna Butler, Brian Houck, Emerson Murphy-Hill, and three co-authors — a team drawn largely from Microsoft's developer-research orbit, plus the University of Victoria's Margaret-Anne Storey — is not that. It's the people who study how engineers actually spend their days putting numbers on beliefs that are currently driving procurement decisions, headcount plans, and some genuinely embarrassing executive dashboards.
The paper's eight myths cluster into three failures: we misjudge where developer time goes, we measure AI's impact with metrics we already knew were broken, and we assume adoption is a tooling problem when it's an organizational one. All three deserve a harder look than the myth format suggests, because the conclusion isn't "AI is overhyped." It's that the industry is doing the arithmetic wrong on something real.
Coding was never the bottleneck #
The load-bearing number is 14 percent. That's how much of their time developers spend writing code, per a 2025 study of more than 450 Microsoft engineers — and it's consistent with a decade of time-use research that puts coding between 11 and 18 percent of a typical day. The rest is design, review, meetings, coordination, and the general archaeology of understanding existing systems.
Run that through Amdahl's law and the ceiling on autocomplete-shaped tools gets low, fast. Double your coding speed — a wildly generous assumption — and you've bought roughly a 7 percent improvement in overall throughput. Not nothing. Also not the "10x developer" that shows up in board decks.
The measurement myth compounds this. Organizations are tracking AI impact in lines of code generated and suggestion-acceptance rates — metrics the field discredited decades ago. Bill Gates supposedly likened measuring progress by lines of code to measuring aircraft progress by weight, and yet here we are in 2026, rediscovering the mistake because a vendor dashboard made it convenient again. Acceptance rate is worse: it measures how often developers take a suggestion, not whether the suggestion survived review, shipped, or caused an incident.
The contradictory studies are both right #
Here's where the paper earns its keep. The two most-cited data points in this debate appear to be irreconcilable. GitHub and Microsoft's controlled experiment (Peng et al., 2023) found developers with Copilot finished 55.8 percent faster. METR's randomized trial of experienced open-source maintainers found the opposite: 19 percent slower with AI tools — while the same developers estimated they'd been 20 percent faster.
The resolution is that the studies measured different worlds. Peng's subjects were freelancers building a greenfield JavaScript HTTP server — an isolated, well-specified task with no legacy context. METR's subjects were maintainers working in mature repositories averaging a million-plus lines of code, on projects they knew intimately. AI's gains concentrate where your own knowledge is thinnest: unfamiliar languages, boilerplate, standalone tasks. On code you already know cold, the model is a junior colleague who needs everything explained and still gets the house style wrong.
The variance is the finding. Any single average — 55 percent faster, 19 percent slower — is marketing, in one direction or the other. And METR's perception gap should terrify anyone whose AI ROI case rests on a developer survey: the people being surveyed sincerely believed the opposite of what the stopwatch said.
Adoption is a status problem, not a licensing problem #
The 2025 Stack Overflow survey found 84 percent of developers using or planning to use AI tools, while 46 percent actively distrust their accuracy — distrust that has grown, not shrunk, as adoption spread. Usage without trust isn't productivity; it's verification overhead relocated downstream to code review.
Then there's the finding I'd wager most engineering leaders haven't confronted: the competence penalty. A pre-registered experiment with 1,026 engineers, published last year in Harvard Business Review, had participants rate identical code — the only variable being whether the author was described as having used AI. The AI-assisted engineer was rated 9 percent less competent. For women, the penalty was 13 percent. If admitting you use the tool costs you status in review culture, people will use it quietly and lie on your adoption surveys, and your telemetry will never explain why the productivity numbers don't move.
This is why "good tools sell themselves" dies on contact with an actual organization, and why the enterprise-can-move-like-a-startup myth is in the list too. Startups point AI at greenfield code and sprint to an MVP. Enterprises point it at twenty years of legacy, compliance gates, and integration contracts — precisely the terrain where METR's numbers, not Peng's, apply.
The vendors already read this paper #
Here's the twist the myth framing undersells: the industry's biggest players have already priced in the 14 percent problem. It's the obvious explanation for why 2025 and 2026 became the year of the agent — Copilot's coding agent and Anthropic's Claude Code taking on issue triage, test scaffolding, migrations, and review assistance. If typing code is 14 percent of the job, the other 86 percent is the actual market.
But every myth in this paper applies to agents with more force, not less. Agent-generated pull requests inflate line counts spectacularly, making LOC dashboards even more misleading. Agents shine on well-specified grunt work and flail on tasks needing deep system context — the same variance, amplified. And an agent's output lands squarely on the trust problem: someone still has to review it, in a culture that may quietly penalize whoever merges it.
So, practically: audit where your team's hours actually go before buying anything, because your bottleneck is more likely review latency or requirements churn than authoring speed. Point AI spend at the measured constraint. Track lead time, review turnaround, and change-failure rate — never lines of code, never acceptance rate. Treat self-reported speedup as sentiment data. And if you want honest adoption, make senior engineers use the tools in the open, where the status signal runs the right way.
The paper's real contribution isn't deflating GenAI — it's insisting that a genuine shift deserves real measurement. The teams that get this right won't be the ones with the most seats licensed. They'll be the ones who knew what their 86 percent looked like.
Sources & further reading #
[Eight Myths on Software Engineering and GenAI](https://queue.acm.org/detail.cfm?id=3807963)— queue.acm.org -
[RDEL #146: Which popular beliefs about GenAI and software engineering hold up to research?](https://rdel.substack.com/p/rdel-146-which-popular-beliefs-about)— rdel.substack.com -
[8 myths on software engineering and AI](https://getdx.com/blog/8-myths-on-software-engineering-and-ai/)— getdx.com -
[Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)— metr.org -
[The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://arxiv.org/abs/2302.06590)— arxiv.org -
[AI - 2025 Stack Overflow Developer Survey](https://survey.stackoverflow.co/2025/ai)— survey.stackoverflow.co -
[Research: The Hidden Penalty of Using AI at Work](https://hbr.org/2025/08/research-the-hidden-penalty-of-using-ai-at-work)— hbr.org
[Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer
Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to.
Discussion 0 #
No comments yet
Be the first to weigh in.