{"slug": "ai-productivity-gains-are-real-but-smaller-in-production", "title": "AI productivity gains are real, but smaller in production", "summary": "A new meta-analysis of 23 studies on GenAI coding assistants, published on arXiv, found a statistically significant but moderate productivity gain in programming. However, the effect was highly context-dependent and smaller in real-world open-source and enterprise settings compared to controlled experiments, with no significant improvement in learning. The article concludes that while AI tools help with well-scoped tasks like boilerplate code, their impact is reduced by the complex, non-local friction of production engineering, such as legacy constraints and cross-team dependencies.", "body_md": "A new meta-analysis on GenAI coding assistants landed on arXiv this month, and I think it is a useful cold shower for both sides of the argument.\nThe paper looked across 23 studies and found a statistically significant productivity gain from GenAI assistance in programming. Not magic. Not fake. A real effect.\nBut the effect was moderate, highly context-dependent, and smaller in open-source and enterprise settings than in controlled experiments. It also found no statistically significant learning gain.\nThat is basically the whole AI coding debate in one sentence: the tools help, but the demos are not the work.\nAnd honestly, that feels right.\nI use AI coding tools. They are useful. They save time on boilerplate, tests, refactors, library glue, CLI flags, and those annoying \"I know what I want but do not want to type it\" moments.\nBut if you spend enough time around real systems, you also know why the big productivity promises feel slippery. Programming in production is not just generating code. It is understanding weird constraints, changing old systems without breaking them, negotiating ownership, reading logs, dealing with flaky tests, waiting for reviews, managing releases, and discovering that the simple change touches a tax rule from 2019.\nThe assistant can help with parts of that.\nIt does not delete the rest.\nControlled experiments are useful because they isolate the thing being measured. Give one group an AI assistant, give another group no assistant, compare completion time or output. That can tell us something.\nBut it does not tell us everything.\nMost real engineering work is full of non-local friction. The task description is incomplete. The test suite is slow. The architecture has history. The senior reviewer remembers the migration that almost broke billing. The \"obvious\" library is banned because of licensing.\nAI tools are at their best when the problem is well-scoped and the feedback loop is tight. \"Write this parser.\" \"Generate a test for this function.\" \"Convert this object shape.\" \"Explain this error.\" \"Draft the boring first version.\"\nThey are weaker when the hard part is not typing but judgment.\nThat distinction matters because many productivity claims quietly move between those two worlds. A tool can be very impressive on a contained programming task and still produce a much smaller gain inside a mature company codebase.\nNot because the tool is bad.\nBecause the bottleneck moved.\nWhen people say \"AI makes developers faster,\" I always want to ask: faster at what?\nFaster at producing lines of code? Probably.\nFaster at getting through a small implementation task? Often.\nFaster at safely changing a critical system with unclear requirements, old tests, cross-team dependencies, and production risk? Maybe. Sometimes. It depends.\nThat boring answer is the honest one.\nProductivity in software is a bundle of different activities: discovering the real problem, designing the change, writing code, testing it, reviewing it, deploying it, observing it, and owning the consequences.\nIf a team measures only output, it will overestimate the gain. More code appears. More PRs open. More tickets move. Everyone feels busy in a very modern way.\nBut if review queues grow, incidents become harder to explain, and engineers spend more time validating generated work, the net gain is smaller than the dashboard suggests.\nThe uncomfortable part is that both can be true at once. The individual developer may feel faster, while the team system does not improve by the same amount.\nI am not surprised the effect looks smaller in open-source and enterprise contexts.\nEnterprise software has a tax. Some of it is bureaucracy, yes. But a lot of it is real complexity.\nYou have domain rules, compliance rules, security policies, observability standards, release windows, internal libraries, private APIs, legacy constraints, and customers using features in ways nobody would design today.\nGeneral coding assistants do not automatically know that local truth.\nThey can infer from the repo, read docs if you wire them in, follow patterns if the patterns are consistent, and call tools if the organization gives them safe access. But they do not walk in with the production memory of the team.\nThat is why I keep coming back to context as infrastructure. The companies that get more value from AI coding tools will not be the ones that merely buy the fanciest model. They will be the ones that make their local truth legible: good tests, good internal docs, good runbooks, good platform APIs, good examples, good constraints, and good review culture.\nAI does not remove the need for engineering maturity. It puts a multiplier on it.\nIf your codebase is coherent, the assistant has something coherent to imitate. If your platform has golden paths, it can follow them. If your tests encode real behavior, it gets useful feedback. If your architecture is folklore and Slack archaeology, it will confidently automate the confusion.\nThe paper also found no statistically significant effect on learning outcomes.\nThat part deserves more attention.\nA coding assistant can help someone complete a task without helping them become better at the underlying skill. Tutorials, copy-paste answers, and framework magic have always had this shape. AI just makes the bypass smoother.\nYou can get working code before you understand the problem. You can ask for an explanation that sounds clear enough to continue. You can ship a fix without building the mental model that would let you debug it next month.\nFor experienced engineers, that can be fine. They already have enough structure in their head to judge the output. The assistant becomes leverage.\nFor juniors, the risk is different. If the tool gives you answers before you have struggled enough to form taste, it can slow down the painful but necessary part of learning: building your own debugging instincts.\nThat does not mean juniors should avoid AI tools. That would be silly. These tools are now part of the job. But using them well is a skill. The goal should not be \"get the answer.\" The goal should be \"understand why this answer is right, where it might be wrong, and what I would check next.\"\nThe career problem is not that AI will make learning impossible. It is that AI makes shallow completion feel like learning.\nThose are different things.\nIf I were leading a team, I would not try to settle the AI productivity question with vibes.\nI would measure smaller, more honest things.\nHow much faster do common tasks move from start to merged? Do review cycles get shorter or longer? Are generated tests catching real regressions or just decorating the diff? Are incidents easier or harder to debug after AI-assisted changes? Are junior engineers becoming more independent?\nI would also separate task types.\nAI might create a 30 percent improvement on boilerplate-heavy product work, almost no improvement on ambiguous architecture decisions, and a negative improvement on security-sensitive changes if the team gets lazy.\nA single productivity number hides all of that.\nThe smarter move is to build a map of where the tool is strong, where it is neutral, and where it needs guardrails.\nMy current posture is simple: use AI aggressively for acceleration, but do not outsource ownership.\nLet it draft. Let it explain. Let it search. Let it write the boring tests. Let it sketch alternatives. Let it find the API you forgot. Let it do the first pass on the migration script.\nThen make the human team own the final shape.\nThat means reviewers still need to understand the change. Tests still need to encode intent. Architecture decisions still need rationale. Production systems still need observability. Someone still needs to know why the code is this way and not that way.\nThe meta-analysis is useful because it does not give either camp the easy slogan.\nThe \"AI is useless hype\" camp is wrong. There is real productivity value here.\nThe \"AI makes everyone 10x\" camp is also wrong, or at least wildly incomplete. Real-world software work contains too much context, coordination, and consequence for a coding assistant to turn every engineer into a demo video.\nThe truth is less flashy and more useful: AI coding assistants are real leverage when the surrounding engineering system can absorb them.\nThat last clause is the job.\nBetter context. Better tests. Better review habits. Better internal platforms. Better measurement. Better judgment.\nThe tool can make a good engineering system faster.\nIt can also make a confused system produce confusion at higher speed.\nSo yes, use the tools.\nJust do not confuse faster typing with faster engineering.", "url": "https://wpnews.pro/news/ai-productivity-gains-are-real-but-smaller-in-production", "canonical_source": "https://dev.to/pvgomes/ai-productivity-gains-are-real-but-smaller-in-production-6c2", "published_at": "2026-05-24 00:02:03+00:00", "updated_at": "2026-05-24 00:33:11.029196+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "research", "enterprise-software"], "entities": ["arXiv", "GenAI"], "alternates": {"html": "https://wpnews.pro/news/ai-productivity-gains-are-real-but-smaller-in-production", "markdown": "https://wpnews.pro/news/ai-productivity-gains-are-real-but-smaller-in-production.md", "text": "https://wpnews.pro/news/ai-productivity-gains-are-real-but-smaller-in-production.txt", "jsonld": "https://wpnews.pro/news/ai-productivity-gains-are-real-but-smaller-in-production.jsonld"}}