{"slug": "ai-doesnt-make-developers-faster", "title": "AI Doesn’t Make Developers Faster", "summary": "A developer's analysis of AI coding-assistant productivity research finds the evidence mixed and context-dependent: Peng et al.'s GitHub Copilot study showed developers completing an HTTP server task 55.8% faster, and Cui et al.'s randomized trials across Microsoft, Accenture and a Fortune 100 firm found 4,867 developers completed 26% more tasks with AI access. By contrast, METR's study of 16 experienced open-source maintainers working 246 real issues in familiar repositories found AI made them 19% slower, even as they believed it had sped them up roughly 20%.", "body_md": "### But it does, doesn’t it?\n\nYes.\n\nExcept when it doesn’t.\n\nThere’s been a number of studies into whether AI coding assistants actually make developers faster, and some of them provide genuinely good evidence for some serious speed gains. The one everyone quotes is the [GitHub Copilot study](https://arxiv.org/abs/2302.06590) by Peng et al., where they took 95 developers, gave them all the same task (build an HTTP server in JavaScript) and let half of them use Copilot. The Copilot group finished 55.8% faster, which is a legit speed increase, can’t argue with that.\n\nThen there’s the paper by [Cui et al.](https://pubsonline.informs.org/doi/10.1287/mnsc.2025.00535) that shows people powering through way more work than they usually would with the help of AI. They ran three proper randomised experiments inside real companies - Microsoft, Accenture and an anonymous Fortune 100 - with 4,867 developers doing their actual jobs rather than one toy task, and the devs with AI access completed 26% more tasks. As an early adopter and hardcore AI user myself, I am certainly not going to claim “AI productivity is all hype” - the gains are real, the studies are decent, and anyone who uses these tools every day knows *something* is happening.\n\nHowever, the hype around these specific AI productivity stats does make me wary. I’ve seen the numbers from these studies bandied about all over the place and I feel like they’ve somewhat escaped their experimental context, especially the Copilot one - “AI makes developers 55% faster” turns up everywhere, and it just feels... incomplete. 55% faster at what, exactly?\n\n## Then there’s METR\n\nThen of course there’s studies that say the exact opposite, because that’s how science works, but they also tell a pretty convincing story.\n\n[METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) ran an experiment with 16 experienced open-source maintainers working through 246 real issues in repositories they already knew really well, with each task randomly assigned to be done either with or without AI. With AI, they were **19% slower**.\n\nAnd the best bit is that before they started, the devs expected AI to make them about 24% faster, and afterwards, having been measurably slower, they *still* thought it had made them about 20% faster. There is a disconnect between reality and the devs perception and it’s fascinating\n\nI love it because I completely believe it. Using AI *feels* fast - code appears instantly, stuff is visibly happening on screen - and meanwhile, much less memorably, you’re spending time writing prompts, waiting, reading what came back, spotting the thing that’s wrong, correcting it and working out what the thing actually did to your file. None of that feels like work in the moment, but it all goes on the clock. Which is worth remembering next time a shiny-suited sales rep tells you their AI allows devs to “save ten hours a week”, because feeling faster and being faster are not the same thing at all.\n\nThis study is not be end of the story either - you must bear in mind it’s a small and very specialised group of people and it was early-2025 tooling. METR themselves have since said newer tools are [probably better](https://metr.org/blog/2026-02-24-uplift-update/) - they tried to rerun it and found a lot of devs didn’t want to take part if they might get stuck in the no-AI group, which is kind of its own result really.\n\n## Do these studies actually contradict each other?\n\nYes and no, but mostly no - I think they’re just asking different questions. Peng asked how quickly you can implement a well-defined thing, Cui asked something closer to how much work gets done in a real workplace, and METR asked how quickly someone who deeply understands a big, mature system can make real changes to it.\n\nThose are different jobs. The Peng task is almost pure implementation - here’s a clear spec, go and write the code - whereas a task on a mature codebase is a very different beast - only a small part of it is the actual implementation and the rest is understanding what’s already there, deciding what to do, navigating existing behaviour, not breaking the contracts other things depend on, checking it works and making it all fit together.\n\nSo if AI halves the time it takes to write the code, what that does to the whole task depends entirely on how much of the task was writing the code. On a greenfield HTTP server, loads. On a five-year-old repo you know inside out, where the hard bit is knowing *which* three lines to change... not much, and if the AI’s suggestions then need unpicking you can end up in a worse place than you started.\n\nWhich is a much more useful way of looking at it than trying to average 55%, 26% and -19% into some nonsense universal number. There isn’t one, there’s just “it depends what the work is”, which is annoying, but true.\n\n## Maybe “faster developer” is the wrong way to think about it\n\nWhat AI is very obviously doing is making it cheaper to produce a plausible implementation, something that looks like a solution. I’ve started thinking of this as a *first draft* - it looks like a solution, it might even be a good one, but like any first draft it hasn’t earned the right to be the finished thing yet, and nobody sensible ships one without reading it themselves. It still has to be understood, tested, reviewed, integrated, deployed and run, and ultimately someone has to decide whether it solved anything useful at all.\n\nSo instead of:\n\nAI makes software engineers 50% more productive.\n\nI’d much rather say:\n\nAI makes the *first draft* of software cheap.\n\nThe second one is a lot less exciting on a slide, but it’s much easier to defend, and importantly it survives all three studies. I think if you are a big AI user like I am, if you sit and actually think through your daily processes with whatever tools you use, you’ll find it holds for you two. Anecdotally I can tell you I’ve never successfully one-shotted a change (for work projects), there’s always been some tweaks or fine tuning needed after the first cut lands.\n\n## Writing code and shipping code are already drifting apart\n\nThere’s one more study I should mention because it shows something slightly different and, dare I say it, realistic... [Demirer, Musolff and Yang](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6859839) have a working paper looking at AI tool usage across more than 500,000 GitHub developers. They’ve found that when devs adopt AI tools they see roughly +240% commits, +80% projects and +30% releases (it’s a working paper and the numbers have moved around as the dataset’s grown, so don’t hold me to the exact percentages).\n\nNow this does *not* mean 210% of the code went in the bin - that’s not how those measures work, even though it’s a very tempting way to read it. The interesting bit is the shape, a gigantic change in coding activity and a much smaller change in actual released software - the further you get from “code exists” towards “software ships”, the smaller the effect gets. Something happens in between, and what exactly that something is - review, testing, integration, release processes, people starting more things than they finish - this is the question I think everyone is trying to answer in one way or another.\n\n## It changes the question\n\nIf implementation gets cheaper, devs can finish sooner - 12 tickets completed before lunch then the afternoon off - I’m sure that’s happening somewhere. However I’m not sure that’s really in our nature... If AI helps devs write code faster, I imagine many of use are using this extra capacity to build more, or build something harder, try three approaches instead of one, do the refactor that was never worth a week, build the little internal tool nobody could ever justify... generally get a lot more ambitious about what’s worth attempting at all. I can’t tell you which of those things is actually happening, again anecdotally I’d say probably all of them.\n\nThe standard productivity model quietly assumes something like *same work + less effort = finished sooner*, but why on earth would the work stay the same? The important thing to remember when trying to reconcile this with those studies above is that experiments like those *have* to keep the work and the tasks constant in order to be able to compare results. Which is wildly different to real life - no company is sticking to their 2024 roadmap and asking the team to build exactly that, just quicker.\n\n## So, does AI make developers faster?\n\nIt’s not a completely meaningless question - for a well-defined, bounded task we can measure it, and the answer is often yes, by quite a lot. It’s just not a particularly good question if you want to understand what’s happening to software teams and the organisations around them. I think the more interesting question is:\n\nWhat happens when producing another first draft becomes dramatically cheaper?\n\nBecause once you ask that, a whole load of other questions fall out of it. Do we ship sooner, or just attempt more? Does scope quietly grow? Where do the bottlenecks move to? Who’s checking all this stuff? Does quality suffer? And what the hell does “developer productivity” even mean anymore?\n\nI don’t have answers to most of those yet, but I’m starting with the first one, because if software engineers suddenly have loads more implementation capacity I can’t see any reason to assume they’ll keep building exactly the same things... we never have before.", "url": "https://wpnews.pro/news/ai-doesnt-make-developers-faster", "canonical_source": "https://theotherjenai.substack.com/p/ai-doesnt-make-developers-faster", "published_at": "2026-10-20 09:10:03+00:00", "updated_at": "2026-10-20 09:19:21.885607+00:00", "lang": "en", "topics": ["ai-tools", "ai-products", "ai-research", "developer-tools", "artificial-intelligence"], "entities": ["GitHub", "Copilot", "METR", "Microsoft", "Accenture", "Peng et al.", "Cui et al."], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/ai-doesnt-make-developers-faster", "markdown": "https://wpnews.pro/news/ai-doesnt-make-developers-faster.md", "text": "https://wpnews.pro/news/ai-doesnt-make-developers-faster.txt", "jsonld": "https://wpnews.pro/news/ai-doesnt-make-developers-faster.jsonld"}}