AIArticle The researchers behind SPACE say your GenAI dashboards grade the slice of engineering that was never the bottleneck.
Mariana Souza The researchers who spent the last decade building the field of developer-productivity measurement have looked at how the industry is scoring GenAI, and their verdict is blunt: most of the numbers are wrong. "Eight Myths on Software Engineering and GenAI," published in ACM Queue by a team including Brian Houck, Jenna Butler, and Margaret-Anne Storey — three co-authors of the SPACE framework — is doing numbers on Hacker News right now, and it's tempting to file it under AI skepticism. That's a misread. The paper barely argues about what models can do. It argues that organizations are measuring AI's impact in the wrong place, with metrics the field discredited before ChatGPT existed.
The 14 percent problem #
The anchor statistic: a 2025 Microsoft study of more than 450 engineers found they spend roughly 14 percent of their time actually writing code — about 18 percent on a good day, 11 on a bad one, consistent with a decade of prior time-use research. The rest goes to design, review, debugging, meetings, incident response, and the coordination overhead of building software with other people.
Run the Amdahl's law math and the ceiling gets low fast. If coding is 14 percent of the job and AI wrote every line instantly, the team ships about 16 percent faster. That's a nice win. It is not the "10x developer" pitch that's justifying eight-figure license spend.
The best counterargument — and the Hacker News thread made it forcefully — is that the 86 percent isn't fixed. When code gets cheap, the workflow around it restructures: you prototype three approaches instead of scheduling an alignment meeting to debate one, test harnesses and migrations stop being quarter-long projects, and some coordination work exists only because code used to be expensive. That's the strongest version of the optimist's case, and it's probably where the real gains live. But notice what kind of claim it is: a second-order effect nobody has measured yet. Which is precisely the paper's complaint — the industry keeps booking projections as if they were realized gains.
Faster code, slower everything else #
What actually happens when you accelerate the 14 percent is that the work piles up downstream. DX research cited by the authors puts typical code-throughput gains from AI tools at 7.8 percent, and one internal coding agent they examined had about half its pull requests accepted — 15 percent abandoned outright, another 15 percent stuck in review. Review is the new bottleneck: more code, produced faster, with less human context behind each line, flowing into a review process that hasn't gotten any faster.
Against that backdrop, the metrics most AI dashboards track — lines of AI-generated code, suggestion-acceptance rate — aren't just noisy. They reward exactly the behavior that clogs the system: bigger diffs, faster merges, less scrutiny. The authors reach for Bill Gates's old line that measuring software by lines of code is like measuring aircraft progress by weight. The industry agreed with that for thirty years, then quietly forgot it the moment vendors started printing "percentage of code written by AI" on quarterly slides.
The variance nobody prices in #
The single most instructive datapoint in this debate remains METR's randomized controlled trial from mid-2025: 16 experienced open-source maintainers, 246 real tasks in their own mature repositories, early-2025 tooling (mostly Cursor with Claude 3.5/3.7 Sonnet). With AI, they took 19 percent longer — while estimating afterward that AI had made them 20 percent faster. That perception gap should terrify anyone whose AI ROI evidence is a self-reported developer survey, which is to say most of the industry.
Two honest caveats. METR itself now describes that result as historical — tools and workflows have moved since early 2025. And the study measured one specific, hard regime: experts working in codebases they know deeply, where AI has the least to add. Both camps in this argument are quoting stale numbers at each other — skeptics wielding a study whose own authors have moved on, boosters citing vendor benchmarks on greenfield toy tasks. The finding that does survive is variance: outcomes swing wildly with task familiarity, codebase maturity, and even phrasing — one study the paper cites found 46 percent of semantically equivalent prompts produced different code, and 28 percent changed its correctness. "Does AI make developers faster?" is simply not a well-formed question. "On which tasks, for whom, at what review cost" is.
Adoption is a trust problem, not a tooling problem #
The 2025 Stack Overflow survey found 84 percent of developers using or planning to use AI tools while only 29 percent trust the accuracy of the output — down from 40 percent a year earlier, with active distrust (46 percent) now outrunning trust (33 percent). Usage up, trust down: that's not a rollout succeeding, it's a mandate being absorbed.
And there's a darker adoption blocker the paper surfaces that deserves more attention than it gets: the competence penalty. In an experiment published via Harvard Business Review, 1,026 engineers rated identical code 9 percent lower on competence when told the author used AI — a 13 percent penalty for women against 6 for men, harshest from male engineers who don't use AI themselves. In the firm studied, a year after rolling out a state-of-the-art assistant, only 41 percent of engineers had tried it. Some of your engineers aren't lagging because they can't install a plugin. They're rationally protecting their reputations. Every "AI usage" leaderboard and disclosure mandate makes that calculus worse.
What to change on Monday #
The paper's practical upshot compresses well. Kill volume metrics — anything counting AI-generated lines or acceptance rates is measuring airplane weight. Instrument the system instead: end-to-end cycle time, review queue latency, rework rate, and defect escape rate on AI-heavy diffs. If you're deploying coding agents, budget review capacity the way you would for a team of very fast juniors — smaller PRs, stronger CI, provenance in commit metadata — because the constraint has moved to the human on the other side of the diff. And before renewing licenses, find your actual bottleneck: if it's requirements churn or review latency, a faster code generator is pointed at the wrong constraint.
The verdict this paper earns: it's right about the present tense and appropriately silent about the future. It doesn't claim AI won't transform software engineering — it claims we're grading the transformation with numbers that would have flunked a DORA review in 2019. The teams that come out ahead won't be the ones with the highest AI-generated-LOC ratio. They'll be the ones who re-instrumented the other 86 percent of the job while their competitors optimized the slice that was never the bottleneck.
Sources & further reading #
[Eight Myths on Software Engineering and GenAI](https://queue.acm.org/detail.cfm?id=3807963)— queue.acm.org -
[Eight Myths on Software Engineering and GenAI (HN discussion)](https://news.ycombinator.com/item?id=49176830)— news.ycombinator.com -
[Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)— metr.org -
[Research: The Hidden Penalty of Using AI at Work](https://hbr.org/2025/08/research-the-hidden-penalty-of-using-ai-at-work)— hbr.org -
[AI - 2025 Stack Overflow Developer Survey](https://survey.stackoverflow.co/2025/ai)— survey.stackoverflow.co -
[8 myths on software engineering and AI](https://newsletter.getdx.com/p/8-myths-on-software-engineering-and)— newsletter.getdx.com -
[RDEL 146: Which popular beliefs about GenAI and software engineering hold up to research?](https://rdel.substack.com/p/rdel-146-which-popular-beliefs-about)— rdel.substack.com
[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.