Measuring the impact of AI coding tools: Capacity, not horsepower

DX's Q2 AI Impact Report finds that organizations should measure AI coding tools by their effect on innovation capacity rather than by metrics like 'Developer Horsepower' that estimate human effort replaced. The report advises separating what to measure from how, and recommends tracking throughput and deployments as system-level signals, citing a study of over 100,000 developers where coding agents raised commit volume by up to 180%.

Measuring the impact of AI coding tools: Capacity, not horsepower Separate what to measure from how, then measure across dimensions. Welcome to the latest issue of Engineering Enablement , a weekly newsletter sharing research and perspectives on developer productivity. DX’s Q2 AI Impact Report is now available with the latest research on AI’s impact across engineering organizations. Read the full report. https://getdx.com/resources/?utm source=newsletter When an organization invests in an AI coding tool, the next question is almost always some version of “How do we prove it’s working?” It’s a fair question, and one many organizations are wrestling with today. I was recently asked to weigh in on one such proposal: a metric called “Developer Horsepower,” defined as useful work per day and calculated by multiplying AI-assisted pull requests by an estimate of the human effort each would have required. It’s an intuitive idea, and a genuinely thoughtful attempt at a hard problem. But I think it starts one step too early. Rather than asking how much human work the AI replaced, I’d ask whether AI has increased the organization’s capacity to deliver innovation and whether that additional capacity is sustainable . That framing leads to a very different measurement strategy, and one that I believe is both easier to defend and more actionable. First, separate “what” from “how” Two different questions get tangled together in most measurement conversations. The first is what to measure. The second is how: not just how you collect the data surveys versus telemetry but how strong your evidence needs to be. Is a correlation enough, or do you need a full causal study? Proving causation is genuinely valuable, and I’d never talk someone out of it. If you’re doing something novel, or you want to publish in a peer-reviewed journal, there are good approaches available, from difference-in-differences designs to dose-response studies. But for most organizations, a full causal study is overkill. There is now substantial causal evidence https://newsletter.getdx.com/p/five-studies-that-are-changing-how that AI coding tools can improve coding throughput under many conditions. You don’t need to re-prove that AI increases coding throughput; the field has done that work. In most cases you can measure the correlations in your own environment and lean on the existing causal literature to interpret them. That’s a far lighter lift, and it’s honest as long as you’re not claiming something the literature doesn’t support . With that settled, the interesting question is what to measure. What to measure: innovation capacity, across dimensions The goal isn’t to isolate the tool. It’s to answer whether your engineering system now has more capacity to deliver innovation, and whether that capacity is sustainable. I’d build that up in layers. Throughput: Is more total work happening? PR throughput is a perfectly valid system-level signal for whether more work, in aggregate, is moving through your engineering system. It gets criticized, and rightly so, as a measure of any individual developer, because not all PRs are equal in size or complexity. But at the system level it holds up well. If you can normalize for PR complexity, that’s better still TrueThroughput https://getdx.com/truethroughput/ is one approach to doing that . But even without normalization, aggregate throughput remains a useful directional indicator. I explore this in more depth in the recent article: Revisiting the DX Core 4 in the age of AI https://newsletter.getdx.com/p/revisiting-the-dx-core-4 Deployments: Does the work survive to shipped software? This is the step most teams skip, and it’s the most important one. Recent research shows that AI increases coding activity far more than it increases shipped software. In one large study of over 100,000 developers https://www.nber.org/papers/w35275 , coding agents raised commit volume by up to 180%, yet the effect on actual releases attenuated to roughly 20-30%. The authors attribute the gap to a “weak link”: the upstream speedup runs into human bottlenecks downstream, in review, integration, and release. Counting merged PRs alone may overstate impact, because a large share of that activity attenuates before it reaches production. Tracing your throughput increase all the way through to deployments tells you how much of the upstream gain actually survives. I’d frame the result as innovation capacity: the organization’s realized capacity to deliver innovation. Note the deliberate boundary. Whether that additional capacity translates into business value is a separate question, and often not an engineering one. Deployment frequency is the practical proxy for delivered work; business impact is the true north if you can reach it. Conflating the two is how measurement programs lose credibility. Innovation Time Ratio: Are the savings being reinvested? AI only creates organizational value if the time it saves is reinvested in higher-value work. This is exactly what a metric like Innovation Time Ratio https://queue.acm.org/detail.cfm?id=3819080 captures: the share of developer time spent creating new value versus running the business or carrying administrative load. For most organizations, survey data is the logical place to start, unless you already have robust calendar and development telemetry. And self-report is less of a concern here than it first appears, because you’re looking at change over time. Whatever reporting bias exists tends to stay consistent across measurements, so it largely cancels out in the delta. Quality: A guardrail, not an afterthought. More work moving faster is only progress if it isn’t just churn. A quality signal such as change failure rate or incident mitigation time keeps the speed gains honest. If throughput climbs while failure rates climb with it, you haven’t gained capacity. You’ve moved the cost somewhere less visible. Satisfaction: The sustainability guardrail. Finally, watch developer satisfaction. Throughput and experience can decouple, and gains bought by burning out your engineers aren’t gains you get to keep. Sustainable capacity requires healthy developers. If higher throughput comes at the expense of satisfaction, you’ve likely borrowed from future capacity rather than increased it. Taken together, those layers answer the question leadership is actually asking: not “how much code did the tool produce?” but “does our system have more capacity to deliver innovation, and can we sustain it?” “Developer Horsepower” Which brings me back to the proposal I was asked about. I understand the appeal of a single “horsepower” number, but ultimately I’d recommend a different approach. To be clear, my aim isn’t to single out this particular metric. It’s a reasonable attempt at a real problem, and the team that proposed it is asking exactly the right question. I want to use it as a worked example of the kind of probing I do whenever someone hands me a composite score: what is the unit actually measuring, and how much variation is it hiding? Those questions apply to any single-figure productivity metric, not just this one. My concern isn’t that “Developer Horsepower” is impossible to calculate. It’s that I’m not sure it is answering the question that really needs answering. Leadership doesn’t ultimately care how many “human-equivalent hours” an AI system replaced. They care whether their engineering organization can deliver more innovation, more reliably, and more sustainably than before. The core problem is that it tries to collapse a multidimensional question into one figure. That runs against a principle the field has largely converged on: engineering productivity can’t be captured by a single metric https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/ . A single number simplifies reporting, but it obscures the tradeoffs that matter and makes it difficult to understand what’s actually driving change. The specific construction compounds this in several ways. First, the normalization itself is non-standard and difficult to interpret. “Human-equivalent hours” isn’t an industry term, and a figure built on a bespoke conversion is difficult for anyone outside the team to trust or reason about. More fundamentally, it’s not obvious what those hours are supposed to represent. One of the more surprising findings from our AI Native Developer research was that developers spend only about 14% of their work week actively writing code. The rest is spread across activities like design, code review, debugging, learning, meetings, planning, documentation, and coordinating with teammates. AI doesn’t simply replace coding time; it changes how engineers spend time across many of those activities. Some work disappears, some shifts, and entirely new work like reviewing AI-generated code or managing context emerges. Trying to convert an AI-assisted pull request into a fixed number of “human hours saved” assumes those relationships are stable and additive. In practice, they’re neither. Even if “human-equivalent hours” were the right unit, a flat estimate is almost certainly inaccurate. Eight hours per PR treats every pull request as identical, but PRs vary enormously in size, complexity, and review effort. The research also increasingly suggests that AI and human effort remain largely complements rather than substitutes https://www.nber.org/papers/w35275 . Engineers still spend substantial time reviewing, integrating, validating, testing, and coordinating around AI-generated code. A clean “hours replaced” conversion assumes substitution where the evidence points toward augmentation. Where to start If this sounds like more than you can measure today, start anyway. You do not need to be able to measure all of this with a mature telemetry stack in order to begin. Don’t have the telemetry? Use surveys. If you have precise, telemetry-driven data for every commit, review, build, and deployment, you’re in an uncommonly good position. Most organizations aren’t, and that’s fine. Targeted surveys will get you moving quickly, and they’re better suited to some of these dimensions than telemetry is anyway reinvested time and satisfaction, for instance . The usual objection is self-report bias, but it matters less than people expect here, because you’re watching change over time. Whatever bias exists tends to stay consistent across measurements, so it largely cancels out in the delta. The measure only has to be directionally correct to be useful. Don’t feel lost if you can only start with a couple of metrics. You don’t need all five dimensions on day one. Pick one or two metrics per dimension, balancing an objective signal with a subjective one, and add more as your instrumentation matures. A speed metric from telemetry paired with a satisfaction question from a survey tells you more than five telemetry metrics alone. Too many metrics dilute focus and make it harder to see what’s actually changing. Start small, learn what moves, and expand deliberately. As your instrumentation matures, work toward the full picture. The richer your measurement, the more confidently you can answer whether your capacity to deliver innovation is real and sustainable. The bottom line The pressure to produce one clean number that proves an AI tool’s horsepower is understandable, but it answers the wrong question. The better question is whether your engineering system has more capacity to deliver innovation, and whether that capacity is sustainable. Measure that across a few well-chosen dimensions, lean on the causal work the field has already done, and you’ll have something far more defensible than any single horsepower figure, and far more useful for deciding what to do next. AI shouldn’t be judged by how much human work it appears to replace. It should be judged by whether it gives your engineering organization a greater and more sustainable capacity to deliver innovation. This week’s featured DevProd job openings. See more open roles here https://getdx.com/resources/devex-jobs/ . Ashby is hiring an Staff Platform Engineer https://jobs.ashbyhq.com/Ashby/0f5dbf59-687b-4d88-88a7-73ee0a66b48d?utm source=PRgMeEgv1Z | Remote Cart is hiring a Sr. Software Engineer II, Developer Experience https://www.linkedin.com/jobs/view/4404135082 | Santa Clara, CA; San Francisco, CA; New York, NY Cashea is hiring an Infrastructure & Developer Productivity Platform Engineering Manager https://cashea.na.teamtailor.com/jobs/579773-infrastructure-developer-productivity-platform-engineering-manager | Remote Figma is hiring a Staff Software Engineer, Developer Experience https://job-boards.greenhouse.io/figma/jobs/5790627004?gh jid=5790627004&gh src=db0ijm3x4us | Remote; US Morgan Stanely is hiring an AI Platform Engineer - Vice President https://www.linkedin.com/jobs/view/4393043964/ | New York Notion is hiring a Software Engineer, Developer Experience https://jobs.ashbyhq.com/notion/49bdf081-6e20-4323-8c73-6d6b19544ff5 | Hybrid; Hyderabad, India