Metrics for AI-generated code: measure impact, not people A developer has proposed a four-metric framework for measuring the impact of AI-generated code on engineering teams, arguing that tool-supplied figures like acceptance rates and lines of code say little about whether AI actually helps. The approach centers on follow-up defects after merge, rework in review, review time in risky areas, and a short team pulse check, with the author warning that per-person evaluation turns metrics into a measure of how well people hide. The piece cites METR's early-2025 study, in which 16 experienced open-source developers took 19 percent longer with AI on 246 real tasks while estimating they had been 20 percent faster, and the 2025 DORA report, which found AI adoption positively related to throughput but negatively related to delivery stability. Metrics for AI-generated code are a touchy subject in many teams right now. Management wants to know whether the licenses pay off. The team senses that something is changing but finds it hard to pin down. And as soon as someone says “dashboard”, the first developers look skeptical, because they suspect that people will end up being compared after all. This question keeps me busy because I understand both sides. Without numbers, all you have is gut feeling, and with AI that feeling is wrong surprisingly often. With the wrong numbers, however, you do more harm than with none. In this article I show you a small set of four metrics that tell you something about the impact of AI on your team, ground rules so that nobody feels watched, and a script that pulls the numbers from GitHub. My article Automation bias: when teams trust AI blindly https://sascha-ms.de/en/automation-bias-ai-teams?utm source=devto&utm medium=crosspost was about the problem. This one is about making it visible. In short Acceptance rates and lines of code say little about whether AI helps your team. Follow-up defects after the merge, rework in review, review time in risky areas and a short team pulse check are far more telling. The biggest pitfall is evaluating these numbers per person: from that moment on, you only measure how well people hide. What you will learn in this article The honest answer: because our gut feeling about AI tools is unreliable. In early 2025, the research organization METR ran a controlled study with 16 experienced open-source developers. They worked on 246 real tasks from their own large repositories, randomly split into “AI allowed” and “AI not allowed”. With AI, they took 19 percent longer on average. At the same time, they estimated afterwards that AI had made them 20 percent faster. Before anyone passes the study around as proof against AI: the authors themselves stress that it is a snapshot of early-2025 tools, in a specific setting, with few participants. Their page now even notes that the results are out of date. My takeaway is a different one. The gap between feeling and measurement was huge. That probably applies to your team as well, just in a direction you don’t know yet. The 2025 DORA report on AI-assisted software development shows a similar pattern from another angle. Around 90 percent of the nearly 5,000 respondents use AI at work, and more than 80 percent feel more productive because of it. According to the report, AI adoption is now positively related to throughput, but still negatively related to delivery stability. DORA therefore describes AI as an amplifier that magnifies an organization’s existing strengths and weaknesses. If your tests are weak and your feedback loops long, AI mainly gets you more changes that blow up later. That is exactly why you need metrics for AI-generated code. Whether the licenses pay off is the smaller question. More important is seeing early whether your team is actually getting faster or just more hectic. The most convenient numbers come from the tools themselves. For Copilot, for example, GitHub shows active users, the acceptance rate of inline suggestions and lines of code suggested or accepted. These numbers answer one question well: is the tool being used? They say little about its impact. A high acceptance rate can mean that the suggestions are good. It can just as well mean that nobody looks closely anymore, and then we are right in the middle of automation bias. It gets worse when such numbers become targets. Goodhart’s law sums it up: once a measure becomes a target, it stops being a good measure. If you demand that 40 percent of the code be AI-generated, you will get 40 percent AI code. Whether it is any good, you will know less than before. My list of numbers I don’t use to judge the impact of AI: The SPACE framework by Nicole Forsgren and colleagues explicitly warns against using activity metrics in isolation to reward or penalize developers. It recommends several metrics from different dimensions, at least one of them from surveys, and reporting only aggregated results at team or group level. The set I am about to show you follows that advice. The basic idea is simple. Pull requests in which AI played a significant role get a label. Then you compare three metrics between pull requests with and without this label and add a short pulse check in the team. Everything stays at team level, without names. Four metrics, always compared between pull requests with and without the AI label How many merged changes have to be reverted or fixed later? This is the team version of the DORA metric “change fail rate”, which measures the share of deployments that need immediate intervention afterwards, such as a rollback or hotfix. To compare with and without AI, you need a link between the defect and its cause. Two conventions make this easy: when you revert, GitHub automatically writes “Reverts owner/repo 42” into the description, and for fixes you agree on the phrase “Follow-up defect of 42”. This metric matters most to me because it measures what DORA describes as the weak spot: more speed, less stability. If it rises clearly for AI pull requests, that is a signal about your tests and reviews, not about the people who use AI. What share of pull requests gets “Changes requested” at least once? This number cuts both ways, which is exactly what makes it valuable. If it is clearly higher for AI code, the review is apparently thorough but costs time. If it is conspicuously low for AI code, you should ask whether reviews are still as critical as for handwritten code. So a low number is not automatically good here. This is the time from opening to merge, split into normal areas and risk zones such as infrastructure, payments or authentication. You may know the idea of risk zones from my article on the pull request template for AI-generated code https://sascha-ms.de/en/pull-request-template-ai-code?utm source=devto&utm medium=crosspost . In normal areas, AI code may well move faster. In risk zones I would get suspicious if AI pull requests were suddenly merged faster than the rest, because that is where people are supposed to look more closely. The fourth metric doesn’t come from GitHub but from people’s heads. SPACE recommends at least one perceptual measure, and with AI this is especially important because the workload often shifts: less typing, more reading, checking and doubting. I would ask three anonymous questions once a month, each on a scale from 1 to 5: The third question is my favorite. When understanding of your own code drops, the team pays the bill at the next incident at two in the morning at the latest. The best metric is worthless if the team distrusts it. People simply stop setting the “ai-assisted” label, and your numbers only measure who is honest. That is why I have a few fixed ground rules, and I discuss them with the team before the first number is collected. In Germany there is one more point that many leaders notice too late. Under Section 87 1 No. 6 of the Works Constitution Act, the works council has a right of co-determination over the introduction and use of technical systems designed to monitor employees’ behavior or performance. As a non-lawyer I can’t judge whether an evaluation script falls under this, and it depends on your agreements. My advice is pragmatic: talk to the works council before you start and show exactly what is collected and what isn’t. If your country has similar rules, the same applies. A script that doesn’t read any names makes that conversation much more relaxed. Now it gets practical. You need no analytics tool and no database. The GitHub CLI exports merged pull requests as JSON, and a small TypeScript script turns them into a table. The example is a fictional online bookshop with the repository acme/bookshop . gh , logged in with read access to the repository Create the label once in the repository. Agree in the team to set it when AI wrote a significant part of the change, meaning more than individual completions. There is no perfect definition. What matters more is that everyone interprets it similarly. gh label create "ai-assisted" --repo acme/bookshop --color F97316 --description "AI wrote a significant part of this change" In addition, you agree on the phrase “Follow-up defect of NUMBER” in the description of every pull request that fixes a defect from an earlier pull request. The script detects reverts automatically. After four to eight weeks, you export the merged pull requests. Note that the author field is deliberately missing. gh pr list --repo acme/bookshop --state merged --limit 500 \ --search "merged: =2026-09-01" \ --json number,title,body,createdAt,mergedAt,labels,reviews,files prs.json The script splits the pull requests into two groups, looks for references to follow-up defects, counts reviews with “Changes requested” and calculates the median time to merge. I use the median instead of the average because otherwise a single pull request that sat around for three weeks distorts everything. Adjust the risk zone paths to your repository. // ai-metrics.mts // Evaluates merged pull requests at team level: with and without the "ai-assisted" label. // Usage: node ai-metrics.mts prs.json import { readFileSync } from "node:fs"; type Review = { state: string }; type PullRequest = { number: number; title: string; body: string; createdAt: string; mergedAt: string | null; labels: { name: string } ; reviews: Review ; files: { path: string } ; }; const AI LABEL = "ai-assisted"; const RISK PATHS = "infra/", "src/payment/", "src/auth/" ; // adjust to your repo const MIN GROUP SIZE = 5; // smaller groups are not evaluated // "Reverts acme/bookshop 42" GitHub revert or "Follow-up defect of 42" team convention const REFERENCE = / ?:Reverts \w.- +\/ \w.- + |Follow-up defect of \d+ /g; function median values: number : number { const s = ...values .sort a, b = a - b ; const m = Math.floor s.length / 2 ; return s.length % 2 ? s m : s m - 1 + s m / 2; } function percent part: number, whole: number : string { return ${Math.round part / whole 100 } % ; } export function evaluate prs: PullRequest : string { const merged = prs.filter pr = pr.mergedAt ; // Which PRs were later reverted or named as the cause of a follow-up defect? const withDefect = new Set