{"slug": "metrics-for-ai-generated-code-measure-impact-not-people", "title": "Metrics for AI-generated code: measure impact, not people", "summary": "A developer has proposed a four-metric framework for measuring the impact of AI-generated code on engineering teams, arguing that tool-supplied figures like acceptance rates and lines of code say little about whether AI actually helps. The approach centers on follow-up defects after merge, rework in review, review time in risky areas, and a short team pulse check, with the author warning that per-person evaluation turns metrics into a measure of how well people hide. The piece cites METR's early-2025 study, in which 16 experienced open-source developers took 19 percent longer with AI on 246 real tasks while estimating they had been 20 percent faster, and the 2025 DORA report, which found AI adoption positively related to throughput but negatively related to delivery stability.", "body_md": "Metrics for AI-generated code are a touchy subject in many teams right now. Management wants to know whether the licenses pay off. The team senses that something is changing but finds it hard to pin down. And as soon as someone says “dashboard”, the first developers look skeptical, because they suspect that people will end up being compared after all.\n\nThis question keeps me busy because I understand both sides. Without numbers, all you have is gut feeling, and with AI that feeling is wrong surprisingly often. With the wrong numbers, however, you do more harm than with none. In this article I show you a small set of four metrics that tell you something about the impact of AI on your team, ground rules so that nobody feels watched, and a script that pulls the numbers from GitHub. My article [Automation bias: when teams trust AI blindly](https://sascha-ms.de/en/automation-bias-ai-teams?utm_source=devto&utm_medium=crosspost) was about the problem. This one is about making it visible.\n\n**In short**\n\nAcceptance rates and lines of code say little about whether AI helps your team. Follow-up defects after the merge, rework in review, review time in risky areas and a short team pulse check are far more telling. The biggest pitfall is evaluating these numbers per person: from that moment on, you only measure how well people hide.\n\n**What you will learn in this article**\n\nThe honest answer: because our gut feeling about AI tools is unreliable. In early 2025, the research organization METR ran a controlled study with 16 experienced open-source developers. They worked on 246 real tasks from their own large repositories, randomly split into “AI allowed” and “AI not allowed”. With AI, they took 19 percent longer on average. At the same time, they estimated afterwards that AI had made them 20 percent faster.\n\nBefore anyone passes the study around as proof against AI: the authors themselves stress that it is a snapshot of early-2025 tools, in a specific setting, with few participants. Their page now even notes that the results are out of date. My takeaway is a different one. The gap between feeling and measurement was huge. That probably applies to your team as well, just in a direction you don’t know yet.\n\nThe 2025 DORA report on AI-assisted software development shows a similar pattern from another angle. Around 90 percent of the nearly 5,000 respondents use AI at work, and more than 80 percent feel more productive because of it. According to the report, AI adoption is now positively related to throughput, but still negatively related to delivery stability. DORA therefore describes AI as an amplifier that magnifies an organization’s existing strengths and weaknesses. If your tests are weak and your feedback loops long, AI mainly gets you more changes that blow up later.\n\nThat is exactly why you need metrics for AI-generated code. Whether the licenses pay off is the smaller question. More important is seeing early whether your team is actually getting faster or just more hectic.\n\nThe most convenient numbers come from the tools themselves. For Copilot, for example, GitHub shows active users, the acceptance rate of inline suggestions and lines of code suggested or accepted. These numbers answer one question well: is the tool being used? They say little about its impact. A high acceptance rate can mean that the suggestions are good. It can just as well mean that nobody looks closely anymore, and then we are right in the middle of automation bias.\n\nIt gets worse when such numbers become targets. Goodhart’s law sums it up: once a measure becomes a target, it stops being a good measure. If you demand that 40 percent of the code be AI-generated, you will get 40 percent AI code. Whether it is any good, you will know less than before.\n\nMy list of numbers I don’t use to judge the impact of AI:\n\nThe SPACE framework by Nicole Forsgren and colleagues explicitly warns against using activity metrics in isolation to reward or penalize developers. It recommends several metrics from different dimensions, at least one of them from surveys, and reporting only aggregated results at team or group level. The set I am about to show you follows that advice.\n\nThe basic idea is simple. Pull requests in which AI played a significant role get a label. Then you compare three metrics between pull requests with and without this label and add a short pulse check in the team. Everything stays at team level, without names.\n\n*Four metrics, always compared between pull requests with and without the AI label*\n\nHow many merged changes have to be reverted or fixed later? This is the team version of the DORA metric “change fail rate”, which measures the share of deployments that need immediate intervention afterwards, such as a rollback or hotfix. To compare with and without AI, you need a link between the defect and its cause. Two conventions make this easy: when you revert, GitHub automatically writes “Reverts owner/repo#42” into the description, and for fixes you agree on the phrase “Follow-up defect of #42”.\n\nThis metric matters most to me because it measures what DORA describes as the weak spot: more speed, less stability. If it rises clearly for AI pull requests, that is a signal about your tests and reviews, not about the people who use AI.\n\nWhat share of pull requests gets “Changes requested” at least once? This number cuts both ways, which is exactly what makes it valuable. If it is clearly higher for AI code, the review is apparently thorough but costs time. If it is conspicuously low for AI code, you should ask whether reviews are still as critical as for handwritten code. So a low number is not automatically good here.\n\nThis is the time from opening to merge, split into normal areas and risk zones such as infrastructure, payments or authentication. You may know the idea of risk zones from my article on the [pull request template for AI-generated code](https://sascha-ms.de/en/pull-request-template-ai-code?utm_source=devto&utm_medium=crosspost). In normal areas, AI code may well move faster. In risk zones I would get suspicious if AI pull requests were suddenly merged faster than the rest, because that is where people are supposed to look more closely.\n\nThe fourth metric doesn’t come from GitHub but from people’s heads. SPACE recommends at least one perceptual measure, and with AI this is especially important because the workload often shifts: less typing, more reading, checking and doubting. I would ask three anonymous questions once a month, each on a scale from 1 to 5:\n\nThe third question is my favorite. When understanding of your own code drops, the team pays the bill at the next incident at two in the morning at the latest.\n\nThe best metric is worthless if the team distrusts it. People simply stop setting the “ai-assisted” label, and your numbers only measure who is honest. That is why I have a few fixed ground rules, and I discuss them with the team before the first number is collected.\n\nIn Germany there is one more point that many leaders notice too late. Under Section 87 (1) No. 6 of the Works Constitution Act, the works council has a right of co-determination over the introduction and use of technical systems designed to monitor employees’ behavior or performance. As a non-lawyer I can’t judge whether an evaluation script falls under this, and it depends on your agreements. My advice is pragmatic: talk to the works council before you start and show exactly what is collected and what isn’t. If your country has similar rules, the same applies. A script that doesn’t read any names makes that conversation much more relaxed.\n\nNow it gets practical. You need no analytics tool and no database. The GitHub CLI exports merged pull requests as JSON, and a small TypeScript script turns them into a table. The example is a fictional online bookshop with the repository `acme/bookshop`.\n\n`gh`), logged in with read access to the repository\nCreate the label once in the repository. Agree in the team to set it when AI wrote a significant part of the change, meaning more than individual completions. There is no perfect definition. What matters more is that everyone interprets it similarly.\n\n```\ngh label create \"ai-assisted\" --repo acme/bookshop --color F97316 --description \"AI wrote a significant part of this change\"\n```\n\nIn addition, you agree on the phrase “Follow-up defect of #NUMBER” in the description of every pull request that fixes a defect from an earlier pull request. The script detects reverts automatically.\n\nAfter four to eight weeks, you export the merged pull requests. Note that the `author` field is deliberately missing.\n\n```\ngh pr list --repo acme/bookshop --state merged --limit 500 \\\n  --search \"merged:>=2026-09-01\" \\\n  --json number,title,body,createdAt,mergedAt,labels,reviews,files > prs.json\n```\n\nThe script splits the pull requests into two groups, looks for references to follow-up defects, counts reviews with “Changes requested” and calculates the median time to merge. I use the median instead of the average because otherwise a single pull request that sat around for three weeks distorts everything. Adjust the risk zone paths to your repository.\n\n```\n// ai-metrics.mts\n// Evaluates merged pull requests at team level: with and without the \"ai-assisted\" label.\n// Usage: node ai-metrics.mts prs.json\nimport { readFileSync } from \"node:fs\";\n\ntype Review = { state: string };\ntype PullRequest = {\n  number: number;\n  title: string;\n  body: string;\n  createdAt: string;\n  mergedAt: string | null;\n  labels: { name: string }[];\n  reviews: Review[];\n  files: { path: string }[];\n};\n\nconst AI_LABEL = \"ai-assisted\";\nconst RISK_PATHS = [\"infra/\", \"src/payment/\", \"src/auth/\"]; // adjust to your repo\nconst MIN_GROUP_SIZE = 5; // smaller groups are not evaluated\n\n// \"Reverts acme/bookshop#42\" (GitHub revert) or \"Follow-up defect of #42\" (team convention)\nconst REFERENCE = /(?:Reverts [\\w.-]+\\/[\\w.-]+#|Follow-up defect of #)(\\d+)/g;\n\nfunction median(values: number[]): number {\n  const s = [...values].sort((a, b) => a - b);\n  const m = Math.floor(s.length / 2);\n  return s.length % 2 ? s[m] : (s[m - 1] + s[m]) / 2;\n}\n\nfunction percent(part: number, whole: number): string {\n  return `${Math.round((part / whole) * 100)} %`;\n}\n\nexport function evaluate(prs: PullRequest[]): string[] {\n  const merged = prs.filter((pr) => pr.mergedAt);\n\n  // Which PRs were later reverted or named as the cause of a follow-up defect?\n  const withDefect = new Set<number>();\n  for (const pr of merged) {\n    for (const match of `${pr.title}\\n${pr.body}`.matchAll(REFERENCE)) {\n      withDefect.add(Number(match[1]));\n    }\n  }\n\n  const groups = {\n    \"with AI\": merged.filter((pr) => pr.labels.some((l) => l.name === AI_LABEL)),\n    \"without AI\": merged.filter((pr) => !pr.labels.some((l) => l.name === AI_LABEL)),\n  };\n\n  const rows = [\n    \"| Group | PRs | Follow-up defects | Rework in review | Median to merge (normal) | Median to merge (risk zone) |\",\n    \"|---|---|---|---|---|---|\",\n  ];\n\n  for (const [name, list] of Object.entries(groups)) {\n    if (list.length < MIN_GROUP_SIZE) {\n      rows.push(`| ${name} | ${list.length} | not enough data | – | – | – |`);\n      continue;\n    }\n    const defects = list.filter((pr) => withDefect.has(pr.number)).length;\n    const rework = list.filter((pr) => pr.reviews.some((r) => r.state === \"CHANGES_REQUESTED\")).length;\n    const hours = (pr: PullRequest) =>\n      (Date.parse(pr.mergedAt as string) - Date.parse(pr.createdAt)) / 3_600_000;\n    const isRisk = (pr: PullRequest) =>\n      pr.files.some((f) => RISK_PATHS.some((path) => f.path.startsWith(path)));\n    const normal = list.filter((pr) => !isRisk(pr)).map(hours);\n    const risk = list.filter(isRisk).map(hours);\n    const time = (w: number[]) => (w.length >= MIN_GROUP_SIZE ? `${median(w).toFixed(1)} h` : \"not enough data\");\n    rows.push(\n      `| ${name} | ${list.length} | ${percent(defects, list.length)} | ${percent(rework, list.length)} | ${time(normal)} | ${time(risk)} |`,\n    );\n  }\n  return rows;\n}\n\nif (process.argv[2]) {\n  const prs = JSON.parse(readFileSync(process.argv[2], \"utf8\")) as PullRequest[];\n  console.log(evaluate(prs).join(\"\\n\"));\n}\nnode ai-metrics.mts prs.json\nrm prs.json\n```\n\nWith test data, the output looks like this. The table is Markdown, so you can paste it straight into a retro page or an issue.\n\n```\n| Group | PRs | Follow-up defects | Rework in review | Median to merge (normal) | Median to merge (risk zone) |\n|---|---|---|---|---|---|\n| with AI | 20 | 5 % | 20 % | 5.0 h | 5.0 h |\n| without AI | 22 | 5 % | 18 % | 4.0 h | 6.0 h |\n```\n\nHow do you read this? In this example, follow-up defects and rework are similar in both groups. The risk zone stands out: there, AI pull requests get through faster than the others. That is no proof of sloppy reviews, but it is a good question for the next retro.\n\n`--limit`, `gh pr list` returns only 30 entries. Then you only evaluate the last few days.\nNumbers alone change nothing. I would hold a retro of about 45 minutes on this topic once a quarter. First, the team spends ten minutes silently looking at the table and the pulse check results. Everyone writes down what surprised them and what they expected. Then you talk about these questions:\n\nYou end with a single experiment for the next quarter, for example a second review for AI code in infrastructure or a new kind of test. Afterwards you compare whether the metric has moved. If you want to record decisions about this, [architecture decision records](https://sascha-ms.de/en/architecture-decision-records-template?utm_source=devto&utm_medium=crosspost) work for team rules too.\n\nAs a leader, your main job here is to listen and not to look for culprits. If someone in this round says “That was so-and-so’s PR”, that is the moment to repeat the ground rules, kindly but clearly.\n\nI don’t believe in proving the impact of AI with a single number, whether it speaks for or against AI. The tools change so quickly that every study is history after six months. What remains are the questions: Do we deliver reliably? Do we still understand our code? Are people doing well with it? Four metrics and an honest conversation are enough for that.\n\nI would push back on anyone who wants to build rankings from such numbers, even with good intentions. The moment a developer considers leaving off the AI label is the moment your entire measurement system breaks. Trust is not a soft extra here; it is the precondition for the data being right at all. People first applies to metrics too.\n\nPreferably not with a single productivity number. A small set of outcome metrics such as follow-up defects and review rework works better, combined with a regular anonymous survey and evaluated at team level.\n\nIt shows whether the tool is used and whether the suggestions generally fit. It says nothing about quality and stability, and as a target it rewards accepting suggestions uncritically.\n\nI advise against it because it destroys trust and with it the quality of the data. In Germany, the works council also has co-determination rights over technical systems for monitoring performance or behavior. Clarify this with them beforehand and, if in doubt, with your legal department.\n\nAt least several weeks, and preferably a quarter. In small teams, individual pull requests shift the rates considerably, so trends over several periods are more meaningful than a single value.\n\nCreate the label this week and bring the ground rules to your next retro. Let the team decide which paths count as risk zones. In six weeks you will have enough data for a first table. And I’m curious: which metric do you use today to assess the impact of AI on your team, and how much do you trust it? Feel free to share in the comments.\n\n*This post was created with AI assistance.*\n\n👉 **Next step:** More for team leads: my [cheat sheets](https://sascha-ms.de/en/cheat-sheets/?utm_source=devto&utm_medium=crosspost) put the essentials on one printable page.\n\n📬 **Stay updated:** New articles on serverless, AI and leading tech teams via [RSS, browser push or the weekly newsletter (in German)](https://sascha-ms.de/en/stay-updated/?utm_source=devto&utm_medium=crosspost).\n\n*Originally published on [sascha-ms.de](https://sascha-ms.de/en/metrics-ai-generated-code?utm_source=devto&utm_medium=crosspost).* More articles on serverless, AI in everyday development and leading tech teams: [sascha-ms.de/en](https://sascha-ms.de/en/?utm_source=devto&utm_medium=crosspost)", "url": "https://wpnews.pro/news/metrics-for-ai-generated-code-measure-impact-not-people", "canonical_source": "https://dev.to/sascha-ms/metrics-for-ai-generated-code-measure-impact-not-people-4f5b", "published_at": "2026-11-12 15:49:03+00:00", "updated_at": "2026-11-12 15:52:10.926555+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "ai-products"], "entities": ["METR", "DORA", "GitHub", "Copilot", "Nicole Forsgren"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/metrics-for-ai-generated-code-measure-impact-not-people", "markdown": "https://wpnews.pro/news/metrics-for-ai-generated-code-measure-impact-not-people.md", "text": "https://wpnews.pro/news/metrics-for-ai-generated-code-measure-impact-not-people.txt", "jsonld": "https://wpnews.pro/news/metrics-for-ai-generated-code-measure-impact-not-people.jsonld"}}