How to Measure Engineering Productivity With AI Coding Tools A developer's analysis argues that lines of code, prompts and accepted suggestions measure activity rather than engineering progress, citing research including METR's 2025 trial in which experienced developers were 19% slower with AI while believing they were 20% faster, and DORA's 2024 finding of a 1.5% drop in delivery throughput and 7.2% drop in stability associated with AI adoption. The piece proposes a four-layer monthly scorecard covering adoption, trust, acceleration and impact, keeping each layer separate so adoption numbers never substitute for delivered value. Lines of code, prompts and accepted suggestions measure activity, not progress. Here is what the research says about AI coding tools and a four-layer scorecard to use instead. To measure engineering productivity with AI coding tools, track delivery speed, stability and product outcomes, not lines of code, prompts or accepted suggestions. Those are activity signals, and AI inflates them for free. Controlled research shows why it matters: in METR's 2025 trial, experienced developers were 19% slower with AI while believing they were 20% faster. Measure the system around the tool, in four layers. Key takeaways Lines of code were always a weak proxy. With AI coding tools they become actively misleading, because generating code now costs almost nothing. A dashboard can fill up with commits, pull requests, generated tests and accepted suggestions while the product stands still. Three pieces of evidence show the distortion. The SPACE framework, from Nicole Forsgren and colleagues at Microsoft Research and GitHub, made the general point in 2021: developer productivity cannot be measured by a single metric or by activity levels alone Forsgren et al., 2021 https://www.microsoft.com/en-us/research/publication/the-space-of-developer-productivity-theres-more-to-it-than-you-think/ . Its five dimensions are satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. Activity is one of five, not the whole picture. The risk is a feedback loop: the organisation rewards visible output, engineers generate more, reviewers inherit the load, and leaders mistake motion for progress. We call this the product velocity illusion https://www.specky.space/blog/product-velocity-illusion . The results depend heavily on the task, the codebase and the developer. Here is the evidence side by side. | Study | Setting | Finding | |---|---|---| | Peng et al., 2023 | Controlled task: build an HTTP server in JavaScript | Copilot group finished 55.8% faster arXiv https://arxiv.org/abs/2302.06590 | | METR, July 2025 | RCT: 16 experienced developers, 246 real issues in large open-source repos | 19% slower with AI; developers believed 20% faster METR https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ | | METR, February 2026 | 57 developers, 800+ tasks, late-2025 tools | Estimates of 18% and 4% faster, but confidence intervals include zero; METR calls it weak evidence METR https://metr.org/blog/2026-02-24-uplift-update/ | | DORA, 2024 | Survey-based modelling | AI adoption associated with an estimated 1.5% drop in delivery throughput and 7.2% drop in stability Google Cloud https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report | | DORA, 2025 | Nearly 5,000 professionals | AI now linked to higher throughput, still to lower stability Google Cloud https://cloud.google.com/blog/products/ai-machine-learning/announcing-the-2025-dora-report | | Faros AI, 2025 | Telemetry, 10,000+ developers, 1,255 teams | 21% more tasks, 98% more PRs, no company-level gain | | Stack Overflow, 2025 | Survey, 30,000+ developers | 52% say AI improved their productivity; 45% say debugging AI code takes longer Stack Overflow https://survey.stackoverflow.co/2025/ai | Three lessons for managers: DORA's 2024 data also showed real benefits: for every 25% increase in AI adoption, documentation quality rose an estimated 7.5%, code quality 3.4% and code review speed 3.1%. AI helps. The question is whether your system turns that help into delivered value. Google's four-phase framework adoption, trust, acceleration, impact and DORA's delivery metrics combine into a scorecard you can run monthly. Keep each layer separate so adoption numbers never stand in for impact. | Layer | Question | Metrics | Warning sign | |---|---|---|---| | 1. Adoption | Are engineers using the tools? | Weekly active users, share of changes with AI assistance | Treated as a target in itself | | 2. Trust | Do engineers trust the output appropriately? | Acceptance rate, survey on trust, share of AI changes reworked in review | Very high acceptance plus rising change fail rate | | 3. Delivery | Is the system faster and stable? | DORA's five metrics: change lead time, deployment frequency, failed deployment recovery time, change fail rate, deployment rework rate DORA https://dora.dev/guides/dora-metrics-four-keys/ | Throughput up, stability down | | 4. Product impact | Did customers or the business benefit? | Target user behaviour, adoption of shipped features, retention, support load | Many releases, flat outcomes | Add two team-health signals that AI tends to move: median PR size and median time to first review. They are the earliest warnings that output is outrunning review capacity. Google's guidance names defining success criteria and a baseline as the most important and most often skipped step. A practical sequence: Say a six-engineer team adopts an AI coding agent. After eight weeks, the dashboard shows lines changed up 140% and merged PRs up 60%. Change lead time fell from 3.0 days to 2.2 days. But change fail rate rose from 8% to 14%, median PR size doubled, and time to first review went from 4 hours to 10. A lines-of-code dashboard calls this a triumph. The four-layer scorecard says adoption and throughput improved while stability got worse. The fix: cap PR size, add the PM's acceptance check before code review, and require agents to list untested paths. Four weeks later, change fail rate is back to 9% and lead time holds at 2.4 days. That is a real gain. Make six things explicit for each piece of work: Then ask the team to make generated work legible: files and systems changed, assumptions the agent made, tests run, untested paths, rollback plan, and a link back to the product decision. Treat generated code as a proposed change, not an authority. The PM's side of this review is covered in how to review what AI agents build https://www.specky.space/blog/review-what-ai-agents-build-pm-acceptance-checklist . Give engineers the problem, the evidence, the acceptance criteria, the constraints and the intended outcome, then let engineering own the technical design. Agents only know what is written down, so the brief carries more weight than it did. A structured AI coding agent PRD template https://www.specky.space/blog/ai-coding-agent-prd-template and the product context an AI coding agent needs https://www.specky.space/blog/product-context-for-ai-coding-agents both help. Keep slices small. DORA lists working in small batches among the seven capabilities that amplify AI's benefits, alongside a clear AI stance, AI-accessible internal data and strong version control practices. Replace "how many lines did we produce?" with seven questions: Setting adoption targets. "80% of engineers using AI daily" is an input. Teams will hit it without improving anything. Trusting self-reports alone. METR's perception gap is the warning. Pair surveys with delivery data. Ignoring stability. Throughput gains that come with more rollbacks and hotfixes are borrowed time. Comparing unlike teams. A senior team with AI against a junior team without it tells you nothing about AI. Skipping the policy. Define approved tools, data boundaries, review thresholds and the sensitive actions that need human approval. Our guide to how AI changed product team workload https://www.specky.space/blog/ai-created-new-product-workload-pm-engineers-builders covers the wider operating contract. Specky keeps product and engineering connected to the reason for the work. Its Product Graph links customer signals, research, decisions, PRDs, tickets and outcomes, and its MCP server lets coding agents read that context before they plan. That gives managers a loop beyond code activity: signal, decision, implementation, review, release, outcome. It is not another engineering dashboard; it stops the product reasoning getting lost between the roadmap and the repository. No. Lines of code and accepted suggestions are activity measures that AI inflates at no cost. GitClear's data shows copy-pasted code rising and refactoring falling as AI use grew. Track DORA's delivery metrics, review time, change size and the customer outcome each change was meant to move instead. No single metric works. Use four layers: adoption are tools used , trust is output accepted appropriately , delivery DORA's throughput and stability metrics and product impact did customer behaviour or business results change . Report them separately so adoption never stands in for impact. It depends. A 2023 controlled task found developers 55.8% faster with Copilot. METR's 2025 trial on mature open-source codebases found experienced developers 19% slower. METR's 2026 follow-up points towards modest speedups but with wide uncertainty. Gains are largest on well-defined, greenfield tasks. At least six to eight weeks after a baseline period. Google recommends four two-week sprints before judging acceleration, because engineers need time to learn the tools and early numbers mostly reflect adoption. Agree success criteria with engineers and business leads before the pilot starts. Approved tools, the data each tool may access, review thresholds by risk, actions that always need human approval, how generated changes are tested and audited, and who owns production incidents. DORA calls this a clear and communicated AI stance and lists it first among the capabilities that make AI adoption pay off. This story was originally published on 15 October 2026 at https://www.specky.space/blog/manage-engineers-ai-coding-tools-without-lines-of-code https://www.specky.space/blog/manage-engineers-ai-coding-tools-without-lines-of-code .