{"slug": "is-ai-making-your-teams-better-or-just-busier", "title": "Is AI making your teams better, or just busier?", "summary": "McKinsey's 2025 State of AI survey finds only 39% of organizations report any EBIT impact from AI, despite 88% using AI in at least one business function. The article argues that measuring AI adoption through usage metrics is insufficient and proposes two KPIs to measure AI effectiveness: enabling new outcomes and embedding AI in workflows.", "body_md": "AI adoption programs tend to end in the same place. Tools are accessible, usage is up, and there's a dedicated Slack channel for wins. Six months later, nothing about how the team works has fundamentally changed. People are doing the same things – just slightly faster.\n\nAnd it’s easy for programs to stall when you’re measuring the wrong thing. Adoption (whether people have access and whether they're using the tools) is visible and easy to report. It tells you nothing about whether the team is actually getting better at anything.\n\nAccording to [ McKinsey's 2025 State of AI survey](http://mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai), only 39% of organizations report any EBIT impact from AI – and even then, it's typically less than 5%. That's despite 88% of companies surveyed using AI in at least one business function. That's not an access problem. It's a measurement problem.\n\nThat's the harder question: is AI making teams more capable, or just more busy? Answering it means measuring AI effectiveness – not just AI activity.\n\n## Key takeaways\n\n- Usage metrics measure whether people are using AI. They tell you nothing about whether teams are getting better at anything.\n- Two KPIs separate activity from capability: one measures whether AI is enabling outcomes that weren't possible before; the other measures how embedded AI is in how work actually gets done.\n- Each KPI is scored 1–5 monthly by team leads based on evidence, not assertion. No examples means no score improvement.\n- Most teams that do this honestly start at 2. What matters is trajectory – consistent improvement over time, not a high opening number.\n\n## Why usage is the wrong metric to judge AI adoption\n\nThe obvious way to measure the impact of AI is usage. Is everyone using it? How often?\n\nBut usage tells you nothing about quality. Open an AI tool every day, do the same things slightly faster, and you register as high adoption – without changing what the team is capable of.\n\nAnd the alternative approach of applying traditional engineering metrics doesn't solve this either. When we looked at layering AI measurement on top of cycle time, sprint velocity, and story points, we ran into the same issue from a different direction: AI turns estimation upside down. Engineers can now spin up multiple agents and complete work in hours that would previously have taken a sprint. The math stops working. We decided not to try to make it work – we started fresh with measurements that fit the new reality.\n\nWhat matters comes down to two questions: are people doing things they couldn't do before, and is AI embedded in how they deliver or bolted on around the edges? Those need different measurements.\n\n## The difference between using AI and being good at it\n\nThe people who get the most from AI aren't operating as tool users. They're operating more like managers delegating to a capable but junior team member – defining the task clearly, choosing the right model or tool for the work, reviewing output critically rather than accepting it, and persisting through sessions that don't come together the first time.\n\nThat last point is what separates the people who pull ahead. The teams that make the most progress aren't separated from others by tool access or natural ability. The difference is persistence: months of daily effort, through failed sessions and workflows that took weeks to get right.\n\nOne useful signal that this practice is developing: people start spending more time on the brief than on the output. Being specific enough for an agent to run with a task forces harder thinking about what you actually want from the result. The pressure of that clarity is where the compounding starts.\n\n## Two KPIs for measuring AI effectiveness\n\nWe measure AI effectiveness with two company-level KPIs, each scored 1–5 by team leads on a monthly basis. Individual scores average up to a team level, then the department, and then the company. The scoring basis is always evidence, not assertion: show me what you did, show me why it's better. No examples means no score improvement, because the point is to articulate what's changed, not assert you're using AI more.\n\n### KPI 1: New outcomes unlocked by AI\n\nMeasures whether AI is expanding what's possible: not whether people are using it, but whether it's enabling new outcomes that weren't there before.\n\n### KPI 2: AI as a multiplier\n\nMeasures how embedded AI is in day-to-day delivery: automated workflows running independently, deliverables that couldn't have been produced at that quality, speed, or volume without AI, and output that exceeds what one person could produce alone.\n\nA concrete example of what score 4 looks like: at Ably, we're seeing engineers complete the work that used to get deprioritized – technical debt, maintenance, the improvements that always lost out to feature work – while keeping feature output the same. Previously, you had to choose. Now some engineers are doing both. That's the multiplier.\n\n## Why trajectory matters more than current score\n\nA score reflects the current state. Accountability is about improvement.\n\nA 2 in January is fine. A 2 in March when you were 2 in February is a problem. You're measuring trajectory, not snapshots. Leaders score based on evidence: show me what you did, show me why it's better. No examples means no score improvement, because the point is to articulate what's changed, not assert you're using AI more.\n\nTo keep scores calibrated across teams, we compare them at a leadership level. A 4 in engineering should mean roughly the same thing as a 4 in marketing. Sense-checking prevents grade inflation and keeps the company-level number honest.\n\nThe first time a team lead scores someone and can't find an example, it's uncomfortable. That discomfort is the point. It means either the person isn't doing it, or they're not talking about it. Both are things to fix.\n\n## What to put in place to drive improvement\n\nMeasurement creates accountability. It doesn't create capability on its own. These are the mechanisms we've found necessary to move scores rather than just track them.\n\n**Monthly scorecards are reviewed together.** Team leads score their people, review them together, and capture wins in a structured format: the task, what was done, and what was qualitatively different. \"I saved two hours\" is useful. \"I produced something that would have taken a week and two people\" is the goal.\n\n**AI as a dimension in the progression framework.** Explicit expectations by level for what good looks like with AI. Not a separate track, but part of how you define effectiveness at every seniority.\n\n**A shared skill repository.** Everyone contributes at least one skill and improves at least one. The expectation is ongoing participation: build what you've learned, improve what exists, share what's working. At Ably, our skill repository lives in GitHub and is accessible via our internal MCP, so anyone in the company can use it through their AI assistant without needing to configure tooling locally. The repository gets better because everyone uses it, not because one person maintains it.\n\n**An AI tax on new processes and spend.** Any new process proposal has to answer how AI would handle it. Any new tool or subscription request has to answer whether AI could already do this. It's a forcing function for thinking AI-first rather than as an afterthought.\n\n**Regular sharing in the flow of work.** Teams ask what they did that week that they couldn't have done without AI – in standups, in async channels, in whatever rhythm fits. We're around 80 people, which means there's no excuse for learnings to stay within a single team. If people aren't talking about what they're doing, the organization isn't learning.\n\n## Where most teams start – and where Ably is now\n\nHonest self-assessment matters when first adopting AI. Most teams starting this process score around 2 on the first round, not 3. That was true at Ably. Our first scoring run in January came in at 2 across both KPIs, with a couple of teams short of that.\n\nThat result wasn't disappointing – it was useful. It showed the metric was working and gave us an honest baseline. Starting high is the thing to be skeptical of.\n\nMost of our teams are now at 3 or approaching it. The challenge is getting to 4 and 5, which is the EOY target. A score of 3 means AI is embedded in some workflows and some automation is running. A score of 4 means it's the default for most work. Getting there requires persistence through the uncomfortable early rounds – not a new tool or a new process.\n\nWhat measurement reveals that adoption metrics never will: whether the team is fundamentally different because of AI, or just slightly faster. Most teams that score honestly find they're at 2. That number is useful – it's the starting point, not the verdict. We'll share what the scores look like as they move.\n\nThis is the second article in a series on how Ably is building an AI-first organization. The first, [ How we built an AI-first culture at Ably](https://ably.com/blog/building-ai-first-culture-at-ably), covers how we got here, if you'd like to read more.\n\n*Is realtime AI part of how your team ships? **See what Ably AI Transport adds.*", "url": "https://wpnews.pro/news/is-ai-making-your-teams-better-or-just-busier", "canonical_source": "https://ably.com/blog/measure-ai-effectiveness-teams", "published_at": "2026-06-26 10:45:49+00:00", "updated_at": "2026-06-26 11:08:12.485636+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-products", "ai-tools", "ai-research", "ai-ethics"], "entities": ["McKinsey"], "alternates": {"html": "https://wpnews.pro/news/is-ai-making-your-teams-better-or-just-busier", "markdown": "https://wpnews.pro/news/is-ai-making-your-teams-better-or-just-busier.md", "text": "https://wpnews.pro/news/is-ai-making-your-teams-better-or-just-busier.txt", "jsonld": "https://wpnews.pro/news/is-ai-making-your-teams-better-or-just-busier.jsonld"}}