Scaling AI Agents: What We Learned From 9 Real-World Cases

JetBrains published a report, "How Top Teams Scale AI Agents," documenting nine case studies on how engineering organizations deploy AI coding agents, including Fin's results of 3x pull request throughput, a 54% reduction in its defect backlog, and 50% lower cost per PR over 16 months. The report also cites an organization that reached 95% AI adoption without any gain in developer velocity, and a Skiller Whale customer where a group trained to supervise agents roughly doubled PR throughput while the rest of the organization declined. JetBrains argues that agent scaling depends on fixing the surrounding development workflow — standardized build tools, fast tests, and shared skills, plugins, MCP servers and evaluations — rather than adding more agents.

Agentic AI https://blog.jetbrains.com/blog/category/agentic-ai/ Best Practices https://blog.jetbrains.com/blog/category/best-practices/ JetBrains Air https://blog.jetbrains.com/blog/category/jetbrains-air/ Scaling AI Agents: What We Learned From 9 Real-World Cases Our brand-new report explores what it takes for software development leaders to put agents to work. One organization reached 95% AI adoption without seeing an improvement in developer velocity. Another tripled its pull request throughput while reducing its defect backlog by 54%. What explains the difference? To find out what makes AI agents effective beyond individual developers, I spoke with engineering leaders and practitioners from Fin, Semrush, Skiller Whale, and other organizations. The findings are now available in our new report, How Top Teams Scale AI Agents https://lp.jetbrains.media/how-top-teams-scale-ai-agents-report/ , featuring nine real-world case studies and practitioner accounts, including insights from our own engineering teams at JetBrains. The report looks beyond adoption rates and individual productivity gains to explore what actually changes when teams start relying on AI agents – how they organize work, maintain code quality, share knowledge, manage costs, and measure results. Here are three lessons that stood out to me. 1. More agents won’t fix a broken development workflow At Fin, the initial goal was to double pull request throughput without compromising code quality, security, or performance. Rather than simply rolling out more coding agents, Fin established a dedicated team, Team 2x, to manage the infrastructure around them, including shared skills, plugins, MCP servers, and evaluations. The team tested agent skills against real examples, backtested its code-review skill against historical PRs, and analyzed agent sessions to identify recurring mistakes and missing context. But increasing output created new problems. CI came under pressure, and code quality became inconsistent. Fin responded by investing in linting and standardization. Over 16 months, Fin reported: - 3× PR throughput - 54% reduction in its defect backlog - 50% lower cost per PR, despite higher total AI spending The especially interesting part is how much work Fin put into making the rest of its engineering system keep up. 2. Your codebase needs to be ready for agents Even capable agents struggle when they can’t reliably build a project, run tests, or find the context they need. This is particularly challenging in large, complex repositories, where developers often rely on years of accumulated knowledge about build configurations, dependencies, and project conventions. At JetBrains, we’ve been tackling this problem in our own large monorepo. Our experience points to several practical requirements: standardized build tools, fast tests, accessible APIs, and environments that agents can set up themselves. The lesson is straightforward: Before investing in more agents, look at what prevents the ones you already have from completing tasks reliably. 3. Individual AI adoption doesn’t automatically improve team performance In one organization Angie Jones discussed, AI adoption reached 95%, yet developer velocity remained unchanged. The team shifted its focus from encouraging AI use to helping developers delegate meaningful work. Champions who built richer repository-level context were eventually opening pull requests at three to four times their previous pace. Skiller Whale encountered a different version of the same problem. In one customer engagement, a group trained to supervise AI agents roughly doubled its PR throughput, while throughput declined across the rest of the organization. These examples highlight something easy to overlook in AI adoption initiatives: Developers need to learn how to structure tasks, provide context, evaluate results, and share effective practices with colleagues. Otherwise, organizations risk ending up with a handful of highly productive AI power users rather than more productive engineering teams. What does it take to scale AI agents? Across these conversations, one pattern kept emerging: The organizations seeing meaningful results were consistently changing the systems and practices around them. That meant making development environments agent-ready, establishing shared standards, improving review processes, investing in team learning, and measuring outcomes beyond adoption. There’s no single playbook that works for every engineering organization, but there are concrete approaches worth learning from, including the ones that didn’t work the first time. Talking to these teams gave me plenty to think about, and I’m sure there are many more stories worth sharing. I’d love to hear about your experiences with AI agents, the lessons you’ve learned, or the challenges you’re still figuring out, in the comments section. Subscribe to JetBrains Blog updates