Roblox’s approach to tripling AI code review acceptance rates

Roblox's AI-generated code review comments now achieve an acceptance rate of nearly 70%, up from 50–55% for human-written comments, after the company mined historical review feedback into organization-specific rules called exemplars, VP of Engineering Andrew Swerdlow said on the Engineering Enablement podcast. About 5% of Roblox pull requests auto-merge after passing AI code review, programmatic safety checks and risk assessments, with humans approving the rules rather than individual changes. Roblox also reduced the cost of individual code review rule evaluation through session tracing and caching, and tracks accepted, rolled-back and unattended code rather than raw AI-generated code volume.

Listen and watch now on YouTube https://youtu.be/j3YuAfNCZGk , Apple https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476 , and Spotify https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D . In this episode of Engineering Enablement , I talk with Andrew Swerdlow, VP of Engineering at Roblox, about how the company is building trust in AI agents to review, test, and deploy code autonomously. Andrew explains how Roblox turns past code review feedback into rules for agents and why engineering is shifting toward specifications and intent. We explore what that means for the skills engineers need, how Roblox measures AI’s impact on productivity, and how it manages costs. We also discuss the guardrails needed to let agents safely ship code with minimal human intervention. Some takeaways: Building trust in autonomous development - Generating code is no longer the biggest bottleneck. Code review, testing, and deployment still slow delivery. Realizing AI’s full potential requires automating more of the software development lifecycle. - Autonomy has to be earned through trust. Strong testing infrastructure, production canaries, and automated rollback capabilities are essential for allowing agents to operate independently. Organization-specific rules lifted AI review acceptance to nearly 70% - Generic AI models lack organization-specific knowledge. Companies need ways to capture their internal engineering practices and make that information available to agents. - Code review comments are a valuable source of institutional knowledge. Roblox mines historical feedback to create rules called exemplars that guide AI-generated code and code reviews. - Organization-specific guidance makes AI code review more effective. Roblox’s AI-generated review comments now have an acceptance rate of nearly 70%, compared with 50–55% for human-written comments. Humans approve the rules, and about 5% of PRs auto-merge - Human judgment is shifting from reviewing individual changes to defining the rules. Engineers can establish and approve guardrails that allow AI to review code without requiring constant human intervention. - Roblox is already allowing some code to reach production autonomously. About 5% of PRs are automatically merged after passing AI code review, programmatic safety checks, and risk assessments. Intent, specs, and judgment are becoming the core engineering skills - Engineering work is shifting from code toward intent, architecture, and specifications. As agents handle more implementation details, engineers will increasingly collaborate on higher-level designs. - Experience and judgment remain essential. Senior engineers are often better equipped to challenge AI-generated solutions and anticipate problems with scalability, maintainability, and system design. - Engineering interviews may need to evaluate process rather than just results. How efficiently candidates use AI, develop specifications, and arrive at solutions could become more important than traditional coding exercises. Roblox tracks accepted, rolled-back, and unattended code - The amount of AI-generated code is becoming a less meaningful metric. What matters is whether that code survives in production, delivers value, and requires less human intervention. - Roblox is experimenting with new measures of productivity. Agent autonomy, feature velocity, and developer sentiment help reveal whether AI tools are genuinely improving engineering outcomes. Session tracing and caching are the main levers on AI cost - Observability is essential for understanding AI costs and waste. Roblox tracks agent sessions, token usage, and code outcomes to identify inefficient workflows and opportunities for improvement. - The right infrastructure can dramatically reduce costs. Roblox reduced the cost of individual code review rule evaluations by 90% through caching and shared context. - Flexibility matters in a rapidly changing AI landscape. Rather than building every capability internally or committing to one vendor, Roblox supports multiple models and tools so engineers can adopt the most effective solutions. In this episode, we cover: 00:00 https://www.youtube.com/watch?v=j3YuAfNCZGk Intro 02:20 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=140s Building trust in AI across the software development lifecycle 04:44 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=284s When human oversight helps and when it gets in the way 06:50 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=410s What AI agents need to know about your organization 08:45 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=525s How Roblox turns engineering knowledge into rules for AI agents 12:40 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=760s How AI code review compares to human code review 14:59 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=899s Capturing engineering knowledge beyond documentation 18:14 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1094s How AI changes what engineers need to understand 20:17 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1217s The shift to spec-driven development 23:24 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1404s How AI changes how we evaluate engineers 26:33 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1593s Rethinking engineering interviews in the age of AI 28:23 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1703s How Roblox tracks AI-generated code 35:56 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2156s Model routing and choosing the right AI tools 39:50 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2390s Measuring AI’s impact on engineering productivity 44:44 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2684s Optimizing AI costs at scale 48:12 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2892s Builders at Roblox and the future of software engineering 50:19 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=3019s Building trust in autonomous code review 53:11 https://www.youtube.com/watch?v=j3YuAfNCZGk&t=3191s The cultural shift toward autonomous engineering Where to find Andrew Swerdlow: • LinkedIn: https://www.linkedin.com/in/andrewswerdlow/ https://www.linkedin.com/in/andrewswerdlow/ • X: https://x.com/andrewswerdlow https://x.com/andrewswerdlow • Website: https://andrewswerdlow.net https://andrewswerdlow.net Where to find Brian Houck: • LinkedIn: https://www.linkedin.com/in/brianhouck https://www.linkedin.com/in/brianhouck Referenced: • CAFE S : Your Agent Is Only as Good as Its Context https://getdx.com/whitepaper/the-cafes-framework/ • Roblox https://www.roblox.com/ • Tech Leadership by Andrew Swerdlow https://www.amazon.com/dp/B0CK45BHLX • Malte Ubl - Vercel | LinkedIn https://www.linkedin.com/in/malteubl/ • Diff Risk Score: AI-driven risk-aware software development - Engineering at Meta https://engineering.fb.com/2025/08/06/developer-tools/diff-risk-score-drs-ai-risk-aware-software-development-meta/