{"slug": "robloxs-approach-to-tripling-ai-code-review-acceptance-rates", "title": "Roblox’s approach to tripling AI code review acceptance rates", "summary": "Roblox's AI-generated code review comments now achieve an acceptance rate of nearly 70%, up from 50–55% for human-written comments, after the company mined historical review feedback into organization-specific rules called exemplars, VP of Engineering Andrew Swerdlow said on the Engineering Enablement podcast. About 5% of Roblox pull requests auto-merge after passing AI code review, programmatic safety checks and risk assessments, with humans approving the rules rather than individual changes. Roblox also reduced the cost of individual code review rule evaluation through session tracing and caching, and tracks accepted, rolled-back and unattended code rather than raw AI-generated code volume.", "body_md": "Listen and watch now on **[YouTube](https://youtu.be/j3YuAfNCZGk), [Apple](https://podcasts.apple.com/us/podcast/engineering-enablement-by-abi-noda/id1619140476), and [Spotify](https://open.spotify.com/show/3NxjyIsuxeDMQtisDqBy7D)**.\n\nIn this episode of *Engineering Enablement*, I talk with Andrew Swerdlow, VP of Engineering at Roblox, about how the company is building trust in AI agents to review, test, and deploy code autonomously. Andrew explains how Roblox turns past code review feedback into rules for agents and why engineering is shifting toward specifications and intent. We explore what that means for the skills engineers need, how Roblox measures AI’s impact on productivity, and how it manages costs. We also discuss the guardrails needed to let agents safely ship code with minimal human intervention.\n\n## **Some takeaways:** \n\n### **Building trust in autonomous development**\n\n- **Generating code is no longer the biggest bottleneck.** Code review, testing, and deployment still slow delivery. Realizing AI’s full potential requires automating more of the software development lifecycle.\n- **Autonomy has to be earned through trust.** Strong testing infrastructure, production canaries, and automated rollback capabilities are essential for allowing agents to operate independently.\n\n### **Organization-specific rules lifted AI review acceptance to nearly 70%**\n\n- **Generic AI models lack organization-specific knowledge.** Companies need ways to capture their internal engineering practices and make that information available to agents.\n- **Code review comments are a valuable source of institutional knowledge.** Roblox mines historical feedback to create rules called exemplars that guide AI-generated code and code reviews.\n- **Organization-specific guidance makes AI code review more effective.** Roblox’s AI-generated review comments now have an acceptance rate of nearly 70%, compared with 50–55% for human-written comments.\n\n### **Humans approve the rules, and about 5% of PRs auto-merge**\n\n- **Human judgment is shifting from reviewing individual changes to defining the rules.** Engineers can establish and approve guardrails that allow AI to review code without requiring constant human intervention.\n- **Roblox is already allowing some code to reach production autonomously.** About 5% of PRs are automatically merged after passing AI code review, programmatic safety checks, and risk assessments.\n\n### **Intent, specs, and judgment are becoming the core engineering skills**\n\n- **Engineering work is shifting from code toward intent, architecture, and specifications.** As agents handle more implementation details, engineers will increasingly collaborate on higher-level designs.\n- **Experience and judgment remain essential.** Senior engineers are often better equipped to challenge AI-generated solutions and anticipate problems with scalability, maintainability, and system design.\n- **Engineering interviews may need to evaluate process rather than just results.** How efficiently candidates use AI, develop specifications, and arrive at solutions could become more important than traditional coding exercises.\n\n### **Roblox tracks accepted, rolled-back, and unattended code**\n\n- **The amount of AI-generated code is becoming a less meaningful metric.** What matters is whether that code survives in production, delivers value, and requires less human intervention.\n- **Roblox is experimenting with new measures of productivity.** Agent autonomy, feature velocity, and developer sentiment help reveal whether AI tools are genuinely improving engineering outcomes.\n\n### **Session tracing and caching are the main levers on AI cost**\n\n- **Observability is essential for understanding AI costs and waste.** Roblox tracks agent sessions, token usage, and code outcomes to identify inefficient workflows and opportunities for improvement.\n- **The right infrastructure can dramatically reduce costs.** Roblox reduced the cost of individual code review rule evaluations by 90% through caching and shared context.\n- **Flexibility matters in a rapidly changing AI landscape.** Rather than building every capability internally or committing to one vendor, Roblox supports multiple models and tools so engineers can adopt the most effective solutions.\n\n## **In this episode, we cover:**\n\n([00:00](https://www.youtube.com/watch?v=j3YuAfNCZGk)) Intro\n\n([02:20](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=140s)) Building trust in AI across the software development lifecycle\n\n([04:44](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=284s)) When human oversight helps and when it gets in the way\n\n([06:50](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=410s)) What AI agents need to know about your organization\n\n([08:45](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=525s)) How Roblox turns engineering knowledge into rules for AI agents\n\n([12:40](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=760s)) How AI code review compares to human code review\n\n([14:59](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=899s)) Capturing engineering knowledge beyond documentation\n\n([18:14](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1094s)) How AI changes what engineers need to understand\n\n([20:17](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1217s)) The shift to spec-driven development\n\n([23:24](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1404s)) How AI changes how we evaluate engineers\n\n([26:33](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1593s)) Rethinking engineering interviews in the age of AI\n\n([28:23](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=1703s)) How Roblox tracks AI-generated code\n\n([35:56](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2156s)) Model routing and choosing the right AI tools\n\n([39:50](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2390s)) Measuring AI’s impact on engineering productivity\n\n([44:44](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2684s)) Optimizing AI costs at scale\n\n([48:12](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=2892s)) Builders at Roblox and the future of software engineering\n\n([50:19](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=3019s)) Building trust in autonomous code review\n\n([53:11](https://www.youtube.com/watch?v=j3YuAfNCZGk&t=3191s)) The cultural shift toward autonomous engineering\n\n**Where to find Andrew Swerdlow:**\n\n• LinkedIn: [https://www.linkedin.com/in/andrewswerdlow/](https://www.linkedin.com/in/andrewswerdlow/)\n\n• X:  [https://x.com/andrewswerdlow](https://x.com/andrewswerdlow)\n\n• Website: [https://andrewswerdlow.net](https://andrewswerdlow.net)\n\n**Where to find Brian Houck:**\n\n• LinkedIn: [https://www.linkedin.com/in/brianhouck](https://www.linkedin.com/in/brianhouck)\n\n## **Referenced:**\n\n• [CAFE(S): Your Agent Is Only as Good as Its Context](https://getdx.com/whitepaper/the-cafes-framework/)\n\n• [Roblox](https://www.roblox.com/)\n\n• [Tech Leadership by Andrew Swerdlow](https://www.amazon.com/dp/B0CK45BHLX)\n\n• [Malte Ubl - Vercel | LinkedIn](https://www.linkedin.com/in/malteubl/)\n\n• [Diff Risk Score: AI-driven risk-aware software development - Engineering at Meta](https://engineering.fb.com/2025/08/06/developer-tools/diff-risk-score-drs-ai-risk-aware-software-development-meta/)", "url": "https://wpnews.pro/news/robloxs-approach-to-tripling-ai-code-review-acceptance-rates", "canonical_source": "https://newsletter.getdx.com/p/robloxs-approach-to-tripling-ai-code", "published_at": "2026-10-30 14:33:04+00:00", "updated_at": "2026-10-30 14:48:05.980758+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "developer-tools", "mlops"], "entities": ["Roblox", "Andrew Swerdlow", "Engineering Enablement"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/robloxs-approach-to-tripling-ai-code-review-acceptance-rates", "markdown": "https://wpnews.pro/news/robloxs-approach-to-tripling-ai-code-review-acceptance-rates.md", "text": "https://wpnews.pro/news/robloxs-approach-to-tripling-ai-code-review-acceptance-rates.txt", "jsonld": "https://wpnews.pro/news/robloxs-approach-to-tripling-ai-code-review-acceptance-rates.jsonld"}}