Only 5% of enterprises trust AI agents in production despite 85% piloting them. Amazon's Bryan Silverthorn outlines why reliability, not benchmarks, defines success.
The enterprise AI sector faces a stark reality: 85% of enterprises are piloting AI agents, yet a mere 5% trust them in production. This gap isn't just about refining models, it's about understanding and measuring what truly matters. At VB Transform 2026, Bryan Silverthorn from Amazon's AGI lab emphasized that bridging this gap requires more than just better benchmarks.
Decoding AI Agent Reliability #
Silverthorn, who guides multimodal agent training at Amazon, suggests a framework that dissects reliability into consistency, predictability, robustness, and safety. This approach, rooted in Princeton research, untangles the complexities often obscured in evaluations. The problem? AI agents often excel in controlled environments but falter when facing real-world applications.
Consider a case where an agent flawlessly handled software QA for two months, only to collapse due to a minor software change. The vision encoder misinterpreted serial numbers based on their screen location, a small shift, yet a critical failure. The takeaway isn't just about better models. it's about rigorous measurement aligned with application stakes.
Amazon's 'Intern' Philosophy #
Silverthorn's perspective isn't just technical, it's cultural. Amazon mocks its agents as 'interns' to illustrate their potential and fallibility. Like interns, these AI agents can perform great feats or derail spectacularly. Managing these 'interns' isn't just about software prowess but involves strategic risk assessment and mitigation.
In Amazon's AGI lab, agents occasionally make mistakes, but this trade-off is accepted for research velocity. Imagine an agent running experiments 24/7, autonomously executing high-level research plans. This philosophy underscores the need for adaptive management rather than rigid control.
Beyond Pilot Purgatory #
For enterprises trapped in pilot limbo, the solution isn't smarter agents, it's smarter management. AI's self-improvement remains a distant dream, and reliance solely on computer use is obsolete. Future agents will integrate with APIs, MCP, and more. The goal isn't to ask if an agent can succeed once, but if it can do so consistently, a thousand times over. Enterprises breaking free from the 85% ceiling won't boast the most intelligent agents, but the most effective managers. If the AI can hold a wallet, who writes the risk model? The intersection is real. Ninety percent of the projects aren't. But the ones that succeed? They'll define the industry.
Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
AGI Artificial General Intelligence.
AI Agent An autonomous AI system that can perceive its environment, make decisions, and take actions to achieve goals.
Encoder The part of a neural network that processes input data into an internal representation.
MCP Model Context Protocol (MCP) is an open standard created by Anthropic that lets AI models connect to external tools, data sources, and APIs through a unified interface.