Anthropic's Sonnet 5 takes center stage in a new benchmark test against leading models. The results challenge expectations and redefine model evaluations.
Anthropic's latest release, Sonnet 5, marks another milestone in the evolving landscape of frontier AI models. But rather than relying on subjective assessments or fleeting "vibe checks," a more rigorous approach has been taken to understand its true performance.
Building the Benchmark #
The How I AI Bench was born out of necessity. Tired of one-off tests devoid of repeatability, I constructed an evaluation harness with Claude Code in under 45 minutes. This system juxtaposes Sonnet 5 with other leading models like Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro. By focusing on PRD quality, prototype generation, agentic task completion, and agent personality, this setup aims to provide a comprehensive measure of each model's capabilities.
The Evaluation Methodology #
In an industry often swayed by human intuition alone, this experiment blended 70% human vibe scoring with 30% LLM judge scoring. This dual approach offers a balanced view, recognizing the subjective element of human judgment while anchoring it with machine-based evaluation.
Surprising Outcomes #
The results were unexpected. Sonnet 5, anticipated to outperform all its predecessors, showed strengths in PRD quality but lagged in areas like agent personality when compared to Gemini 3 Pro. If models are to work autonomously, understanding their distinctive strengths becomes vital.
Setting Up for Future Success #
For those interested in conducting their own assessments, setting up a local HTML scoring page allows for the intuitive evaluation of AI outputs. This setup facilitates the export of scores as JSON, enabling more detailed data analysis over time.
Implications for the Future #
This isn’t a mere announcement of a new product. It’s a convergence of technology and methodology, pushing boundaries of how models are evaluated. But the critical question remains: If agents have wallets, who holds the keys to their autonomy?
In these comparisons, we're not just looking at performance metrics. We're probing the very foundation of agentic AI and its potential to redefine industries. As AI models gain more autonomy, the financial plumbing for these systems becomes increasingly important.
Get AI news in your inbox
Daily digest of what matters in AI.
Key Terms Explained #
Agentic AI Agentic AI refers to AI systems that can autonomously plan, execute multi-step tasks, use tools, and make decisions with minimal human oversight.
Anthropic An AI safety company founded in 2021 by former OpenAI researchers, including Dario and Daniela Amodei.
Benchmark A standardized test used to measure and compare AI model performance.
Claude Anthropic's family of AI assistants, including Claude Haiku, Sonnet, and Opus.