# Supabase Open-Sources Evals to Grade Claude Code, Codex and OpenCode > Source: > Published: 2026-08-01 16:04:11+00:00 *Supabase just open-sourced Evals, a free benchmark that scores AI coding agents like Claude Code, Codex and OpenCode on real backend tasks, and the early results show a wide gap between models working alone and models given the right context.* Supabase, the open-source Firebase alternative used by hundreds of thousands of developers, released Evals on August 1, 2026 under an Apache-2.0 license. It's not a marketing leaderboard dressed up as research. It's a working test suite anyone can run. Install it with pnpm, spin up containers on ports 54321 through 54329, and watch an AI agent try to build a database schema, fix a broken row-level security policy, or debug a failing Edge Function. You can see the live results yourself at supabase.com/evals, filtered by product, by task stage, or by agent. That transparency is the point. Most agent benchmarks today come from the vendors selling the agents. This one comes from the company whose product the agents are actually using, which makes it closer to a real trust layer than a press release. The results are revealing. According to Supabase's own writeup announcing the project, Opus 5 and Kimi K3 both hit 100% on the Build-stage tasks without any extra help. Sonnet 5 did not. Left alone, it scored 78%. Give it access to Supabase's "skills," structured context files that teach an agent the platform's conventions, and it jumped to 100%. GPT-5.6 Sol followed the same arc, climbing from 89% to 100% once skills were loaded. Even GPT-5.4 mini improved, from 78% to 89%. That's the real finding here, not who "wins." A generic frontier model and a model that actually knows how Supabase wants migrations written are two different products for a working developer. The gap between 78% and 100% matters. It's the difference between an agent you have to babysit and one you can trust to open a pull request unsupervised. Evals doesn't grade agents against toy problems or a static answer key. Every scenario runs against a containerized version of Supabase's actual hosted stack plus a local CLI project. The agent calls the real MCP server and the real command-line tools. Not a mock. Scoring blends deterministic checks, did the data access work, did the function return the right output, with an LLM acting as judge for anything that needs semantic reading. Agents get exactly one retry before the final grade locks in. Supabase also flagged where agents still fall short, and this is the part vendors usually leave out of their own marketing. Agents kept hand-writing SQL migrations instead of using Supabase's declarative schema tooling. They verified authentication manually instead of reaching for the @supabase/server package built for exactly that job. And documentation habits varied enormously. Codex-based agents checked Supabase's docs in nearly every scenario, reading around eight pages each time. Claude Code consulted docs in under 40% of runs, according to MarkTechPost's reporting on the release. Frankly, that documentation gap is the most interesting number in the whole release. It suggests some agents are pattern-matching from training data rather than checking whether the platform actually works the way they assume it does. That's exactly the kind of failure that looks fine in a demo and breaks in production. Supabase says the regression suite behind the public leaderboard reruns daily, so scores move as model providers ship updates rather than sitting frozen at launch. That's a meaningful design choice. Static benchmarks go stale the moment a lab ships a patch. A benchmark that regrades itself every day stays honest about which agent is actually current. None of this settles the broader argument about whether AI agents can be trusted with real backend work. What it does is give developers something better than vendor claims to argue from. It offers a public, repeatable test, tied to a tool people actually ship with, that anyone can rerun themselves. As more infrastructure companies start publishing their own versions, this kind of narrow, product-specific eval may end up being the thing that actually decides which agent a team picks. Not the sprawling general benchmarks labs love to cite. **Also read:** [GEMA Wins Landmark German Court Ruling Against AI Music Startup Suno](https://startupfortune.com/gema-wins-landmark-german-court-ruling-against-ai-music-startup-suno/) • [KSNET Brings Solana Pay and AI Agent Payments to 330,000 Korean Merchants](https://startupfortune.com/ksnet-brings-solana-pay-and-ai-agent-payments-to-330000-korean-merchants/) • [OpenAI's unreleased Astra model solved ten open math problems for $2,000](https://startupfortune.com/openais-unreleased-astra-model-solved-ten-open-math-problems-for-2000/)