{"slug": "supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex", "title": "Supabase Releases Evals: an Open Source Benchmark That Scores Claude Code, Codex and OpenCode on Real Supabase Tasks", "summary": "Supabase has open sourced Supabase Evals, a benchmark and framework for testing AI coding agents on real Supabase tasks, scoring agents such as Claude Code, Codex, and OpenCode on building schemas, debugging Edge Functions, and fixing RLS policies. The benchmark, available at supabase.com/evals, runs scenarios against real containerized Supabase stacks and uses deterministic checks plus LLM-as-a-judge, with findings showing Opus 5 and Kimi K3 scored 100% unaided in the Build stage, while skills closed gaps for other models.", "body_md": "Supabase has [open sourced Supabase Evals](https://supabase.com/blog/introducing-supabase-evals), its benchmark and framework for testing how well AI agents build using Supabase. It runs coding agents including [Claude Code](https://www.anthropic.com/claude-code), Codex, and [OpenCode](https://opencode.ai/) against real tasks, such as building a schema, debugging a failed [Edge Function](https://supabase.com/edge-functions), or fixing a broken RLS policy, then scores the result. It powers the public leaderboard at [supabase.com/evals](https://supabase.com/evals) and an internal regression suite monitored daily.\n\n**Is it deployable?**\n\nYes, today. [ supabase/evals](https://github.com/supabase/evals) is public under Apache-2.0 and runs locally via\n\n`pnpm`\n\n.**Industries**: Developer tooling, cloud infrastructure, data platforms, and regulated backends in fintech or healthcare, where an agent writing a wrong[RLS](https://supabase.com/docs/guides/database/postgres/row-level-security)policy is a security incident.**Applications**: Regression-testing docs and skill edits, gating SDK releases, and comparing agent harnesses head to head.** Constraints**: Local-stack runs need a Docker daemon, provider API keys, and ports 54321–54329 free.\n\n**How the harness works**\n\nSupabase defined three dimensions: **products** (database, auth, storage, edge-functions, realtime, cron, queues, vectors, data-api), **topics** (RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema), and **stages** (build, deploy, investigate, resolve). It then picked the smallest scenario set touching each dimension once, grounded in support tickets, bug reports, and GitHub issues.\n\nScenarios split into two suites. **Benchmark** scenarios cover breadth and are published. **Regression** scenarios cover known failure modes, refresh daily, and do not move published scores.\n\nEvery scenario runs against a real environment. The framework boots a hosted-like stack and a local CLI project in containers, so agents call the actual [MCP server](https://supabase.com/docs/guides/getting-started/mcp) and CLI. A `platform-lite`\n\nruntime exposes a Management API-compatible surface backed by [ @supabase/lite](https://github.com/supabase/supabase-lite). Scoring combines deterministic checks with LLM-as-a-judge. Agents get one retry before grading.\n\nEach eval directory holds `PROMPT.md`\n\n(task plus frontmatter), `EVAL.ts`\n\n(the scorer), and optional `remote/`\n\nand `local/`\n\nstarting states. Shipping a `local/`\n\nworkspace, or declaring `interface: cli`\n\n, boots a Docker sandbox with the real CLI installed.\n\n**Findings**\n\nAgents pass most scenarios with no skill loaded. In the Build stage, Opus 5 and Kimi K3 both scored 100% unaided. Skills closed the rest of the gap: Sonnet 5 rose from 78% to 100%, GPT-5.6 Sol from 89% to 100%, and GPT-5.4 mini from 78% to 89%.\n\nThree weaknesses surfaced. Agents hand-write migrations instead of using [declarative schemas](https://supabase.com/docs/guides/local-development/declarative-database-schemas), prompting a [skill guidance update](https://github.com/supabase/agent-skills/pull/120). Agents verify auth by hand rather than reaching for [ @supabase/server](https://supabase.com/blog/introducing-supabase-server), prompting a\n\n[package selection guide](https://supabase.com/docs/guides/auth/choosing-a-server-package). And docs usage varies sharply: Codex / GPT-5.6 reads roughly 8 docs pages per scenario versus about 2 for Claude Code, which checks docs in under 40% of scenarios even with skills loaded.\n\n**Key Takeaways**\n\n- Supabase open sourced\n`supabase/evals`\n\nunder Apache-2.0. - Scenarios run against real containerized Supabase stacks, not mocks.\n- Scoring mixes deterministic checks with LLM-as-a-judge; one retry allowed.\n- Skills mattered least for top models, most for smaller ones.\n\nCheck out the** Technical details **and\n\n**Also, feel free to follow us on**\n\n[GitHub Repo](https://github.com/supabase/evals).**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nMichal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.", "url": "https://wpnews.pro/news/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex", "canonical_source": "https://www.marktechpost.com/2026/08/01/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex-and-opencode-on-real-supabase-tasks/", "published_at": "2026-08-01 09:52:49+00:00", "updated_at": "2026-08-01 10:49:39.706075+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research", "developer-tools"], "entities": ["Supabase", "Claude Code", "Codex", "OpenCode", "Opus 5", "Kimi K3", "Sonnet 5", "GPT-5.6"], "alternates": {"html": "https://wpnews.pro/news/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex", "markdown": "https://wpnews.pro/news/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex.md", "text": "https://wpnews.pro/news/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex.txt", "jsonld": "https://wpnews.pro/news/supabase-releases-evals-an-open-source-benchmark-that-scores-claude-code-codex.jsonld"}}