# Beating a text-to-SQL benchmark: can you get better than plain Claude? > Source: > Published: 2026-07-07 16:44:58+00:00 [← All posts](/blog) # Beating the BIRD-INTERACT benchmark: can you get much better than plain Claude by adding a semantic layer? The best way to make a product good is to use it in a wide variety of cases. At Motley, we had already learned a lot from our customers and design partners while building [SLayer](https://github.com/MotleyAI/slayer), the open-source semantic layer for agents. The next natural place to look for that variety was text-to-SQL benchmarks. ## TL;DR - With the Claude SDK and SLayer (the open-source semantic layer we developed), an agent reached a **75.3% pass rate** on BIRD-INTERACT (a-interact, mini-interact), against the official best of**36.33%** on the comparable Lite leaderboard. - The biggest lever was not the semantic layer, but the **agent harness**. The Claude SDK alone took a raw-SQL agent to 71.7%, while the PydanticAI-based setup yielded a much more modest improvement. - A substantial share of the benchmark’s gold answers are wrong, so we built an annotation agent and report results against both the original and the annotated answers. - SLayer still added an edge on top: 75.3% vs 71.7% against the original gold answers, and 83.7% vs 79.0% against the corrected (“annotated”) ones. - The SLayer ingestion ran fully autonomously for this run, so the numbers would probably be higher in a human-curated setup. If you’re unfamiliar with SLayer, here’s what it does: sitting between the agent and the database, it holds the definitions the agent queries against: semantic models of your tables and fields (hand-written or auto-ingested from the schema), metric definitions like “net revenue”, and free-text knowledge attached as memories. Instead of writing raw SQL, the agent queries through structured JSON that SLayer compiles into SQL. A text-to-SQL benchmark exercises a large part of that job, answering business questions over an unfamiliar database with a knowledge base attached, so it was a natural way to stress-test SLayer. So we ran one. With the Claude SDK and SLayer, **an agent reached a 75.3% pass rate on BIRD-INTERACT (a-interact, mini-interact), compared to an official best of 36.33% on the comparable Lite leaderboard** 1. Boom. Headline number. The reality is more subtle, and the process to get there more interesting. This post is the first in a series: here we cover how we got there and what the run taught us. Later posts go deeper on the individual lessons. ## The benchmark maze Text-to-SQL benchmarks are a fascinating, messy domain. Even if you just look at the BIRD family, it amounts to almost 20 different sub-benchmarks. First there is the original BIRD, with its full and dev subsets. Then there is LiveSQLBench, with Base-Full, Base-Lite, Large-Full, and Large-Lite variants (and the occasional SQLite variant thrown in). Then there is BIRD-INTERACT with Full, Lite, and mini-interact variants, and each of those comes in c-interact and a-interact flavors, with separate leaderboards depending on whether the agent was budget-constrained. And then there is BIRD-CRITIC, with its own sub-benchmarks. Laid out, the family looks like this: | Benchmark | Variants | Notable flavors | |---|---|---| | BIRD (original) | full, dev | single-shot text-to-SQL | | LiveSQLBench | Base-Full, Base-Lite, Large-Full, Large-Lite | additional SQLite variant | | BIRD-INTERACT | Full, Lite, mini-interact | c-interact / a-interact, budget-constrained vs not | | BIRD-CRITIC | own sub-benchmarks | SQL debugging / critique | That is close to 20 distinct leaderboards before you have written a line of code. We wanted to run SLayer in conditions close to how it would actually be used, which meant the Claude SDK for the agent and Opus as the LLM. That made a single full benchmark run cost on the order of $1000 for the smaller benchmarks, more for the larger ones. Running, and re-running to correct the issues that inevitably arose, across all the flavors was not an option. We had to settle on one to start with. ## Why a-interact, mini-interact We chose **BIRD-INTERACT, a-interact flavor**. What sets BIRD-INTERACT apart is that it tries to simulate a real working environment rather than a one-shot question. Each database is coupled with a hierarchical knowledge base, metadata files, and a function-driven `user_sim` simulator, so the agent can solicit clarifications, retrieve knowledge, and recover from execution errors on its own, with no human in the loop. The **a-interact** flavor takes this furthest: it is an open-ended agentic setting where the model decides for itself when to question the user simulator, when to explore the database, and how many turns to take before submitting a final answer. That is the closest of all the flavors to how agents actually operate in the real world, which is why we picked it. We chose **mini-interact** because it is based on SQLite rather than Postgres, so it was less hassle to run, and it is smaller than the full version. Put against the alternatives, here is why those two choices won: | Criterion | c-interact | a-interact | Full | Lite | mini-interact | |---|---|---|---|---|---| | Agent sets its own turn budget | no | yes | – | – | – | Provides a `user_sim` to question | yes | yes | yes | yes | yes | | Database engine | – | – | Postgres | Postgres | SQLite | | Number of query tasks | – | – | 600 | 195 | 300 | | Closest to real agent use | no | yes | – | – | – | There are no official leaderboard results for mini-interact yet. There are for the Lite version, which uses similar tasks but on Postgres databases rather than SQLite. The best result on a-interact there is **36.33%**, using Opus-4.6 with a simple agent wrapper provided by the benchmark authors. That is the number we compare against, with the SQLite-versus-Postgres caveat noted up front. ## The broken gold answers The next challenge is that a substantial share of the gold answers the benchmark provides are simply wrong. This has been pointed out several times, for example [here](https://github.com/niklaswretblad/the-effects-of-noise-in-text-to-SQL) and [here](https://github.com/uiuc-kang-lab/text_to_sql_benchmarks). This is a real problem. To improve both SLayer and the agent harness, we need to distinguish cases where the agent actually went wrong from false negatives, where the agent’s answer was correct but evaluated against a wrong reference answer. ``` -- Task: "Show the accuracy ratio and confidence levels for each registration: -- registration ID, project ID, accuracy, error values, the calculated ratio (RAR), -- and the confidence level it translates to." -- Benchmark gold answer (the confidence-level branch): CASE WHEN rar > 1.5 AND sr.refmark LIKE '%Target%' THEN 'High Confidence' WHEN rar BETWEEN 1.0 AND 1.5 THEN 'Medium Confidence' ELSE 'Low Confidence' END AS confidence_level -- Why it is wrong: per KB #44, 'High Confidence' requires LogMethod to contain 'Target'. -- The gold matches against refmark instead, which only ever holds numeric codes -- ('40', '31', '25'), so the High-Confidence branch never fires and returns 0 rows. -- Matching on logmethod ('Target-based', 'Hybrid', 'Feature-based') gives the intended -- 50 High-Confidence rows. ``` So the first thing we built was an **annotation agent**. It went through each task and decided three things, given all the data the task made available to the agent: - whether the gold answer was the single correct answer, - whether it was compatible with the task at all, - and whether there were other answers that would be equally valid given the task statement. These annotations are themselves probabilistic, so we report success rates two ways: implied by the original gold answers, and implied by the annotations. The annotated numbers are not gospel truth either, but they give a much better evaluation signal than relying on the original answers alone. ## The experimental design To separate the effect of the agent harness from the effect of using SLayer, we built two versions of every agent variant, as similar to each other as possible. One used SLayer to introspect the database and the associated knowledge base and to submit the answer. The other used raw SQL and the tools that shipped with the benchmark repo. For the SLayer version, we used the standard SLayer schema ingestion to pre-populate the models, then deterministically created a field for each documented component of a JSON column in the original schema. The documentation for those was itself structured JSON, so the ingestion was trivial. ``` // Original schema: one documented JSON column, returns.return_details // (the benchmark documents each leaf of the JSON as structured text) { "column_meaning": "JSONB column grouping return reason, authorization, and shipping metadata.", "fields_meaning": { "fraud": { "risk_level": "VARCHAR(50). Fraud risk level for the return. NULL means not assessed." } } } # SLayer field created deterministically from it (one per documented JSON path) - name: return_details__fraud__risk_level sql: JSON_EXTRACT(return_details, '$.fraud.risk_level') type: TEXT description: "Fraud risk level for the return. NULL means no fraud risk level assessed." meta: derived_from: json_col: return_details path: [fraud, risk_level] ``` Worth being clear about one thing here: for the benchmark, this entire ingestion ran fully autonomously and deterministically, with no human or LLM in the loop. That was deliberate, to keep the comparison clean and the result reproducible. In real use it is not how SLayer is meant to work. Human input at this stage to encode the knowledge base items would be massively additive. So **treat the numbers below as a floor for what an automated setup gets you**. We also ingested the knowledge base items, raw pieces of text that document relevant concepts and can refer to each other, as SLayer **memories**. The way the knowledge base items cross-referenced each other matched the memories structure perfectly. For the agent harness, we tried two setups: **PydanticAI** with a custom sub-agent structure, spawning exploration and definition sub-agents for every concept needed.**Claude SDK** with the appropriate set of tools. We used Opus-4.7 throughout for the agent, and Sonnet for the `user_sim` . ## What the benchmark taught SLayer Because the benchmark contains such a varied set of queries, running it against the SLayer version was a great workout for SLayer itself. It triggered a couple of edge-case bugs. It pointed out some types of queries the DSL could not yet represent at all, which we then added. And it gave us a collection of almost-but-not-quite-correct tool calls the agent made, which we used to enlarge our set of patterns that we auto-correct for during query parsing. We are taking a unique approach in designing tools for SLayer: we account for potential failures by the agent and self-correct them when the intent is obvious. You can learn more in our earlier post, [How forgiving are your MCP tools?](https://motley.ai/blog-posts/how-forgiving-are-your-mcp-tools/) ``` # What the agent wrote (SQL-style aggregation -- appeared 265 times across the run): count(*) sum(revenue) # What SLayer now auto-corrects it to (its native colon syntax): *:count revenue:sum ``` This is the part that matters beyond the score. A benchmark with this much variety surfaces gaps a curated test suite never would. ## The results This is where we could boast of SLayer improving on the current leading model result by more than 2X: 75.3% with SLayer against the official 36.33% top-of-benchmark result. But the more honest takeaway is that **the choice of agent harness made the bigger impact on the pass rate**. We started with custom agents built using PydanticAI (basic loops with tools, where the tools included spawning task-specific sub-agents recursively). However, using claude_sdk yielded much higher pass rates (43% for PydanticAI-based vs 64% for claude_sdk, for a 53-task set we were using for that comparison), so we switched to claude_sdk throughout. Unfortunately, a comprehensive comparison of the two across the whole task set was not done for cost reasons. The Claude SDK agents were a different story, with a significant improvement in performance overall. There, the SLayer-using agent performed marginally better than its raw-SQL twin. | Harness | Variant | Pass rate (vs original gold) | Pass rate (vs annotated) | |---|---|---|---| | Benchmark authors’ wrapper (Lite, Opus-4.6) | raw SQL | 36.33% | n/a | | Claude SDK | raw SQL | 71.7% | 79.0% | | Claude SDK | SLayer | 75.3% | 83.7% | So the Claude SDK with SLayer reached a **75.3%** pass rate against the original gold answers, and **83.7%** against the annotated ones. To its credit, BIRD-INTERACT does build ambiguity in: that is the point of the interactive setting, where the agent questions a user simulator to pin down intent. But it resolves that ambiguity once, per task, against a single canonical answer. Business questions are different. The same question comes back month after month, phrased differently and asked by different people, and the answer has to stay consistent every time. No benchmark scores that yet, and it is exactly what a semantic layer is for: it fixes what “active customer” or “net revenue” means once, so every agent resolves it the same way instead of re-deriving it per prompt. Delivering that consistency is the whole point of SLayer. ## What we learned **Generating SLayer DSL was not a bottleneck.** A common worry is that LLMs might be bad at generating structured languages that were not in their training set. As it turns out, if the syntax is natural and described in the agent’s prompt, it works just fine. The agent did invent a couple of DSL constructs that should logically have worked but were not yet supported, and we made sure they were. But its ability to produce valid DSL never limited the score. ``` // SLayer query the agent generated. The formula uses SLayer-native syntax with no SQL // equivalent: a colon aggregation (computed_net_worth:max) wrapped in a rank() window // transform. The agent wrote no window function, no OVER clause, and no GROUP BY. {"source_model": "expenses_and_assets", "dimensions": ["expemplref", "totassets", "totliabs", "computed_net_worth"], "measures": [{"formula": "rank(computed_net_worth:max)", "name": "ranking"}], "order": [{"column": "computed_net_worth", "direction": "desc"}], "limit": 10} -- compiled to (SLayer turns rank(...:max) into a windowed RANK() OVER inside a CTE pipeline) SELECT "expenses_and_assets.expemplref", "expenses_and_assets.totassets", "expenses_and_assets.totliabs", "expenses_and_assets.computed_net_worth", "expenses_and_assets.ranking" FROM ( WITH base AS ( SELECT expenses_and_assets.expemplref AS "expenses_and_assets.expemplref", expenses_and_assets.totassets AS "expenses_and_assets.totassets", expenses_and_assets.totliabs AS "expenses_and_assets.totliabs", CAST(expenses_and_assets.totassets - expenses_and_assets.totliabs AS REAL) AS "expenses_and_assets.computed_net_worth", MAX(CAST(expenses_and_assets.totassets - expenses_and_assets.totliabs AS REAL)) AS "expenses_and_assets.computed_net_worth_max" FROM expenses_and_assets AS expenses_and_assets GROUP BY expenses_and_assets.expemplref, expenses_and_assets.totassets, expenses_and_assets.totliabs, CAST(expenses_and_assets.totassets - expenses_and_assets.totliabs AS REAL) ), step1 AS ( SELECT *, RANK() OVER (ORDER BY "expenses_and_assets.computed_net_worth_max" DESC) AS "expenses_and_assets.ranking" FROM base ) SELECT * FROM step1 ) AS _outer ORDER BY "expenses_and_assets.computed_net_worth" DESC LIMIT 10 ``` **Context size made a huge difference.** One buggy run, whose only difference was that it mistakenly imported many irrelevant tool definitions, performed markedly worse. The agent never used any of the irrelevant tools, but its reasoning ability degraded noticeably, leading to worse answers. The lesson is blunt: a smaller, cleaner surface is not just cheaper on tokens, it makes the agent reason better. That is the same intuition behind giving an agent a governed query layer instead of a wide raw schema. **Autoingestion can get you far, but only that far.** There are a lot of promises around autoingestion of context for semantic layers, and to be fair it can get you far, as the benchmark shows. But our conviction is that a human-curated, agent-assisted semantic layer setup would perform even better. The BIRD context fits comfortably in an agent’s context window, which makes it a relatively favorable case for autoingestion; the richer, messier schemas of real production systems are exactly where a few minutes of human curation pay off most. ## Limitations and what is next The token cost of Opus did not let us run many iterations to optimize performance further. The open-source models that could have made cheaper iteration possible did not, until recently, have the context window to serve as drop-in replacements. That changed with GLM-5.2, so watch this space. This was the first run, on one flavor of one benchmark. The series continues with deeper write-ups on the annotation agent, the auto-correction patterns the run surfaced, and what changes when you swap in a long-context open model. SLayer is MIT-licensed and on PyPI: ``` pip install motley-slayer ``` Repo and docs: [github.com/MotleyAI/slayer](https://github.com/MotleyAI/slayer) ## Footnotes - The two numbers are closer to like-for-like than two different leaderboard names suggest. Both are measured under the a-interact setting, where the agent sets its own turn budget, so the interaction protocol is the same on both sides. And the task lineage is shared: mini-interact is the transformed data from bird-interact-lite and bird-interact-full, which are in turn built on the single-turn unambiguous text-to-SQL tasks of LiveSQLBench-Base-Lite and LiveSQLBench-Base-Full-v1. [↩](#user-content-fnref-lite-mini)