← All posts Semantic layers (metric storage combined with a SQL generation engine, plus a DSL to express queries against that storage) are widely promoted as a way to make AI agents’ data queries more robust and more correct. At the same time, experiments on toy datasets often show little or no difference in text-to-SQL accuracy between an agent generating raw SQL and an agent querying through a semantic layer.
We saw this ourselves in our BIRD-INTERACT run: the agent harness was the biggest lever, and SLayer added a real but modest edge on top. Our hypothesis was that this apparent discrepancy comes from the size of the toy datasets: everything worth knowing about them fits comfortably into a single context window, so the semantic layer has little to add. On a production-scale database, where it doesn’t fit, the layer should start to matter.
So we tested that.
TL;DR #
- We ran two versions of the same agent, with the same harness and near-identical prompts, against 265 randomly selected read-only tasks fromLiveSQLBench-Large, a benchmark built to emulate production-scale databases. - One version used the benchmark’s native tools (raw schema access, knowledge base lookups, direct SQL). The other used the tools exposed by SLayer, our open-source semantic layer, which had deterministically ingested the database schema and knowledge base. - The raw-SQL agent passed 114 of 265 tasks (43.0%). The SLayer agent passed** 153 (57.7%): a relative improvement of over 34%**. - A second SLayer run against the pre-built definitions kept the same pass rate at roughly the same token cost as the raw agent.
The benchmark: “large” is not an exaggeration #
LiveSQLBench-Large is part of the BIRD family, built specifically to emulate production environments with many tables and many columns per table.
Each of its 18 PostgreSQL databases has on average 54 tables and 986 columns (ranging from 691 to 1,249 columns per database), with about 2 million rows in total. Each database ships with roughly 60 “knowledge base” items, text snippets describing metrics and business rules, 1,090 in total. Some columns are fixed-schema JSON, and the benchmark provides a text description of each leaf.
For comparison, the earlier LiveSQLBench-Base-Full averages 11 tables and 93 columns per database, so this is roughly a 10x jump in schema complexity. According to the benchmark authors, a task prompt that includes the full schema and knowledge base averages about 84K tokens, versus 14K for the base version. The PostgreSQL dumps range from 4 MB to 340 MB, about 1.1 GB in total. Of the benchmark’s 480 tasks, 332 are read-only (the rest are CRUD-style write tasks). We randomly selected 265 of those for cost reasons.
The setup #
The benchmark ships a default agent (LiveSQLBench-Agent, built on the Google ADK) with a basic agentic loop and eight tools: run arbitrary SQL, fetch the full schema as CREATE TABLE statements, look up column meanings, list and fetch knowledge base items, and submit the final SQL. In the stock setup, submitting ends the task, so the agent gets exactly one attempt.
We changed two things to bring the evaluation closer to real life:
The harness. In our earlier work on the closely related mini-interact benchmark, merely switching the agent harness from PydanticAI to the Claude SDK massively improved the success rate, both with and without SLayer. So all evaluations here use the Claude SDK harness. This is also how SLayer is meant to be used in practice: from a mature agent harness such as Claude.Multiple tries. We allowed the agent several attempts at the answer, so the harness can learn from previous tries, as it does in real-world use.
Both agents ran with an identical harness. The only difference between the prompts was the description of how to use the respective tools.
Building the semantic layer config #
The hard part of a fair comparison was giving the SLayer agent a valid, rich semantic layer config. In the real world, humans own that task.
We solved it in two stages. First, we deterministically ingested everything that could be auto-ingested: SLayer’s schema auto-ingestion produced the first draft of the config, which we then enriched with the knowledge base items represented as SLayer memories, and with the leaves of the fixed-schema JSON columns encoded as individual SLayer columns.
We initially tried running a separate agent to convert the knowledge base text into SLayer fields and metrics, but that gave no performance increase, most likely because that agent had no way to verify the validity of the definitions it created.
The only automatic validity check available is whether an answer built on those definitions is accepted by the benchmark’s scorer. So the SLayer evaluation ran in two stages:
Build-up run. The agent started from the basic semantic layer setup and was allowed to add and modify dimension and measure definitions while solving each task. Where it succeeded, we took that as a signal that its modifications were correct.Preconfigured run. The semantic layer enriched by the first run became the starting point for a second, read-only run. Since this run was only possible for tasks the first run solved, it could not pass more tasks; its value was in comparing turns and tokens against the raw agent when the definitions already exist.
Results #
| Condition | Tasks passed (/265) | Pass rate | Relative vs. raw | Token usage |
|---|---|---|---|---|
| SQL + raw context | 114 | 43.0% | baseline | baseline |
| SLayer: build-up run | 153 | 57.7% | +34.2% | higher than raw |
| SLayer: preconfigured run | 153 | 57.7% | +34.2% | ≈ same as raw |
The main result confirms the hypothesis: the SLayer agent’s pass rate was over 34% higher than the same agent writing SQL directly. And that held even though the build-up agent had to both encode the intermediate metric definitions in the semantic layer and then call them correctly to get the final answer. Its token usage was accordingly higher, from the extra work of converting knowledge base text into model definitions.
The secondary result: rerunning against the enriched semantic models kept the same high pass rate at roughly the same token consumption as the raw agent. We actually expected the preconfigured run to come in below raw, and we are currently investigating why it didn’t. Watch this space.
What this means #
Semantic layers show their value specifically where production databases live: schemas too large to hold in a context window, with business definitions scattered across documentation. On toy datasets the agent can brute-force its way to correct SQL. At 986 columns per database, it can’t.
And this run likely understates the effect. The definitions here were bootstrapped from text snippets by an agent. Starting from a human-curated set of formal definitions, the way SLayer is used in production, the gap should be wider still.
SLayer is open source (MIT): github.com/MotleyAI/slayer. If you want to reproduce this run or argue with the methodology, Discord is open.