# The Semantic Layer Benchmark Problem: Why Nobody Has Run the Test That Actually Matters > Source: > Published: 2026-09-22 17:31:22+00:00 Every vendor selling a semantic layer has a number. Snowflake says Cortex Analyst lifts Claude 3.5 Sonnet from 57% to 78% on BIRD subsets. dbt says GPT-5.3 Codex hits 100% through their gateway versus 84% on raw text-to-SQL. Cube says adding a four-kilobyte semantic markdown lifts three frontier models by 17 to 23 points, landing around 68%. Denodo claims 87% on BIRD and 83% on Spider. The pattern is consistent, the lift is real, and the conclusion everyone draws is the same: semantic layers win. The problem is that none of these tests answer the question they claim to answer. ## The Flaw Hiding in Every “Semantic Layer” Benchmark There are two genuinely different claims a semantic layer can make: 1. **Context claim.** Business definitions — metrics, dimensions, join paths, grain rules, disambiguation conventions — improve LLM accuracy. This is about*information* . 2. **Compiler claim.** A deterministic compilation step, where named metrics become fixed SQL, improves reliability beyond what context alone provides. This is about*mechanism* . Every published benchmark conflates them. The “semantic layer” arm always gets richer context *and* a different mechanism at the same time. The “text-to-SQL” arm always gets raw DDL and has to invent joins, metrics, and grain from scratch. So when Snowflake reports a 21-point lift, you cannot tell whether Cortex Analyst won because it gave the model better definitions, or because it stopped the model from writing SQL at all. Both changed simultaneously. The same is true of dbt. Their text-to-SQL arm receives ACME_small.ddl — plain schema. Their semantic-layer arm receives a discovery API plus enforced joins. The model in the gateway path doesn’t even write SQL; it names metrics and MetricFlow compiles. That’s a different *task*, not a controlled comparison of the same task under two mechanisms. Cube is the only one that isolated the context variable cleanly — same questions, same model, same single-shot protocol, one arm gets DDL, the other gets DDL plus a hand-authored markdown document describing measures and conventions. No compiler, no runtime, no tool calls. Just the document pasted into the prompt. And the result is striking: all three models converge to roughly 68%, statistically indistinguishable from each other. The document accounts for essentially all the variance; which model you use barely matters. But Cube never published the other half. They showed what rich context buys you. They did not show what a live Cube compiler adds on top of that same context. The only Cube number for an actual semantic layer is the 2023 Delphi result — 100% on the data.world insurance questions — but that was a different benchmark, different questions, and a different era of models. So the cleanest experiment in the literature is half-finished, and the most-cited experiments are confounded. ## What the Public Benchmarks Actually Measure The academic suites — Spider, BIRD, BEAVER, Spider 2.0 — are scored by **execution accuracy**: run the generated SQL and the gold SQL against the same database, compare result sets row-for-row. That correctly accepts different SQL that returns the same numbers, which is good. But they test schema-only or schema-plus-evidence text-to-SQL. None of them include a semantic layer arm at all. They measure how well models write SQL against raw or lightly-annotated schemas, not how semantic layers perform. The vendor benchmarks that do include semantic layers use toy or custom sets. dbt’s ACME Insurance extract is eleven questions, each run twenty times, on a dataset totaling about 68 kilobytes. Cube’s Contoso set is a hundred questions. These are fine for isolating effects, but they are not Spider or BIRD, and they are not enterprise schemas with hundreds of columns and messy real-world values. ## The Determinism Question Nobody Has Measured The strongest argument for a compiler is not first-answer accuracy. It is consistency. Ask the same question two ways — “claims by region last quarter” and “last quarter’s claims broken down by geography” — and a model with rich context in front of it may produce two different SQL structures. SQLStructEval sampled ten queries per Spider question and found GPT-5-mini produced on average 1.9 distinct AST structures per question; the majority structure appeared only about 65% of the time. Claude 4.5 Opus was more concentrated at 69%, but still drifted. The paper measured structures, not result sets, so we don’t have a clean “identical answer X% of the time” number. But the failure mode that matters is silent: two queries that both execute, both look plausible, but one double-counts a join or drops a filter, and you get two different numbers for the same metric with no error thrown. A compiler makes that impossible by construction. A context-fed model cannot. ## The Ideal Test The experiment that would settle this is simple to describe and has not been published. **Step 1: Run it on BIRD.** BIRD is the right default public dataset. It is the only widely used suite that sits in the realistic middle: 12,751 question–SQL pairs across 95 large databases totaling 33.4 GB, dirty real-world values, external knowledge the model must use, and official execution-accuracy scoring. It is hard enough to expose whether a compiler does real work, small enough to rerun for a consistency pass, and it already has a published vendor baseline — Snowflake’s vanilla Claude 3.5 Sonnet at 57% versus Cortex Analyst at 78% on four BIRD-dev subsets. Any new result is immediately comparable. Spider is too clean for this question. Its schemas are normalized textbook examples, so rich context closes most of the gap and the compiler’s advantage shrinks toward zero. Spider 2.0 is the opposite problem: too hard and too enterprise-specific, with top models often in the mid-teens to mid-twenties, which makes it a ceiling test rather than a discrimination test. BEAVER is the right *second* stage — real private warehouse logs, 812 tables, near-zero off-the-shelf accuracy — but access is restricted, so it should follow BIRD, not replace it. On BIRD, both arms must receive the **identical** semantic artifact: the same YAML or equivalent document covering metrics, dimensions, join paths, grain rules, and disambiguation. Then vary only the mechanism. - **Arm A — LLM + semantic context.** The model sees the schema plus the full semantic definitions and writes SQL itself. - **Arm B — LLM + semantic compiler.** The model selects named metrics, dimensions, and filters. A deterministic compiler emits the SQL. Hold the model constant. Use at least one current frontier model, not only Claude 3.5 Sonnet. Score both arms with BIRD execution accuracy. Then add a consistency pass: ask each question *N* times (and, separately, paraphrase the same question) and report the percentage of identical result sets. That single table — BIRD EX for Arm A versus Arm B, plus a consistency rate — would do more to settle this debate than every vendor benchmark combined. Until someone runs it, every claim that “the semantic layer is indispensable” is selling a conclusion the data does not yet support. ## What This Means If you are evaluating a semantic layer for ad-hoc exploration on tidy data, the evidence says the investment case has weakened. Rich context gets you most of the way there, and frontier models are genuinely good at SQL now. The dbt 2026 rerun shows text-to-SQL nearly doubling from 33% to 64% in three years. If you are evaluating for governed, recurring, high-stakes analytics — board decks, KPI reconciliation, anything that has to produce the same number twice — the compiler’s value is real but unquantified against the right baseline. The published numbers overstate it by comparing against weak baselines. The honest number is probably smaller than the headlines, and it lives in consistency and refusal behavior, not in first-shot accuracy. The field does not need another custom 11-question insurance demo. It needs Step 1 on BIRD: identical context, different mechanism, execution accuracy plus consistency. ## Summary We at Strata are currently setting up and executing this test. Early results show that a well defined description layer alone (YAML in our case) boosts performance by 20%. Now the question that’s important for me and other vendors in this space is whether a semantic layer can improve accuracy even further. By semantic layer we mean one that can be configured via YAML or other means, takes as input higher level abstractions like dimensions and measures with no reference to tables and joins, and uses that input to compile SQL queries. Once we have our results we’ll be publishing here. Be sure to subscribe. The results might go against us, but we will publish either way.