# For agents, a semantic layer is only as good as its query engine > Source: > Published: 2026-11-09 00:00:00+00:00 SLayer came out of building Motley on top of Cube. We were connecting agents to our customers’ warehouses, every one shaped differently, and keeping the Cube models in sync got to be too much, so we wrote our own semantic layer. Since then I’ve been arguing that what matters most for agents is the query engine: what you can compute at the moment you ask, rather than what someone defined in advance. That’s an easy thing to claim and a harder thing to show. So we built a test: one database, one semantic model, and 75 business questions, given to SLayer, Cube and dbt MetricFlow. Counting only answers that express the question exactly as asked, SLayer answered 65, Cube 15 and MetricFlow 12, all working from the same model. ## Why I care about the engine A few weeks ago I wrote that [agents need a grammar, not a list of sentences](https://motley.ai/blog-posts/agents-need-a-grammar/). The short version: the model gives you the words (revenue, a customer, a ship date), and the engine decides what you can say with them. Dashboards ask the same few questions every day, so a list of predefined metrics covers them. People talking to an agent don’t work like that. They ask which carriers deliver late, then the same by supplier, then only the suppliers that got worse since last year. Nobody models that in advance. When the engine can’t build the follow-up, the agent either says it can’t help, or it goes around the semantic layer and writes its own SQL. The data teams we talk to tell us neither is acceptable once the answer reaches a customer or an exec. I wanted to know how often it happens on questions a real business would ask. ## How we set it up We wanted the engine to be the only thing that changed. So: one database, one model, one set of questions, and every answer checked automatically against SQL we wrote by hand. ### The data We used TPC-H at scale factor 1. It’s the standard decision-support dataset, and it describes a business that’s easy to picture: a wholesale distributor selling industrial parts to customers around the world, with orders from 1992 to mid-1998. | Dataset | What it holds | Rows | |---|---|---| | lineitem | Each order line: part, supplier, quantity, price, discount, ship, promised and receipt dates, ship mode, return flag | 6,001,215 | | orders | Order date, status, priority, clerk, total price, customer | 1,500,000 | | partsupp | Each supplier’s offer for a part: available stock and unit supply cost | 800,000 | | part | Name, brand, type, container, size, manufacturer | 200,000 | | customer | Name, market segment, account balance, nation | 150,000 | | supplier | Name, nation | 10,000 | | nation, region | 25 nations in 5 regions, used for both customers and suppliers | 30 | It also has a few of the awkward bits real data has: two facts at different grains (order lines and supplier offers) joined on a two-column key, countries and regions that play two roles (where the customer is, where the supplier is), and three dates on every order line (shipped, promised, received). Operations people love those dates. ### The model We wrote the semantic model once, in [OSSIE](https://ossie.apache.org/), the Open Semantic Interchange format. It’s roughly what a good data team would write in its first week: 10 datasets, 35 described fields, 11 relationships and 11 metrics, from revenue and order count to customer count and available stock. What’s missing is on purpose. No late-delivery rate, no margin, no growth rate, no ranking, no cohort. That’s the stuff a business asks about every day, and in this test each engine had to build it at query time. ### Getting it into each layer Each layer got a mechanical translation of that model. Nothing added, nothing rewritten. | Layer | How the model was loaded | What did not carry over | |---|---|---| | SLayer | Native OSSIE import: 10 of 10 datasets | Nothing | | Cube | Translated to 10 cubes; all load | Nothing | | dbt MetricFlow | Translated to dbt semantic models | All 11 metrics load. Its joins stop at two hops, so customer nation and region are out of reach from an order line | The rules were tight. An answer only counts if it comes from the layer’s own query interface: SLayer’s query API, Cube’s REST API (or its SQL API when the query maps to a single REST query), and MetricFlow’s `mf query`. No raw SQL, no post-processing, no model edits, no columns the model doesn’t declare. Every answer is compared automatically with the ground truth, to one part in a million. ### The questions I was careful about this part. If we write the questions ourselves, knowing what SLayer can do, we’ll pick questions SLayer can answer, even without meaning to. So we had them written by an AI agent that only saw a description of the business and the model’s field names. It never saw a semantic layer, and its only brief was: would someone running this company actually ask this? It generated 50 questions across operations, finance and marketing. Each question was associated with a role and a decision and graded simple/medium/hard. Additionally, we asked another blind agent to generate 25 analytically harder questions. That’s 75 in total: 15 simple, 25 medium, 35 hard. A few of them: | Asked by | Question | To decide | |---|---|---| | Head of Logistics | What share of order lines arrived later than promised, by shipping mode? | Which carrier contracts to renegotiate | | Head of Procurement Finance | What revenue, margin and margin rate did we earn in 1997 by supplier region? | Which sourcing regions deliver the best margins | | CRM Manager | Of customers who bought in 1997, what share bought more than once, by segment? | Where to run a repeat-purchase campaign | | CFO | How much of our 1997 revenue came from our 100 biggest customers? | How exposed we are to a few accounts | | COO | What share of orders arrived complete and on time, by year and priority? | Whether perfect-order performance is improving | | FP&A Manager | What was trailing twelve-month revenue at the end of each month of 1997, against a year earlier? | How to read the trend without seasonality | | CRM Manager | By year of first order, what share of new customers ordered again the next year? | Whether newer cohorts retain as well as older ones | Each batch was frozen in our [git repo](https://github.com/MotleyAI/semantic-layer-bench), with its SQL answers and their hashes. One question came back empty (a quirk of how the dataset is generated), so we moved it to another year and logged the change. Nothing else was touched. ## What happened | Layer | Answered exactly as asked | Correct via workaround or luck | Could not express | Wrong | |---|---|---|---|---| | SLayer | 65 | 10 | 0 | 0 | | Cube | 15 | 0 | 60 | 0 | | dbt MetricFlow | 12 | 0 | 63 | 0 | The “Answered exactly as asked” scoring is strict. An answer counts only if the query asks the question as written, without any non-obvious workaround and without leaning on a quirk of this dataset. If you count every query that returns the right table, SLayer got every one of the other 10 questions right – more on that below. Nobody returned a wrong number, which I was glad to see. If the agent couldn’t answer something based on the semantic layer, it refused. In fact, every miss was due to the inability to express the question. *results_blind.json, semantic-layer-bench, 75 blind questions on TPC-H SF1, strict count, November 2026* Difficulty is where it gets interesting. On simple questions (a defined metric, filtered and broken down) Cube gets 12 of 15 and MetricFlow 9 of 15. On medium questions both drop to 3 of 25. On the 35 hard questions both get zero. SLayer gets 25 of the hard ones as asked; with the other ten we hit our own limits. | Domain | SLayer | Cube | MetricFlow | |---|---|---|---| | Operations (25) | 22 | 3 | 2 | | Finance (26) | 21 | 6 | 5 | | Marketing (24) | 22 | 6 | 5 | By far, the biggest gap is in operations. Most ops questions are about late deliveries, and “late” means one date on the order line is after another, which turns out to be surprisingly hard to say in a semantic query. ## One question, three engines Take the very first ops question: *what share of order lines arrived later than promised, by shipping mode?* A line is late when it was received after the date the supplier committed to. The model has a line count and both dates. It doesn’t have a late-delivery rate. In SLayer, the rate is just a formula on two fields the model already declares: ``` { "source_model": "lineitem", "dimensions": ["ship_mode"], "measures": [ "line_count", {"name": "late_share", "formula": "avg(iif(receipt_date > commit_date, 1.0, 0.0))"} ] } ``` Cube’s REST API gives you the line count by shipping mode, no problem. What it can’t do is define a new measure inside the query, compare one field with another, or divide one measure by another. Its SQL API will take the SQL for the rate, but it runs it around the semantic query, so at that point it’s the database doing the work. MetricFlow can filter to the lines received after the promised date and count them. But the filter applies to the whole query, so you lose the total, and `mf query` can’t divide two metrics anyway. You’d have to add a ratio metric to the model first. The follow-up is usually harder. *Which three brands sold the most in each customer region in 1997?* SLayer ranks within each region in the same request: ``` { "source_model": "lineitem", "dimensions": ["orders.customer.customer_nation.customer_region.customer_region_name", "part.brand"], "measures": ["revenue", {"name": "brand_rank", "formula": "rank(revenue, partition_by=orders.customer.customer_nation.customer_region.customer_region_name, direction='desc')"}], "filters": ["orders.order_date in '1997'", "brand_rank <= 3"] } ``` Cube’s REST API can’t rank within groups. MetricFlow can’t even reach customer region from an order line: it’s four joins away, and MetricFlow stops at two. ## Why the gap opens Only 15 of the 75 questions are a defined metric, filtered and broken down. They’re exactly the 15 Cube answers, and MetricFlow’s 12 are all in there. The other 60 need the engine to build something the model doesn’t have. *Calculations tagged per question after the set was frozen; questions_blind/questions.yaml, semantic-layer-bench* Nothing on that list is exotic. A late-delivery rate, the revenue that sits with the top 100 accounts, how many of last year’s new customers came back: these are Monday-morning questions. An engine that can only fetch defined metrics answers the first one and gets stuck on the follow-up. ## What I take from this To be fair to Cube and MetricFlow, both can answer a lot of those 60 if someone defines the metric first. That’s how they’re meant to be used, and for a known set of dashboards it works fine. With agents, you don’t know the questions in advance. Every new rate or comparison becomes a ticket for the data team, and until it ships, the agent either gives up or writes its own SQL, which brings back the guessed joins and drifting numbers the semantic layer was supposed to fix. My rough rule of thumb: - If the questions are known and stable (a set of dashboards, a monthly pack), defining metrics up front works, and any of the three will do the job. - If an agent is doing the asking, look at the engine first: how far can it go from the definitions you already have, without anyone touching the model? And if you’re choosing one, write down 50 questions your business actually asks before you look at any tool. Then count. ## Where this test is weak We build SLayer and we ran the test, so read it with that in mind. The questions were written by an AI agent. We kept it away from every tool, but real people might ask different things, which is why all 75 are published with who asks them and why. AI agents also ran the queries, with instructions to get the best fair result for each layer and to log everything they tried. If you know Cube or MetricFlow better than we do and spot a query we missed, I’d really like to hear about it. Then there are the ten SLayer answers we didn’t count. Two ask for a running total over a ranking (how many suppliers make up 80% of revenue, say). SLayer’s running total only works along a time axis, so the query turns each rank into a date, runs the total and turns it back. It gets the right number, but I wouldn’t expect an agent to come up with that on its own. On six ranking questions, SLayer can’t apply the tie-break the question asks for, and it only matches because this data has no ties at the cut-off. And two queries identify a part by name plus manufacturer, because the model has no part key; that only works because the pair happens to be unique here. Count those ten and SLayer gets 75 of 75. We picked the strict rule after seeing the results and decided to lead with it anyway. I’d rather defend 65 than explain away 75. Last thing: we only measured what each engine can express. Speed, cost, and what each layer gains when you model ahead aren’t part of this. ## Try it Everything is on GitHub at [MotleyAI/semantic-layer-bench](https://github.com/MotleyAI/semantic-layer-bench): the OSSIE model and each translation, the questions as they were written, the SQL answers and their hashes, every query each layer ran or tried, and the scoring harness. [BLIND_RESULTS.md](https://github.com/MotleyAI/semantic-layer-bench/blob/master/BLIND_RESULTS.md) has every miss with its reason. SLayer is open source under MIT. Install it with `uv tool install 'motley-slayer[all]'`, import an OSSIE model, and point Claude at it over MCP. The [docs](https://docs.motley.ai/slayer/) cover formulas and multi-stage queries. How are you handing your warehouse to agents today?