{"slug": "how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals", "title": "How do you know if your semantic layer is bad? Part 2: offline evals", "summary": "Offline evaluations using questions with fixed, verified answers let teams measure and catch regressions in the semantic layers that AI data agents rely on, according to a Motley blog post. The post recommends pointing each eval question at a closed past period, verifying the answer once against a trusted source such as a dashboard or BI tool to make it a golden answer, then re-running the evals on every change or on a schedule. It advises spawning a fresh subagent per question with read-only semantic layer tools, such as Motley's search, inspect, query and list_resources tools, so the agent cannot read the answers file or alter data.", "body_md": "**TL;DR:** By having a list of questions with known, unchanging answers, you can evaluate your data agents and measure improvements or regressions in the semantic layer.\n\nOf all the reasons to have a semantic layer, the main one today is increasingly letting AI agents answer data questions correctly. How do you know whether your semantic layer is doing a good job of that, and how do you improve it?\n\nThere are many possible friction points. Do the models cover everything users ask? Are the descriptions good enough for the agent to discover and understand the metrics? Are there gaps in context, unexplained gotchas in the data?\n\nIn [Part 1](https://motley.ai/blog-posts/how-do-you-know-if-your-semantic-layer-is-bad-part-1-analyzing-usage), we wrote about how looking at real users’ sessions lets you understand how to make the semantic layer better at answering their real-world questions.\n\nCatching problems when users hit them is great, but would you rather avoid them happening in the first place? Though you can’t ensure this for every possible question, there is a way to check how healthy the system is, or catch regressions when making changes – enter offline evals.\n\n## Preventing errors from happening\n\nThe idea is to ask an agent a question and check if the answer is correct. Again, you can’t do this for every possible question, but it’s good to have a set of basic or important ones – critical to answer correctly no matter what. Then you can measure accuracy, re-evaluate on every change or on a schedule, and know when things start breaking before the users do.\n\nIf we use a software engineering analogy:\n\n- Usage analysis = error tracking\n- Offline evals = integration tests\n\nAccuracy is easy to measure if you have the correct answer you can compare with. Usually you want to point each question at a closed period in the past, where the numbers can’t change, and verify the answer once against a trusted source – like a dashboard or a BI tool, or just by carefully checking the query. Then this answer becomes golden, and whenever you ask an agent this exact question, it should answer with this exact number.\n\nYou might wonder, what if the user’s agent answers differently from mine? Sure, LLMs are non-deterministic, so there isn’t a way to be 100% sure about any behavior. But the main job with evals is to detect friction points in the semantic layer – and those tend to show up consistently. An unclear definition, missing metric or wrong join will cause issues with every agent, and evals help catch those.\n\nThat being said, it’s best to use the same model/harness your users use. Also, exactly because LLMs are non-deterministic, some teams like to run their evals multiple times in a row, to see which errors are consistent and which are intermittent.\n\n## Implementation tips\n\nRunning the evals themselves typically involves the following process for every question:\n\n1. Launch an agent and ask the question\n2. Wait until it answers\n3. Check the answer against the golden one (usually with an LLM or deterministically)\n\nOur favorite simple trick is doing everything in one session. Ask your agent to spawn a subagent to answer each question – subagents typically have their own, fresh context, and can’t see the other questions or the fact that they’re inside an eval.\n\nTo block the subagents from reading the answers file or changing anything, only give them access to read-only semantic layer tools. For example, in Claude Code, you can put something like this into `.claude/agents/eval-runner.md`:\n\n```\n---\nname: eval-runner\ndescription: Answers a single data question using only Motley tools. Used for evals.\ntools: mcp__motley__search, mcp__motley__inspect, mcp__motley__query, mcp__motley__list_resources\n---\nAnswer the user's data question using the Motley tools. Give the final number(s) on the last line.\n```\n\nAnd here’s an example prompt for the main agent:\n\n```\nRead evals/questions.yaml. For each question, spawn a separate eval-runner\nsubagent with ONLY the question text as the prompt. Run them in parallel.\nDon't tell the subagents they are being evaluated, and don't add hints.\n\nWhen all are done, write each answer to evals/results.md.\nOnly then read evals/answers.yaml and grade each answer as PASS / FAIL.\nList the failures with the likely cause.\n```\n\nUsually it’s good to store eval questions and golden answers in the same repo where the models are – this way they are coupled and versioned together, and each particular version of the semantic layer has a question-answer contract attached.\n\n## Where to get eval questions\n\nThe right number of questions in the eval set depends on the size of your semantic layer. Generally, a couple dozen is a good starting point – it will probably grow faster than you think.\n\nIf you already know what your users will want to ask, gather a number of concrete questions pointed at a closed time period, run them against your agent, verify and record the answers. Otherwise, take dashboards or any other sources of truth you currently have, and turn those into evals.\n\nAlso, consider including questions that the agent can’t and shouldn’t answer. Grade them as passed only if the agent actually says it can’t answer – you don’t want it to confidently return a guessed number.\n\nSo you now have an initial question set, say, based on your dashboard numbers, but users will ask all sorts of things – how to increase the coverage?\n\nWe suggest looking at the questions users ask (see how to do it in [Part 1](https://motley.ai/blog-posts/how-do-you-know-if-your-semantic-layer-is-bad-part-1-analyzing-usage)) and taking questions from there. Even better, when a user reports a problem and you fix an issue in the semantic layer, make an eval question based on it with a recorded answer, just like you would write tests for a fixed bug. This way your set will grow naturally and you’ll be sure those issues don’t come back.\n\nRead [Part 1](https://motley.ai/blog-posts/how-do-you-know-if-your-semantic-layer-is-bad-part-1-analyzing-usage) to learn about analyzing actual user sessions and improving the semantic layer based on them.", "url": "https://wpnews.pro/news/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals", "canonical_source": "https://motley.ai/blog-posts/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals", "published_at": "2026-09-25 00:00:00+00:00", "updated_at": "2026-09-25 18:58:52.646781+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "ai-products"], "entities": ["Motley", "Claude Code"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals", "markdown": "https://wpnews.pro/news/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals.md", "text": "https://wpnews.pro/news/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals.txt", "jsonld": "https://wpnews.pro/news/how-do-you-know-if-your-semantic-layer-is-bad-part-2-offline-evals.jsonld"}}