TL;DR: By having a list of questions with known, unchanging answers, you can evaluate your data agents and measure improvements or regressions in the semantic layer.
Of all the reasons to have a semantic layer, the main one today is increasingly letting AI agents answer data questions correctly. How do you know whether your semantic layer is doing a good job of that, and how do you improve it?
There are many possible friction points. Do the models cover everything users ask? Are the descriptions good enough for the agent to discover and understand the metrics? Are there gaps in context, unexplained gotchas in the data?
In Part 1, we wrote about how looking at real users’ sessions lets you understand how to make the semantic layer better at answering their real-world questions.
Catching problems when users hit them is great, but would you rather avoid them happening in the first place? Though you can’t ensure this for every possible question, there is a way to check how healthy the system is, or catch regressions when making changes – enter offline evals.
Preventing errors from happening #
The idea is to ask an agent a question and check if the answer is correct. Again, you can’t do this for every possible question, but it’s good to have a set of basic or important ones – critical to answer correctly no matter what. Then you can measure accuracy, re-evaluate on every change or on a schedule, and know when things start breaking before the users do.
If we use a software engineering analogy:
- Usage analysis = error tracking
- Offline evals = integration tests
Accuracy is easy to measure if you have the correct answer you can compare with. Usually you want to point each question at a closed period in the past, where the numbers can’t change, and verify the answer once against a trusted source – like a dashboard or a BI tool, or just by carefully checking the query. Then this answer becomes golden, and whenever you ask an agent this exact question, it should answer with this exact number.
You might wonder, what if the user’s agent answers differently from mine? Sure, LLMs are non-deterministic, so there isn’t a way to be 100% sure about any behavior. But the main job with evals is to detect friction points in the semantic layer – and those tend to show up consistently. An unclear definition, missing metric or wrong join will cause issues with every agent, and evals help catch those.
That being said, it’s best to use the same model/harness your users use. Also, exactly because LLMs are non-deterministic, some teams like to run their evals multiple times in a row, to see which errors are consistent and which are intermittent.
Implementation tips #
Running the evals themselves typically involves the following process for every question:
- Launch an agent and ask the question
- Wait until it answers
- Check the answer against the golden one (usually with an LLM or deterministically)
Our favorite simple trick is doing everything in one session. Ask your agent to spawn a subagent to answer each question – subagents typically have their own, fresh context, and can’t see the other questions or the fact that they’re inside an eval.
To block the subagents from reading the answers file or changing anything, only give them access to read-only semantic layer tools. For example, in Claude Code, you can put something like this into .claude/agents/eval-runner.md:
---
name: eval-runner
description: Answers a single data question using only Motley tools. Used for evals.
tools: mcp__motley__search, mcp__motley__inspect, mcp__motley__query, mcp__motley__list_resources
---
Answer the user's data question using the Motley tools. Give the final number(s) on the last line.
And here’s an example prompt for the main agent:
Read evals/questions.yaml. For each question, spawn a separate eval-runner
subagent with ONLY the question text as the prompt. Run them in parallel.
Don't tell the subagents they are being evaluated, and don't add hints.
When all are done, write each answer to evals/results.md.
Only then read evals/answers.yaml and grade each answer as PASS / FAIL.
List the failures with the likely cause.
Usually it’s good to store eval questions and golden answers in the same repo where the models are – this way they are coupled and versioned together, and each particular version of the semantic layer has a question-answer contract attached.
Where to get eval questions #
The right number of questions in the eval set depends on the size of your semantic layer. Generally, a couple dozen is a good starting point – it will probably grow faster than you think.
If you already know what your users will want to ask, gather a number of concrete questions pointed at a closed time period, run them against your agent, verify and record the answers. Otherwise, take dashboards or any other sources of truth you currently have, and turn those into evals.
Also, consider including questions that the agent can’t and shouldn’t answer. Grade them as passed only if the agent actually says it can’t answer – you don’t want it to confidently return a guessed number.
So you now have an initial question set, say, based on your dashboard numbers, but users will ask all sorts of things – how to increase the coverage?
We suggest looking at the questions users ask (see how to do it in Part 1) and taking questions from there. Even better, when a user reports a problem and you fix an issue in the semantic layer, make an eval question based on it with a recorded answer, just like you would write tests for a fixed bug. This way your set will grow naturally and you’ll be sure those issues don’t come back.
Read Part 1 to learn about analyzing actual user sessions and improving the semantic layer based on them.