It started from an intuition. Why are examples so often recommended to help LLMs perform well? For our brains, it’s clearly easier to adapt an example than to reconstruct a solution from scratch. Is it the same for LLMs? Does it help a model if it has less “thinking” to do to get from the information it’s given to the answer?
We call that remaining work the derivation distance: how much transformation and composition the model still has to do to get from the facts in its context to the answer. A general rule leaves a long distance. A worked example with the same shape as the question leaves almost none.
There’s a whole range of possibilities when editing context: one can state general rules, provide snippets for common patterns, provide full-fledged examples, and in practice it’s usually a bit of a mix. We set out to evaluate what works best. This study is focused on data analytics use cases. We expect the findings to transfer to other use cases, but that would need validation. It’s easy to run a similar experiment on your own use cases. I’d be interested in the results if you do!
This post is the story of how we approached that, because the wrong turns are also interesting. They surface what makes a good context.
What we found
- Completeness comes first. If a business rule is missing, getting the right answer amounts to playing the lottery.
- Shortening the derivation distance pays off on hard questions: adding a full example library raised Haiku from 70% to 82% and Sonnet from 81% to 93%.
- Removing the rules and metrics beneath that library left standard accuracy unchanged and hard accuracy within one to four points, while cutting context by roughly 30%.
- Opus was already at 98% with complete English descriptions. Metrics and examples added only one percentage point.
- A few relevant examples were not enough. The gains appeared with a broad library covering many query patterns and the small conventions around them.
- Consistency matters more than volume. A single contradiction can erase the benefit of additional context.
- There is no universal best setup. Smaller models benefit from comprehensive examples. Stronger models perform well with leaner context.
Reproducibility
The code needed to reproduce all the experiments mentioned in this article is open source on GitHub.
The setup #
We started from three public analytics datasets: jaffle-shop (a toy cafe chain), theLook (a synthetic e-commerce store), and Olist (a real Brazilian marketplace export). Each comes with a context layer that we initially built to test Cassis: table and column descriptions, business rules written in prose, and named metrics with their SQL. For each dataset, we also generated 12 standard questions and 11 hard questions with known answers.
The questions come in two tiers. The standard tier is straightforward: “what is our total revenue”, “what is revenue by store for the top five stores”, “what is our average order value”. One aggregate, one or two joins, done. The hard tier is multi-step composition: “what is the month-over-month revenue growth rate for each month” needs a CTE that aggregates to months and then a LAG window function that compares each to the one before. “What is each store’s share of total company revenue” needs a window SUM over a grouped aggregate. “How many customers spent more in the last 90 days than in the 90 days before that” needs two non-overlapping date windows, per-customer conditional aggregation, and a comparison. These are not trick questions, and we wrote them so that the rules stated in the context would pin down a single answer (we did not fully succeed on the first try, as you will see). We will come back later to their purpose.
For each question we render the context at a series of levels supposed to vary the derivation effort, and hand it, with the question, to a model. The model writes SQL, we run it against the actual database, and compare the result to the known answer. We do this for three models spanning a wide capability range (Haiku 4.5, Sonnet 4.6, Opus 4.8), repeat each cell three times to catch flakiness, and count a model that writes invalid SQL as wrong. That makes for a total of 2,484 runs.
The grading deserves a paragraph, because it turned out to be one of the wrong turns. The base comparison is standard execution accuracy: treat the result set as an unordered bag of rows, ignore column names (models alias freely), and round floats to four decimals. Our first grader stopped there, and an audit of the failures showed that roughly a third were materially correct answers presented differently: a ROUND(x, 2)
on a growth rate, an extra column here or there, or a date where the reference answer had a midnight timestamp. Nothing in the prompt told the model these conventions, so exact-match grading was unfairly punishing. It distorted every curve in favor of the models that happened to share the preferences of the gold answer generator. The grader now accepts presentation variants of the correct answer: the requested values in any column order, with extra columns alongside, at the model’s own display precision (two decimals minimum), and treating a date and a midnight timestamp as the same instant. It still rejects anything that changes the values themselves: a fraction where a percentage was asked, an entity ID where a name was asked, or a missing column. All numbers below use this grading.
One disclosure before the results, for the same reason. The questions, reference answers, and example library were drafted with Opus 4.8, the strongest model in the lineup, then executed and audited. Tolerant grading removes the presentation bias, but wherever a question leaves a genuinely free choice, like which order to break a tie in, the reference answer embodies Opus’s default. On those choices, grading Opus against this key partly measures self-agreement, and the other models partly measure how well their defaults match Opus’s. Keep that in mind when reading Opus’s near-perfect scores; the comparisons within a single model, which carry most of the conclusions, do not depend on it. We will come back to this when it matters.
The first curve, and why it didn’t say much #
The first test did the simplest thing: test at different context levels, progressively adding more elements from our structure. Level zero was bare table and column names. Each level added more elements of the context layer, from descriptions to formulas to worked examples. More context = less derivation effort, right? The curve looked exactly like the theory predicted: accuracy rose steadily as we added context, the weak model rose more steeply than the strong one, and everything fanned out nicely. But it measured the wrong thing for a simple reason: at level zero, with only tables and column names, the model had no idea of the business rules, such as how revenue should be measured. Each run at level 0 pretty much amounted to rolling dice. Even worse, it did not have temporal information about the dataset, so all time-related questions could not be answered.
So of course, as you add this information, the answers get better. But what this measured was whether the context contained the information required to answer at all. In other words, it measured completeness. And yes, completeness improves results, but that’s not exactly breaking news.
Once the context is complete, a more interesting difference remains: how much work the model still has to do to get from the facts it sees to the answer. That is the derivation distance from the introduction. So we reworked our context levels to hold completeness constant while varying how much of the solution path was already spelled out.
Holding completeness constant #
The fix for completeness is easy in theory: make sure every level has all the facts. It’s a bit harder in practice and took a few iterations of analyzing the failures to converge.
From there, three things matter. First, completeness: is the answer possible from the supplied facts? Second, derivation distance, as defined in the introduction: how much of the solution path is still left to the model? Third, context burden: how much does the context cost to send and maintain, and how many opportunities does it create for contradictions? A good context needs to balance all three.
We ended up with three context levels. First, the facts are present in plain English, scattered across table and column descriptions. The next level adds structured metrics with their names, descriptions, and SQL expressions. The last one includes a library of examples. For each question, we generated one example query that answers a very similar but not identical question. All the examples were included in the context, so the model saw the same context for every question.
This ladder is additive: the examples level also contains the descriptions, rules, and metrics from the levels below it. It tells us that the full package works, but not whether the examples need those preceding layers. We return to that distinction below.
Now, with completeness constant, we should be able to measure derivation distance. It appeared to make a difference.
On the standard question tier, the gradient is real but modest. Pooled across the three datasets, Haiku goes from 82% at the terse-but-complete baseline to 87% with metrics and 94% with examples. Opus barely needs the help: it starts at 94% and reaches 100% with the example library. A context with minimal derivation distance is able to get even the smallest models to very good accuracy.
But the main takeaway is that this benchmark is saturated. With Opus and Sonnet reaching 100%, the variation in the results could be mostly measurement error. This is why we introduced the harder tier of questions.
On the hard tier, the models separate cleanly. Opus runs close to the ceiling from the start: 98% on terse descriptions, 99% with metrics and with examples. The smaller models climb the whole way up. Metrics add a handful of points over descriptions (Haiku 63% to 70%, Sonnet 76% to 81%), and examples add twelve more for each, to 82% and 93%:
While still saturated for Opus, the harder benchmark separates both the models and the context levels more significantly. Descriptions make the question answerable. Metrics remove one local derivation: how a measure is calculated. Examples can remove the harder structural derivation: how to compose the joins, aggregates, windows, and ranking into a complete query. The example library helps more than the whole metrics layer. For Opus, the curve is nearly flat because it is already able to perform that composition from English descriptions.
How to interpret this? #
Let’s dive into what the models actually see.
Take the question “What is the month-over-month revenue growth rate for each month?” At the first level, plain English rules, everything the model knows about revenue is one sentence in the ORDERS
column descriptions:
SUBTOTAL (DOUBLE): Order value before sales tax, in USD.
The basis for revenue and order-value figures.
The fact is there, so the context is complete. But the model has to build the whole query from nothing: the monthly aggregate, the window function, and the growth arithmetic. This is the maximum derivation distance.
With metrics, the model sees this instead:
revenue
- Expression: SUM("SUBTOTAL")
- Table: JAFFLE.ORDERS
That formula tells more clearly how revenue is measured. It says nothing about how to compose a monthly aggregate with a window function. The model has to invent the CTE, pick date_trunc
, figure out LAG
, and wire the growth-rate arithmetic.
When the example library is in the context, among two dozen other solved questions the model finds this:
## What is the month-over-month order-count growth rate (%) for each month?
WITH monthly AS (
SELECT date_trunc('month', ORDERED_AT) AS month, COUNT(*) AS orders
FROM ORDERS GROUP BY 1
)
SELECT month, orders,
100.0 * (orders - LAG(orders) OVER (ORDER BY month))
/ LAG(orders) OVER (ORDER BY month) AS growth_pct
FROM monthly ORDER BY month
Not the same question, but the same shape. Swap COUNT(*)
for SUM(SUBTOTAL)
and you’re done. Much easier!
The capability gap between models is widest before examples enter. At plain descriptions, Opus already clears 98% of the hard tier while Haiku sits at 63%. The stronger model is able to derive a correct answer from high-level rules; the weaker one needs a demonstration.
Is the gain from examples or just more context? #
The main ladder changes two things at once: it shortens the derivation path, but it also makes the context longer. Because the ladder is additive, perhaps examples only work because they arrive on top of all the rules and metric definitions beneath them. Or perhaps the model simply benefits from seeing more context. We ran an additional 621-call ablation to check.
The ablation keeps the complete foundation: global operating facts plus table and column descriptions. Without those, the context would be incomplete. It then adds the same full example library, but removes the separate domain-prose, named-metric, expression, and filter sections. So this is “descriptions plus examples,” not a literal question-plus-examples prompt. The rendered body is still identical for every question in a dataset.
| model | tier | full stack: descriptions + rules + metrics + examples | descriptions + examples | difference |
|---|---|---|---|---|
| Haiku 4.5 | standard | 94% | 94% | 0 pt |
| Sonnet 4.6 | standard | 100% | 100% | 0 pt |
| Opus 4.8 | standard | 100% | 100% | 0 pt |
| Haiku 4.5 | hard | 82% | 81% | -1 pt |
| Sonnet 4.6 | hard | 93% | 89% | -4 pts |
| Opus 4.8 | hard | 99% | 98% | -1 pt |
On the standard questions, removing the separate rules and metric layers changed nothing. On the hard questions, accuracy fell slightly, most visibly for Sonnet. With three samples per cell, none of these differences is statistically clear; the smallest exact matched-test p-value is 0.125. But this experiment is not powered to establish equivalence either. The result is consistent with most of the example benefit surviving without those layers; it does not prove they have no effect. Sonnet’s four-point hard-tier difference may be noise, or it may reflect a real advantage on some compositions.
The context got substantially leaner. Mean context size fell from 7,337 to 5,019 tokens for Haiku and Sonnet, and from 11,852 to 8,276 for Opus: reductions of 32% and 30%, respectively. The rendered text was identical across models; Opus reports a higher count because it uses a different tokenizer. Mean uncached query cost fell by 28% to 29%.
This strengthens the derivation-distance interpretation of the original curve. The useful variable was not simply the number of layers or tokens in the prompt, but whether the context contained reusable solution paths. Once the factual descriptions are complete, the broad example library appears to carry most of the implementation benefit itself. Separate rules and metrics may still provide governance, reusable definitions, and an advantage on some harder compositions; the ablation only shows that stacking every representation is not automatically better.
When more context gives worse results #
We didn’t get these results right away. On our first pass over this grid, Opus did something strange on the hard tier: it scored better with terse descriptions (86%) than with the full metrics layer (77%). Adding context was making the strongest model worse. We studied the failures question by question.
Reading the failing SQL next to the context showed what changed, and it was not the model’s reasoning. The context was contradicting our own answer key.
- The catalog notes for one dataset said to report products with no category “as uncategorized rather than dropping them silently”. The reference answer keeps them as a NULL-labeled group. With terse descriptions Opus produced the NULL group and passed; once the prose entered the context, it did what the prose said, labeled the group ‘uncategorized’, and got marked wrong.
- The metrics prose said “use the metric definitions exactly; do not re-derive them”, and the revenue metric filters to delivered orders. On a cohort question, Opus applied that filter everywhere, including to the first-purchase date that defines the cohort, where the reference answer considers all orders.
- The rest sat on questions with two defensible readings, like “last 90 days” with an open or a closed upper bound, where the reference answer took one reading and nothing in the context said which.
The defect was in the benchmark. A stronger model is also a more obedient one: it reads everything you give it and does what the words say, so every contradiction and every unstated convention in the context becomes a place where it can diverge. The weaker models made the same mistakes; they just had less accuracy to lose.
This highlights how easy it is to inadvertently introduce contradictions in the context, and their devastating effect on performance.
So we fixed the context. We pinned the window convention, scoped metric filters to the measures they define, aligned the category rule with the reference answers, reworded one question that asked about “US states” while its answer counted every country, and trimmed four reference answers that returned columns nobody asked for. On the twenty affected questions, Opus went from 70% at descriptions and 47% at metrics to 93% and 100%. The grids in this post come from a fresh run on the fixed contexts.
The lesson: when a model gets worse as you add context, look for a contradiction before blaming the model. And sometimes, less context can yield better results.
Are we just testing retrieval? #
If examples are what shorten the derivation, we need to know whether the model is adapting a reusable pattern or merely retrieving a near-answer. Because of the way our test set was built (one example similar to each question), it would be fair to ask whether we’re testing the ability of the model to retrieve the matching example, rather than its ability to generalize from a set of examples. The former would be much less useful than the latter, as you obviously can’t write examples for every possible question.
So we ran another version of the test, leave-one-out: for each question, the example library is rebuilt from all the other questions, with that question’s own analog held out. The model gets a gallery written for a different set of questions that target the same tables, and it has to generalize from that gallery.
| hard tier, accuracy | metrics | example library, all-inclusive | example library, leave-one-out |
|---|---|---|---|
| Haiku 4.5 | 70% | 82% | 79% |
| Sonnet 4.6 | 81% | 93% | 97% |
| Opus 4.8 | 99% | 99% | 100% |
Removing the question’s matching example changes very little: Haiku slips three points, Sonnet and Opus land at or above their all-inclusive scores. The example library generalizes well: its value comes from examples written for other questions on the same tables, not from finding the one that matches. This is not merely an exact-analog retrieval test. The models can use examples written for adjacent questions.
We nearly concluded the opposite, twice. With the original exact-match grader, holding out the matching example appeared to cost around 18 points, and the library looked like a lookup table. The grading audit showed that without its twin to imitate, the model still computed the right numbers but formatted them its own way, and the strict grader failed the lot: the matching example was teaching form as much as substance. Fair grading shrank the penalty to a few points, and the context fixes mostly erased those too.
Note: We ran the same comparison with a heavier gallery of near-complete answer templates instead of the examples (the gold query with a single value blanked). This was initially our lowest derivation distance test set, but we removed it as it was completely unrealistic. Including all templates in the context, every model scores 99% or 100%. Leaving the matching one out, the scores fall back to the same range as the held-out examples: Haiku 78%, Sonnet 90%, Opus 100%. A template gallery has no edge over an example gallery once you stop testing it on questions it already answers.
What is each example adding exactly? #
The next question is what property of the library reduces the distance: one relevant analog, generic SQL priming, or broad coverage of possible solution shapes. The held-out library still helps, but one could make a fair objection: maybe any example would, just by showing the model what a SQL query against this schema and warehouse looks like, rather than how these specific tables are used. So we built two libraries of the same size, with three examples each. One contained the three most relevant examples, selected by shared tables and query shape. The other contained the three least relevant. Same amount of context, same number of examples. If the content carries information, the relevant set wins. If it is generic priming, the two tie above the baseline. If the examples are dead weight, they tie at it.
| hard tier, accuracy | no examples | 3 irrelevant examples | 3 relevant examples |
|---|---|---|---|
| Haiku 4.5 | 70% | 70% | 72% |
| Sonnet 4.6 | 81% | 85% | 84% |
| Opus 4.8 | 99% | 96% | 99% |
The answer surprised us: on the fixed contexts, a handful of examples adds nothing at all. Three irrelevant examples leave every model within noise of its no-example baseline, and three hand-picked relevant ones don’t do much better. The lift only appears with the full two-dozen example library, which takes Haiku from 70% to 82% and Sonnet from 81% to 93%.
Before the context consistency fixes this check had told a different story, with three relevant examples recovering about half of the library’s value; much of what those examples were “teaching” was conventions (which reading of a window, how to label a group) that the rules now pin down. There is also an aside here for benchmark builders: under exact-match grading, even irrelevant examples had looked six to seven points helpful to the smaller models, and that entire effect was formatting.
Our first suspect was the selection. Maybe our similarity heuristic just picks the wrong three, and a better retriever would close the gap. We checked which example each successful library run actually imitates, and indeed our heuristic’s top three only contains it about half the time. So we rebuilt the three-example arm twice: once selecting by question-text similarity (what a production retriever would see), and once selecting by token overlap with the reference answer itself, a cheating selector that provably includes the example the models imitate. Neither moved the needle: Haiku 72% and 73% against 82% with the library, Sonnet 88% and 85% against 93%. Having the right example in a three-example context is not the same as having it in a two-dozen-example context.
So we read the failing queries. Almost the entire gap between three relevant examples and the full library comes from 7 of the 99 question × model cells of the hard tier, and most of them fail the same way: the model writes a query with the right logic and misses an unstated convention. On theLook, “which traffic source acquired the most customers in each state” fails for both Haiku and Sonnet in every three-example arm, including the cheating one, and passes with the full library. The ranking query is correct; three sparsely populated states have several sources tied at one customer each, and the reference answer breaks ties alphabetically. The one example that demonstrates that tie-break is a top-brand-per-category ranking that shares no tables with the question, so no similarity score will ever surface it. Another failing query returns store IDs where the reference returns store names. A third uses the live current date instead of the pinned snapshot date: the fact is stated in the rules, but the three selected examples happened to be the ones that never need a date anchor, while the wider library repeats the snapshot literal in every windowed query. The remaining cells are ordinary small-model slips, like returning one row per customer where a single count was asked.
So what happens is that the question is open to interpretation, and the grader is enforcing one possible answer, while several could be valid. The full example library added some information that the smaller models could use to determine the preferred interpretation for ambiguous questions. This is also where the authorship disclosure from the setup comes due. Opus passes the same question in every case, partly because it wrote the reference answer, so it is graded against its own defaults while the smaller models have to guess them.
This is a kind of completeness issue again, but it’s the question that’s incomplete or underspecified rather than the context this time. This highlights the difficulty of writing eval questions: any ambiguity will cause some flakiness.
In a side experiment, we restated each missing convention in the question text itself, like “if several sources tie, report the alphabetically first one”, and re-ran the affected questions. Most of the gap closed: Sonnet’s three-example arm rose to its full-library level, Haiku recovered most of its deficit, and even the no-example baseline jumped about twenty points. None of this changed any conclusion of this article, so the numbers and charts in this post keep the original questions and answer key.
So the example library contributes in two ways. The first is coverage: with two dozen examples, nearly every question finds a structural cousin to adapt. The second is disambiguation: when a question leaves a choice open, for instance on using names or IDs to identify items, the examples shows which is most commonly used, and the models follow that. And real user questions are at least as underspecified as our benchmark’s were, so the disambiguation job is not an artifact of our eval. It is going to help in production as well. The only thing to be mindful of is that these rules conveyed by the examples exist only implicitely, so consistency will be hard to enforce.
This result aligns with research on many-shot in-context learning, which found that expanding from a few demonstrations to many can improve performance across a range of tasks, although the gains vary by model.
Where the noise lives #
We ran every cell three times, which lets us ask a separate question: when a model gets one of these right, is it reliable, or did it get lucky? The answer is reassuring and useful. Across all 828 question-by-context-by-model cells, the three runs agree completely 93% of the time. Either all three fail or all three succeed.
The 7% that disagree are not spread evenly. They pile up in the middle of the ladder (13% of cells are flaky at the descriptions level, 8% at metrics) and thin out at both ends (3% at bare schema, 3% with the example library).
At bare schema almost everything reliably fails. At the top of the ladder, with a library to copy from, almost everything reliably succeeds. Before the convention fixes, the recurring flaky cells sat on questions with two defensible readings, “customers in the last 90 days” with an open or closed boundary, top-five with or without ties: the model alternated between two reasonable interpretations, so the variance lived in the question, not the model. Pinning those conventions is part of what thinned the band. What remains sits where a model has all the facts but no worked pattern to anchor on.
Accounting for the cost #
So far we’ve looked only at accuracy. As we’re seeing more and more folks getting burned by token costs, it would be foolish not to take that into account.
A query’s output here is a short SQL statement, a couple hundred tokens, and thinking stayed near zero across the whole run, so spend is dominated by the context you send in. More spelled-out context is not free. It means more input tokens on every single call. On the hard tier, a query costs between 1,000 and 2,000 input tokens with a bare schema and between 7,000 and 12,000 with the example library. The cost scales with it. The graph below shows the uncached cost of each model and context combination at current API prices.
Adding examples to a cheap model buys accuracy at a fraction of the price of a model upgrade. With descriptions plus examples, Haiku reaches 81% at about $0.0057 per query, matching Sonnet with metrics alone at roughly a third of the cost. The full stack buys Haiku one more point for $0.0080. Sonnet’s lean example context reaches 89% at $0.0172, versus 93% at $0.0242 with the full stack.
The strongest model changes the top of the chart. Opus reaches 98% on nothing but terse descriptions at $0.029 per query. Its descriptions-plus-examples context also scores 98%, at $0.047, and the full stack reaches 99% at $0.065. Once the context is consistent, the strong model does not need it spelled out.
Another thing to keep in mind is that this test covers only a single question and answer. Real analytics work drills down into causes over a thread. In a study of 15 models and six tasks, including text-to-SQL, performance dropped 39% on average from single-turn to multi-turn conversations. The study found high multi-turn unreliability across capability levels, so while we could assume the single-turn ranking here will carry over to a longer conversation, it would require a dedicated evaluation.
The hidden costs #
It’s tempting to read these results as “add examples,” and for the smaller models, this finding holds across the checks we ran.
However, we haven’t accounted for the cost of keeping everything consistent, which proved difficult even in this small benchmark.
It’s easy to write down a set of rules that are independent from one another. It’s much harder with a set of examples: joins, metric computations, and query patterns will typically be reused across many examples. Examples will repeat information already present in other examples, and they may also rephrase information already stated in the surrounding rules. Information duplication creates a significant risk of inconsistencies whenever the context is edited. Nothing kills performance quicker than an inconsistent context. The wrong turn above shows exactly what that looks like, and it only took one contradictory sentence.
The ablation suggests one mitigation: don’t keep every fact in every representation by default. A complete description layer plus examples removed a large block of duplicated rules and metrics without changing standard accuracy. It does not eliminate the maintenance problem because the examples still repeat conventions among themselves. But it reduces the number of layers that can disagree.
So which one do you choose? #
Long story short: completeness is a constraint, acceptable performance is the target, and total cost is what you minimize. Once completeness is secured, choose the cheapest representation that delivers acceptable performance.
“Cheapest” here is broader than API spend. It includes context tokens, model cost, the work required to keep the context current, and the risk created by duplicated or contradictory information.
As always, the answer depends on the use case.
- What accuracy and reliability does the use case require? “Best possible” and “good enough” lead to different choices.
- Which combinations of model and context representation reach that target on your actual questions? Before stacking descriptions, rules, metrics, and examples together, test whether a leaner representation already gets there.
- How often will the context evolve? A representation that is cheap to run may be expensive to maintain, especially when the same fact is duplicated across rules, metrics, and examples.
For a given target performance, the main tradeoff is often between maintenance cost and model cost: a cheaper model may require a richer context, while a stronger model may derive the same answer from a leaner one.
In this benchmark, strong models reached acceptable performance from lean descriptions. Smaller models benefited from a broad example library, but usually did not need the separate rule and metric layers beneath it. That is not a universal recipe; it is the cheapest configuration that met the target here. For data analytics where wrong answers have a real business cost and the context evolves frequently, the largest models may still make more sense. They provide great results from descriptions alone, lowering the risk of inconsistencies and making context maintenance much easier.