# Evaluate the Decision, Not the Prose: Building an Eval Harness for an LLM Backend

> Source: <https://dev.to/paulmurithi/evaluate-the-decision-not-the-prose-building-an-eval-harness-for-an-llm-backend-22gd>
> Published: 2026-11-16 07:41:17+00:00

*SokoFlow AI Build Log, Week 5 of 6: building an eval harness, getting rate-limited by my own test, and what a 90% really means*

I built an exam for SokoFlow's AI layer, ran it, and scored 71%. The target was 85%.

So I did what you're supposed to do: I read the failures before touching any prompt. And some of the failures weren't the model's fault. The answer key expected a product UUID for stock questions, while the app looks products up by *name*. A shopkeeper says "sugar", not `product_id=42`. Part of my exam was wrong.

That's the week in one paragraph: an evaluation harness is software, and it has bugs too.

**TL;DR**

- Week 5 built an eval harness for the
*intelligence layer* only: the LLM, the Orchestrator and the executor's validation. Not tool execution, not the database.- The design calls: a case is a whole
*interaction*, not one message; non-execution is judged on the decision, not the prose; tool selection, argument accuracy and overall pass rate are scored separately; each case carries labels on several independent dimensions instead of one category.- First clean run:
**71.11%**. After fixing the answer key, the tool descriptions and the date handling: **90% (27/30) on three runs in a row**, at about $0.000069 per query against a $0.004 ceiling.- The caveats matter as much as the number: I tuned against the same 30 turns, the repair path never fired, and the margin over the 85% target is one turn.

SokoFlow is a headless WhatsApp ERP for Kenyan SMEs. I've been building it for about five months. There's no app and no UI to learn; for a shopkeeper, the conversation is the interface. Phase 3 adds one AI feature: natural-language shop analytics. A shopkeeper asks "which products are running low?" or "how does this week compare to last week?", and an LLM, using function calling against SokoFlow's existing services, figures out which query to run.

Last week was robustness and guardrails, driven by one question: *what happens when the system can't safely give the user what they asked for?* Weeks 1–4 were mostly about making the system behave safely and predictably. Week 5 is about whether the intelligence layer actually behaves *correctly*. The principle for the week:

**A system that executes tools successfully is not necessarily a system that understands what the merchant asked for.**

The spec set an eval pass rate of at least 85%. My first question: what exactly are we proving when we say that?

Imagine a merchant sends: *"How much did I make in sales last week?"* The AI should propose `get_sales_summary` with the correct date range for last week. But a lot happens on the way: the LLM interprets the question, the `ToolExecutor` validates the proposal, the backend runs the tool, and the formatter writes the reply.

Do I have to test all of that? No. This eval measures the *intelligence layer*, not tool execution or database uptime. That was the first boundary I drew:

```
EVALUATION ENTRY
  Merchant query + trusted evaluation context
        ↓
  LLMClient          interpret merchant intent, propose a tool call
        ↓
  Orchestrator       tool selection, argument validation flow, bounded repair
        ↓
EVALUATION EXIT
  ToolExecutor validation
  "Does the final proposal resolve to an allowed tool
   and satisfy its argument contract?"

OUTSIDE PRIMARY EVALUATION
  Tool execution → Services → Database → Formatting → WhatsApp delivery
```

Next, the unit of measurement. SokoFlow handles multi-turn conversations, which made this hard. Consider:

Are those two independent cases, or is the second a continuation of the first? I chose to treat a case as **a complete interaction with whatever conversational context is needed to interpret it.** One interaction may contain several merchant messages, and each message is an evaluation *turn* with its own expected behaviour.

The reason: if I evaluate message 2 on its own, I'm testing something different from production. In production the system has to understand that the merchant is continuing an analytics conversation, that the previous successful call provides context, that "yesterday" modifies the previous request, and that the new arguments must reflect yesterday, not today. That's a legitimate part of the intelligence layer.

A pass for a tool-calling case is simple: the selected tool and arguments match the expected ones. But SokoFlow also has a *non-execution* path. Imagine a merchant asks "What is the weather today?" That isn't something SokoFlow exposes, so the correct outcome is **no tool call**.

There's a catch. The model could return no tool call *and* an empty or malformed response. Or it could return no tool call *and* a sensible explanation that weather isn't supported. Should both get credit just because neither called a tool?

My first idea was to use the `raw_response` field on the `LLMResponse` schema to tell those two apart. I got skeptical fast. That field holds the raw provider response, for debugging and logging. Making evaluation depend on it would tie the eval to provider-specific structures and undermine the provider-independent `LLMClient` contract from Week 2. This is one more place where measuring AI quality is harder than checking whether a function returns the expected value.

For the weather example, expected behaviour is "no analytics tool call", and there are four possible outcomes:

| Model behaviour | Interpretation | 
|---|---|
| No tool + relevant explanation | Correct | 
| No tool + generic but meaningful explanation | Potentially correct | 
| No tool + empty or malformed response | Incorrect | 
| Analytics tool call | Incorrect | 

The hard part is deciding whether an explanation is *relevant*. One fix is a semantic classifier to judge arbitrary natural-language replies, but that could become a project of its own. So the decision was simple: **evaluate the decision, not the prose.** For expected non-execution cases, a pass needs:

No semantic classifier. `raw_response` stays diagnostic material, not an application-level dependency.

I also saw a chance to get diagnostic data, not just a score. Suppose the expected tool is `get_sales_summary`, and the model picks `get_sales_summary` but supplies the wrong date range. Is that a successful tool selection with failed argument generation, or a complete failure? If I collapse it into one number, I can't tell *where* the model goes wrong.

So there are three metrics: **tool-selection accuracy**, **argument accuracy** and **overall pass rate**. The week's KPI is the last one:

```
PassRate = FullyPassedCases / TotalEvaluationCases × 100
```

SokoFlow has a bounded repair: at most one extra model attempt for specific errors (the retry rules from Weeks 1 and 4). What if the model proposes invalid arguments, then returns the right tool and arguments after the Orchestrator allows one repair? Is that a pass?

Yes. I want to evaluate the system *as designed*, and repair is part of the production architecture. If it recovers from an invalid first proposal within its limit, the system behaved correctly. But for diagnostics I record **first-attempt success** and **repair-recovered success** separately, so I can see how often the model needs the safety net.

I already had one source of cases: the `InteractionLogger` from Week 4. But I needed 25–40, so I needed more. And that raised a question: what makes a dataset *independent*?

Imagine asking an LLM to generate 30 merchant queries with expected tool calls, then evaluating the *same model* on them. The same model is author, judge and jury, and the score gets inflated. So I settled on a hybrid:

`InteractionLogger`.
A representative dataset isn't one with many different questions. It's one that exposes the important ways the system can succeed or fail in realistic use. I focused on three things:

Balancing those guards against an eval that gives reassuring scores while leaving big parts untested. A 100% pass rate doesn't necessarily mean a good system. It might mean I wrote an easy exam.

Representative coverage doesn't mean copying real traffic proportions. Suppose (hypothetically) 70% of real queries were sales summaries. I wouldn't give sales summaries 70% of the eval. This is an engineering evaluation: enough cases for common usage, and deliberate space for rarer capabilities and failure modes.

This was the hardest concept of the week. I wanted to classify each case along three dimensions:

**What I got wrong first:** I treated these as mutually exclusive *categories*, so each case went in one box. My reason was interpretability: when I inspect results, I should see which category is doing badly. The goal was right; the design was wrong.

Here's the mental model that fixed it. Imagine a database of people and the question "who in our company is struggling?" You wouldn't ask for each person *either* their age *or* their city *or* their job title. A person is 30 **and** lives in Nairobi **and** is an engineer. Those aren't competing choices, they're different questions about the same person.

A test case is the same. It has independent properties at once:

If I filed "How much did I make this week?" under *only* "Sales summary", then when failures spike I could never ask "are the failures concentrated in relative-date questions?", because I'd have thrown that label away when I filed it. Filing each case under one box lets me slice results one way and loses information. Giving each case **a value in every column** lets me slice any way I want.

The lesson: **a classification dimension tells you where to look. The intersection of dimensions helps you form hypotheses about why something is failing.**

The seven-tool registry is the functional scope. Not every tool deserves identical representation, but every registered tool deserves coverage. Production traffic would have let me weight tools by real merchant usage, but I don't have it, so I used deliberate engineering coverage instead of pretending to know the distribution. Three layers:

| Category | Cases | 
|---|---|
| `get_sales_summary` | 8 | 
| `get_top_products` | 4 | 
| `get_product_price` | 4 | 
| `get_stock_level` | 3 | 
| `get_low_stock_items` | 2 | 
| `get_sales_trend` | 2 | 
| `get_slow_moving_items` | 2 | 
| Non-execution (no tool) | 5 | 
| **Total turns** | **30** | 

Consider three merchant messages:

All three are about sales, but they make different demands. The first is straightforward, the second needs a relative time expression interpreted, and the third needs a comparison and its two periods. How do I represent that without turning the dataset into artificial grammar exercises? I used one filter for every candidate:

**"Would a real merchant plausibly type this, and would getting it wrong hurt them?"**

If yes, it's a candidate. If it's just a clever way to stress a parser, I left it out for this MVP.

For interaction structure, I considered a conversation like this:

```
Merchant:  How much did I sell yesterday?            (turn 1)
SokoFlow:  Your sales yesterday were KSh 12,500.     (previous system response)
Merchant:  What about the day before?                (turn 2)
Merchant:  And this week?                            (turn 3)
```

Turns 2 and 3 depend on context. The risk is building multi-turn cases that accidentally hand the model information it wouldn't have in production. The solution: construct every turn so the model sees exactly what production would show it at that moment. Raw messages, raw (even wrong) prior answers, production-shaped state, and never a cleaned-up or repaired transcript.

Now I moved from "what is a correct answer" to "why should anyone trust this evaluation?" The core question: what makes it *reproducible*, so that a run today and a run tomorrow, with nothing changed, can be compared meaningfully? I listed what needs to stay constant:

| Dimension | What needs control | 
|---|---|
| Dataset | Exact evaluation cases and versions | 
| Expected outcomes | A frozen, independently derived answer key | 
| Model | Provider, model identifier and version where available | 
| Prompt | Exact system and developer instructions, and their versions | 
| Tools | Registry definitions, schemas and descriptions | 
| Orchestration | Repair policy, retry limits and execution configuration | 
| Runtime | Relevant application and dependency versions | 
| Measurement | Timing boundaries, cost calculation and aggregation rules | 

Model APIs can also be nondeterministic. Even with identical inputs and configuration I accept that outputs may differ, so reproducibility doesn't mean *identical outputs*. It means being able to reconstruct the experiment, measure the variation and explain what happened.

Another risk: the provider silently updating the model between runs. If that happens, a score shift might come from the provider rather than my prompts or code. So model identity is part of the evaluation baseline. To maximize predictability, temperature was set to 0.0.

I want to be able to open an old report six months from now, see that the pass rate fell from 90% to 75%, and reconstruct enough to investigate. So I record granular results, not just a total. A 75% tells you something went wrong, not *what*. **An evaluation report should be an evidence trail, not merely a scoreboard.** It covers three groups:

I added three measurements: mean pass rate (average observed performance), minimum and maximum (best and worst run), and standard deviation (how widely results fluctuate). I also track **per-turn failure frequency**, which separates persistent weaknesses from intermittent ones. A turn that fails three times in a row deserves different attention from one that fails once and passes twice.

I separated user-perceived latency from component-level latency. For a shopkeeper, the total time across *both* inference attempts (first and repair) is what counts. For an engineer debugging, the initial model call matters too. And for elapsed time I use a monotonic clock, because it only moves forward at a steady rate and isn't affected by system time adjustments.

Another Week 5 target was cost per query at expected volume. First, the cost of one evaluation turn. Three options:

I chose 2 and 3. Option 1 only measures the happy path and hides the cost of a prompt that failed validation and had to be repaired.

```
C_turn    = sum over i = 1..n of C_inference_attempt(i)

C_average = sum of all C_turn / N_evaluation_turns
```

The denominator includes turns that needed no repair. Repair is part of the system's real operating cost, not an optional expense to hide.

What if token usage is unavailable or unreliable? I record where each number came from:

| Classification | Meaning | 
|---|---|
| Provider-reported | Usage returned by the provider | 
| Locally estimated | Calculated from request and response data with a compatible tokenizer | 
| Unavailable | Not enough reliable information to calculate usage | 

A locally estimated token count isn't automatically an accurate billing estimate. Several articles I read note that different tokenizers count the same text differently.

Now we leave the dataset and think commercially. The trap I nearly fell into: picking one number and treating it as fact. I first went with 100 queries a day:

```
100 queries × $0.0001 × 30 days = $0.30 per shop per month
```

That's the temptation: turning an arbitrary assumption into an engineering requirement. A better approach is **several usage scenarios, grounded in real product behaviour where possible:**

| Scenario | Interpretation | 
|---|---|
| Conservative | Lower expected usage among active shops | 
| Expected | Most plausible usage based on available evidence | 
| Power-user | High usage from particularly active shops | 

I don't have production telemetry yet, so the inputs are intended merchant workflows, explicit usage assumptions, and (later) controlled pilot observations. Once real merchants use analytics, I'll replace assumptions with observed data: queries per shop per day, active shops, cost per query, repair frequency. Monthly model spend:

```
C = S × Q × D × c̄

S  = active shops
Q  = daily queries per active shop
D  = operating days
c̄  = average cost per logical query
```

How do I set a defensible maximum cost per query? I borrowed from unit economics: expected revenue per shop, target gross margin, other service-delivery costs, merchant willingness to pay, and an explicit analytics budget.

But an *average* cost per query isn't enough for operational protection. That's where Week 4's limits come in, and each control has a different job:

This led me down a rabbit hole that mirrors the classic machine learning train/validation/test split. Suppose I run the headline evaluation and get 80%. Then I try different temperatures, prompt improvements, model changes and repair settings, and eventually find a configuration scoring 93%.

The danger: if I keep using the frozen headline dataset to *pick* the best configuration, I'm gradually optimizing against it. That 93% might measure how well I've learned those particular 30 cases, not how SokoFlow handles unseen merchant requests. This is **evaluation-set contamination.**

The fix is to split the data into a **development set** and a **held-out test set**, and enforce a gatekeeper rule: the test set runs only after a candidate configuration has already been chosen.

With the harness built, it was time to find out how SokoFlow actually performed. My target was an overall pass rate of at least 85%. The first runs reminded me that building an evaluation and *running* one reliably are different problems.

It was Groq's rate limiting. I started getting HTTP `429 Too Many Requests`. The evaluation was sending requests faster than my token budget allowed, and the retries weren't robust enough. Rather than treating it as a model-performance problem, I stepped back and made the evaluation infrastructure reliable:

`max_tokens`, to avoid exceeding the available budget.` TokenBucket`.` LLMClient` instance creates a shared token bucket, so requests through that client respect one common rate limit instead of each consuming capacity independently.`_post_payload` now detects 429s. It first reads the `Retry-After` header. If that's missing or invalid, it falls back to exponential backoff starting at `evaluation_initial_backoff` (1 second), doubling each attempt and adding 10% jitter. Retries are capped by `evaluation_max_retries` (default 3), and each one is logged with its attempt number and wait time.
I wasn't adding retries to make errors disappear. I wanted the client to respect the provider's limits and behave predictably when it hits them. This 429 retry is for evaluation only. But in future, I will be considering moving this feature to production too. Automatic retries for provider transport failures.

With the 429s handled, the harness completed its runs, but the results weren't what I'd hoped. SokoFlow scored around **71%**, below the 85% target:

```
================================================================
 EVALUATION RESULTS SUMMARY
================================================================
Mean Pass Rate       : 71.11% (Target: ≥ 85.0%)
Pass Rate Min / Max  : 70.00% / 73.33%
Standard Deviation   : 1.92%
KPI Status           : [NOT MET]

Average Cost / Query : $0.000051 (Ceiling: $0.0040)
Budget Ceiling       : [MET]

Tool Selection Acc.  : 76.00%
Argument Acc.        : 64.00%
Non-Execution Acc.   : 100.00%

Mean Turn Latency    : 15420.2 ms
P95 Latency          : 17871.4 ms
----------------------------------------------------------------
```

The infrastructure was now reliable, but the tool-calling behaviour clearly needed work. I went through the failed interactions looking for patterns, instead of randomly changing prompts. What I found:

I corrected the relevant expectations, clarified the tool descriptions and prompts, aligned the date-handling logic and scoring, and added focused tests around the retry and error-handling paths. I also fixed a missing import in the rate-limit retry path and improved its jitter behaviour.

Then came another set of live runs. Across **three runs of 30 evaluation turns each, SokoFlow scored a 90% mean pass rate, with every run exactly 90% (27 out of 30):**

```
================================================================
 EVALUATION RESULTS SUMMARY
================================================================
Mean Pass Rate       : 90.00% (Target: ≥ 85.0%)
Pass Rate Min / Max  : 90.00% / 90.00%
Standard Deviation   : 0.00%
KPI Status           : [ACHIEVED]

Average Cost / Query : $0.000069 (Ceiling: $0.0040)
Budget Ceiling       : [MET]

Tool Selection Acc.  : 100.00%
Argument Acc.        : 92.00%
Non-Execution Acc.   : 80.00%

Mean Turn Latency    : 19553.8 ms
P95 Latency          : 21712.0 ms
================================================================
```

That's a real improvement from the 71.11% I recorded in v1.0.0 (the set of test queries, expected tools/arguments, and scoring expectations used for the run. If the dataset’s contents change, its version should be bumped. The hash can also reveal content changes even if someone forgets to update the version). and it was consistent across all three runs.

Average cost was within the $0.004 ceiling.

Three interactions failed in *every* run. Two were mismatches involving unexpected arguments (`ranking_dimension` and `comparison_period`), and one was an empty model response on a chitchat message. I documented them instead of pretending the system was perfect.

A few honest limits on the 90%:

`LLMClient.generate`, including local token-bucket waits and any HTTP 429 retry/backoff sleeps. Week 3 measured a different path: from conversation entry through to the formatted reply, whereas the evaluation harness uses a direct orchestrator path with a mock database. The measurements were also taken nearly a month apart, so differences in provider conditions may have contributed.
I investigated whether local rate limiting explained the increase. Although token-bucket waits are included in the evaluation timing, the harness runs requests serially, leaving time for the bucket to refill between calls. This makes repeated long waits from the local token bucket an unlikely explanation for the entire gap. I therefore treat the two figures as measurements from different execution paths, not as a direct performance regression comparison. The higher evaluation latency remains something to investigate separately.

What started as a run blocked by rate limits turned into a useful exercise in debugging infrastructure, reading model behaviour, tightening tool contracts and improving the evaluation itself.

Three things, from this week's own mistakes:

Week 6 is the final stretch. Everything from the last six weeks has to come together into a finished, demonstrable project. The focus: a final pass over chaos and edge cases, bringing the build log up to date, recording a Loom demo, writing the Phase 3 blog post, and tagging the release.

The finish line:

`main`.
Six weeks of building, debugging, testing and refining, and now it's time to bring it together. Stay locked in for the next build log.
