Evaluating Your Agentic Harnesses

A developer's evaluation of an agentic harness built around an LLM shows a 75% pass rate, averaging 1,088 tokens and $0.0033 per task across four tasks, including one adversarial task. The harness, now packaged as d4sci_harness.py, supports both offline mock and Anthropic backends, and tracks metrics such as pass rate, token usage, status codes, and replan count.

In previous posts , we borrowed Col. John Boyd’s OODA loop, built a “figher jet” harness around an LLM and even scaled from a single pilot to an air campaign with mission planners, squadrons flying independent sorties in parallel, fuel budgets, and a flight recorder. Now the pilot has returned to base and is time for a debrief. No Air Force declares an aircraft combat-ready without it going through operational test and evaluation: flying against representative missions including missions designed to break it record everything, and compare the numbers against the requirements. Production level agents deserve nothing less, but almost never get it. Successful demos prove nothing more than the harness can work, but it takes an eval suite to demostrate that it usually works while quantifying the consequences of when it doesn’t. We moved the entire harness including typed tools, the plan DAG, the parallel executor, multi-tier memory, the verification hierarchy, multi-dimensional budgets, structured tracing, and the Orchestrator into a standalone module, d4sci harness.py so that you can import it instead of rebuilding it: import d4sci harness as dh from d4sci harness import TOOLS, MemoryStore, Orchestrator llm = dh.set provider "anthropic" or "mock" for offline runs store = MemoryStore orch = Orchestrator provider=llm, tools=TOOLS, memory=store The harness supports both a fully offline and deterministic mock backend, perfect for CI, and and an anthropic backend that gives you realistic planner, summarizer, and critic behavior. Switching backends is a one-line change. The rest of this post is based on live Claude runs. Bare bones eval harness Even a minimalistic eval unit should track a few useful metrics: Pass rate: Functional correctness across tasks Token usage: Cost predictability Status codes: Failure mode distribution failed execute vs failed verify vs budget Replan count : Robustness to bad inputs The suite itself is a mix of correct and adversarial tasks aimed at exploring all code paths. In this example, we use three well-posed requests, and one that we expect to fail so that we can measure the recovery path: EVAL SUITE = {"goal": "Build a comparison report of Paris, Tokyo, and New York.", "required cities": "paris", "tokyo", "new york" }, {"goal": "Build a comparison report of London and Sydney.", "required cities": "london", "sydney" }, {"goal": "Build a comparison report of Paris and London.", "required cities": "paris", "london" }, Atlantis is not in CITY FACTS — this task exercises the re-planning path, so we only require the satisfiable city {"goal": "Build a comparison report of Paris and Atlantis.", "required cities": "paris" }, We loop through each task and extract the metrics out of the RunResult object our orchestrator returns: async def run eval orch, suite : rows, runs = , for task in suite: res = await orch.run task "goal" , task "required cities" runs.append res rows.append { "passed": bool res.verdict and res.verdict.passed , "status": res.status, "replans": res.replan count, "tokens": res.budget.tokens used, "cost": res.budget.cost usd, "tier": res.verdict.tier if res.verdict else "n/a", } return rows, runs And visualize the results in a simple table: An overall pass rate of 75% and an average of 1,088 tokens and $0.0033 per task. Doesn’t seem too bad. We expect a pass rate below 100% as one task is expected to fail, and update the status column with the cause of the failure failed verify vs failed execute . Different problems require different solutions. The ill fated Atlantis task triggered one replan, as expected, confirming the recovery path was triggered correctly. Paris is the only required city in the Atlantis task and is present in the report but the LLM judge , the last rung of the verification hierarchy, refused to call a one-city document a “comparison report of Paris and Atlantis”. This is the kind of nuance that LLMs excel at and that we catch because the eval records the tier that produced the verdict. This suite is intentionally minimal but we architectured in a way that makes it flexible enough to be easily expanded. A closer look at the flight recorder The RunResult objects that the eval loop collects contain all the information we need, making it easy to provide useful visualizations. We start by looking at the API costs. The Atlantis run is the most expensive of the four. It requires two rounds of planning and every tool call it makes along the way. Failure has a price tag Next, we look at the run time of the various tasks. The stacked segments sum the total work each run performed. The diamond marks actual wall-clock time. The gap between the bars and the diamond is the payoff we get for the parallelism we baked in from the ground up. Worker segments dominate across all tasks as calling tools and processing their output takes the most time. The various roles also required different amounts of tokens. Re-planning shows up as a wider planner segment; a report-heavy task shows up as a wider worker segment. Any change to the harness will directly impact these numbers, so we should be on the lookout for unexpected changes. Finally, we analyze the budget pressure. Whenever it crosses 0.9, the orchestrator skips the LLM judge and falls back to the free deterministic check. All four trajectories stay comfortably low on this toy workload but a production dashboard should alert you when a task class starts drifting upward. Closing the gaps Logging performance metrics from day one not only allows us detect when something goes wrong, but it also pays dividends when it’s time to make changes to our system. In the remainder of this post, we close three of the gaps that separate a toy harness from production systems, each with minimal changes to the architecture. Each enhancement includes its own mini-eval so that we have a clear picture of the impact each change has. Re-planning on failure In our eval suite, when the user asks for a report including Atlantis, the population lookup raises KeyError: Unknown city . Let us remember what happens next: The worker marks the node FAILED. classify error returns MISSING INFO . The orchestrator calls planner.replan goal, failed dag with the failure context. The planner emits a new DAG excluding Atlantis. The worker executes the amended plan. We pass the failure context to the planner to avoid getting stuck in a loop doing the same error over and over again. In this way the LLM has all the information it needs to reason about alternatives: skip the city, substitute a known one, or fall back to a different tool. Here’s the real trace: Status: failed verify replanned 1x Replan count: 1 Tokens used: 1802 Tool calls: 16 Planning attempts: 2 Attempt 1: initial plan → 10 nodes Attempt 2: replan → 7 nodes The first plan had 10 nodes, Atlantis included. After the fetch failed, the amended plan shrank to 7 nodes covering only the cities that exist, so we can still deliver value instead of simply aborting on bad input. With Atlantis gone, the final report node depends only on the six nodes that can actually succeed: The per-step trace tells the whole story in one picture: every event on its own row, colored by role, and ordered chronologically. The second planner row in blue naturally separates the two plans. Everything above it belongs to the original failed plan, everything below to the amended one. Real embeddings Our memory store had been retrieving with Jaccard similarity that simply measures word overlap. Jaccard fails exactly when it would be the most useful, when the query and the memory use different vocabulary. For example, a query “Tell me about famous landmarks in France” would never match “Paris is known for the Eiffel Tower” as they have a Jaccard score close to zero due to not having any shared words beyond stopwords . Embeddings are a different way of representing text. Embedding models map text to dense vectors that encode the semantic meaning of the text: snippets that mean the same thing will have similar embedding vectors. The similarity between two snippets can then be estimated using the cosine of the two embedding vectors . We use the all-MiniLM-L6-v2 embedding model to generate our embeddings. This model produces good quality embeddings vectors with 384 dimensions and is small enough to run locally. The cost for encoding the knowledge base is paid only once on add and is amortized over time. At query time we only need to encode the query and compute the cosine similarity at query time, two relatively cheap operations, and if anything fails, we can fallback to Jaccard. Let’s compare identical queries against both version, each with the memory its ideal answer should surface: Query: Tell me about famous landmarks in France Jaccard ✗ Tokyo has the busiest train stations in the w | New York is famous for Times Square and Broad Embeddings ✓ Paris is known for the Eiffel Tower and Frenc | London's Big Ben is an iconic landmark. Query: What happened when comparing cities in Europe? Jaccard ✗ Task comparing Asian cities succeeded with al | Tokyo has the busiest train stations in the w Embeddings ✓ Task comparing Asian cities succeeded with al | Previously compared European cities and found Query: Information about Japanese cities Jaccard ✗ Task comparing Asian cities succeeded with al | Previously compared European cities and found Embeddings ✓ Tokyo has the busiest train stations in the w | Task comparing Asian cities succeeded with al The results are clear: 3-nil for Embeddings. We visualize the similarity of every query against every memory, using both similarity functions. The best score per row is highlighted in bold: Jaccard scores are near zero everywhere as simple word overlap can’t handle paraphrasing. On the other hand, the cosine similarity of embeddings ranks the correct memory first in every row. Specialized workers Different tools will have different requirements, failure modes, and contraints: Production systems should differentiate between them and route based on capability: a FetcherAgent for fast, idempotent lookups where higher concurrency is fine, and a WriterAgent for LLM-backed tools that need lower concurrency and their own retry policy. A unified WorkerAgent exposes the same execute dag interface and routes internally without having to change the orchestrator at all. Our compositional architecture payoff once again Let’s try routing a mixed seven-node DAG through the unified worker. Each bar is colored based on the Agent used to handle it: The timeline shows why the split matters operationally: fetch nodes finish a thousand times faster than writer nodes. The writers own the critical path and are worth giving their own semaphore, retry policy, and monitoring tags. We could further extend this approach along the same line: a BrowserAgent or CodeAgent , etc. each with its own tool subset and trace tags for logging. Whats next Across these three posts we took a ~35-line loop and layered on production-shaped primitives: Each of these pieces is optional, but they all compound on each other. The planner automatically sees the schema of any new tools that are added. Tighten verification, and the eval suite catches regressions. Swap the LLM backend, and the entire harness can be rerun with a one-line change. This composability and compounding is what makes the difference in production. Yet, much is still missing. We have no persistent memory, defenses against prompt-injection, human in the loop capabilities, etc. We’ll explore these in future posts, but the winning approach remains the same: Build small primitives, compose them, measure the result. That’s the whole doctrine.