{"slug": "evaluating-your-agentic-harnesses", "title": "Evaluating Your Agentic Harnesses", "summary": "A developer's evaluation of an agentic harness built around an LLM shows a 75% pass rate, averaging 1,088 tokens and $0.0033 per task across four tasks, including one adversarial task. The harness, now packaged as d4sci_harness.py, supports both offline mock and Anthropic backends, and tracks metrics such as pass rate, token usage, status codes, and replan count.", "body_md": "In \nprevious\n \nposts\n, we borrowed Col. John Boyd’s OODA loop, built a “figher jet” harness around an LLM and even scaled from a single pilot to an air campaign with mission planners, squadrons flying independent sorties in parallel, fuel budgets, and a flight recorder. Now the pilot has returned to base and is time for a debrief.\n\nNo Air Force declares an aircraft combat-ready without it going through operational test and evaluation: flying against representative missions (including missions designed to break it) record everything, and compare the numbers against the requirements. Production level agents deserve nothing less, but almost never get it. Successful demos prove nothing more than the harness \ncan\n work, but it takes an \neval suite\n to demostrate that it \nusually\n works while quantifying the consequences of when it doesn’t.\n\nWe moved the entire harness (including typed tools, the plan DAG, the parallel executor, multi-tier memory, the verification hierarchy, multi-dimensional budgets, structured tracing, and the \nOrchestrator\n ) into a standalone module, \nd4sci_harness.py\n so that you can import it instead of rebuilding it:\n\n```\nimport d4sci_harness as dh\nfrom d4sci_harness import TOOLS, MemoryStore, Orchestrator\n\nllm = dh.set_provider(\"anthropic\")   # or \"mock\" for offline runs\n\nstore = MemoryStore()\norch = Orchestrator(provider=llm, tools=TOOLS, memory=store)\n```\n\nThe harness supports both a fully offline and deterministic \nmock\n backend, perfect for CI, and and an \nanthropic\n backend that gives you realistic planner, summarizer, and critic behavior. Switching backends is a one-line change. The rest of this post is based on live Claude runs.\n\nBare bones eval harness\n\nEven a minimalistic eval unit should track a few useful metrics:\n\nPass rate:\n Functional correctness across tasks\n\nToken usage:\n Cost predictability\n\nStatus codes:\n Failure mode distribution (\nfailed_execute\n vs \nfailed_verify\n vs \nbudget\n)\n\nReplan count\n : Robustness to bad inputs\n\nThe suite itself is a mix of correct and adversarial tasks aimed at exploring all code paths. In this example, we use three well-posed requests, and one that we expect to fail so that we can measure the recovery path:\n\n```\nEVAL_SUITE = [\n    {\"goal\": \"Build a comparison report of Paris, Tokyo, and New York.\",\n     \"required_cities\": [\"paris\", \"tokyo\", \"new york\"]},\n    {\"goal\": \"Build a comparison report of London and Sydney.\",\n     \"required_cities\": [\"london\", \"sydney\"]},\n    {\"goal\": \"Build a comparison report of Paris and London.\",\n     \"required_cities\": [\"paris\", \"london\"]},\n\n    # Atlantis is not in CITY_FACTS — this task exercises the\n    # re-planning path, so we only require the satisfiable city\n    {\"goal\": \"Build a comparison report of Paris and Atlantis.\",\n     \"required_cities\": [\"paris\"]},\n]\n```\n\nWe loop through each task and extract the metrics out of the \nRunResult\n object our orchestrator returns:\n\n```\nasync def run_eval(orch, suite):\n    rows, runs = [], []\n    for task in suite:\n        res = await orch.run(task[\"goal\"], task[\"required_cities\"])\n        runs.append(res)\n        rows.append({\n            \"passed\":  bool(res.verdict and res.verdict.passed),\n            \"status\":  res.status,\n            \"replans\": res.replan_count,\n            \"tokens\":  res.budget.tokens_used,\n            \"cost\":    res.budget.cost_usd,\n            \"tier\":    res.verdict.tier if res.verdict else \"n/a\",\n        })\n    return rows, runs\n```\n\nAnd visualize the results in a simple table:\n\nAn overall pass rate of 75% and an average of 1,088 tokens and $0.0033 per task. Doesn’t seem too bad. We expect a pass rate below 100% as one task is expected to fail, and update the****\nstatus\n column with the cause of the failure (\nfailed_verify\n vs \nfailed_execute\n). Different problems require different solutions. The ill fated Atlantis task triggered one replan, as expected, confirming the recovery path was triggered correctly.\n\nParis is the only required city in the Atlantis task and is present in the report but the \nLLM judge\n , the last rung of the verification hierarchy, refused to call a one-city document a “comparison report of Paris and Atlantis”. This is the kind of nuance that LLMs excel at and that we catch because the eval records the tier that produced the verdict.\n\nThis suite is intentionally minimal but we architectured in a way that makes it flexible enough to be easily expanded.\n\nA closer look at the flight recorder\n\nThe \nRunResult\n objects that the eval loop collects contain all the information we need, making it easy to provide useful visualizations.\n\nWe start by looking at the API costs. The Atlantis run is the most expensive of the four. It requires two rounds of planning and every tool call it makes along the way. Failure has a price tag!\n\nNext, we look at the run time of the various tasks. The stacked segments sum the total work each run performed. The diamond marks actual wall-clock time.\n\nThe gap between the bars and the diamond is the payoff we get for the parallelism we baked in from the ground up. Worker segments dominate across all tasks as calling tools and processing their output takes the most time.\n\nThe various roles also required different amounts of tokens. Re-planning shows up as a wider planner segment; a report-heavy task shows up as a wider worker segment. Any change to the harness will directly impact these numbers, so we should be on the lookout for unexpected changes.\n\nFinally, we analyze the budget pressure. Whenever it crosses 0.9, the orchestrator skips the LLM judge and falls back to the free deterministic check. All four trajectories stay comfortably low on this toy workload but a production dashboard should alert you when a task class starts drifting upward.\n\nClosing the gaps\n\nLogging performance metrics from day one not only allows us detect when something goes wrong, but it also pays dividends when it’s time to make changes to our system. In the remainder of this post, we close three of the gaps that separate a toy harness from production systems, each with minimal changes to the architecture. Each enhancement includes its own mini-eval so that we have a clear picture of the impact each change has.\n\nRe-planning on failure\n\nIn our eval suite, when the user asks for a report including Atlantis, the population lookup raises \nKeyError: Unknown city\n. Let us remember what happens next:\n\nThe worker marks the node \nFAILED.\n\nclassify_error()\n returns \nMISSING_INFO\n.\n\nThe orchestrator calls \nplanner.replan(goal, failed_dag)\n with the failure context.\n\nThe planner emits a new DAG excluding Atlantis.\n\nThe worker executes the amended plan.\n\nWe pass the failure context to the planner to avoid getting stuck in a loop doing the same error over and over again. In this way the LLM has all the information it needs to reason about alternatives: skip the city, substitute a known one, or fall back to a different tool. Here’s the real trace:\n\n```\nStatus:        failed_verify_replanned_1x\nReplan count:  1\nTokens used:   1802\nTool calls:    16\nPlanning attempts: 2\n    Attempt 1: initial_plan → 10 nodes\n    Attempt 2: replan       →  7 nodes\n```\n\nThe first plan had 10 nodes, Atlantis included. After the fetch failed, the amended plan shrank to 7 nodes covering only the cities that exist, so we can still deliver value instead of simply aborting on bad input.\n\nWith Atlantis gone, the final report node depends only on the six nodes that can actually succeed:\n\nThe per-step trace tells the whole story in one picture: every event on its own row, colored by role, and ordered chronologically. The second planner row (in blue) naturally separates the two plans. Everything above it belongs to the original failed plan, everything below to the amended one.\n\nReal embeddings\n\nOur memory store had been retrieving with \nJaccard similarity\n that simply measures word overlap. Jaccard fails exactly when it would be the most useful, when the query and the memory use different vocabulary. For example, a query “Tell me about famous landmarks in France” would never match “Paris is known for the Eiffel Tower” as they have a Jaccard score close to zero due to not having any shared words (beyond stopwords).\n\nEmbeddings are a different way of representing text. Embedding models map text to dense vectors that encode the semantic meaning of the text: snippets that mean the same thing will have similar embedding vectors. The similarity between two snippets can then be estimated using the \ncosine of the two embedding vectors\n.\n\nWe use the \nall-MiniLM-L6-v2\n embedding model to generate our embeddings. This model produces good quality embeddings vectors with 384 dimensions and is small enough to run locally. The cost for encoding the knowledge base is paid only once on \nadd()\n and is amortized over time. At query time we only need to encode the query and compute the cosine similarity at query time, two relatively cheap operations, and if anything fails, we can fallback to Jaccard.\n\nLet’s compare identical queries against both version, each with the memory its ideal answer should surface:\n\n```\nQuery: Tell me about famous landmarks in France\n  Jaccard    ✗  Tokyo has the busiest train stations in the w | New York is famous for Times Square and Broad\n  Embeddings ✓  Paris is known for the Eiffel Tower and Frenc | London's Big Ben is an iconic landmark.\n\nQuery: What happened when comparing cities in Europe?\n  Jaccard    ✗  Task comparing Asian cities succeeded with al | Tokyo has the busiest train stations in the w\n  Embeddings ✓  Task comparing Asian cities succeeded with al | Previously compared European cities and found\n\nQuery: Information about Japanese cities\n  Jaccard    ✗  Task comparing Asian cities succeeded with al | Previously compared European cities and found\n  Embeddings ✓  Tokyo has the busiest train stations in the w | Task comparing Asian cities succeeded with al\n```\n\nThe results are clear: 3-nil for Embeddings. We visualize the similarity of every query against every memory, using both similarity functions. The best score per row is highlighted in bold:\n\nJaccard scores are near zero everywhere as simple word overlap can’t handle paraphrasing. On the other hand, the cosine similarity of embeddings ranks the correct memory first in every row.\n\nSpecialized workers\n\nDifferent tools will have different requirements, failure modes, and contraints:\n\nProduction systems should differentiate between them and route based on capability: a \nFetcherAgent\n for fast, idempotent lookups where higher concurrency is fine, and a \nWriterAgent\n for LLM-backed tools that need lower concurrency and their own retry policy. A unified \nWorkerAgent\n exposes the same \nexecute(dag)\n interface and routes internally without having to change the orchestrator at all. Our compositional architecture payoff once again\n\nLet’s try routing a mixed seven-node DAG through the unified worker. Each bar is colored based on the Agent used to handle it:\n\nThe timeline shows \nwhy\n the split matters operationally: fetch nodes finish a thousand times faster than writer nodes. The writers own the critical path and are worth giving their own semaphore, retry policy, and monitoring tags. We could further extend this approach along the same line: a \nBrowserAgent\n or \nCodeAgent\n, etc. each with its own tool subset and trace tags for logging.\n\nWhats next\n\nAcross these three posts we took a ~35-line loop and layered on production-shaped primitives:\n\nEach of these pieces is optional, but they all compound on each other. The planner automatically sees the schema of any new tools that are added. Tighten verification, and the eval suite catches regressions. Swap the LLM backend, and the entire harness can be rerun with a one-line change. This composability and compounding is what makes the difference in production.\n\nYet, much is still missing. We have no persistent memory, defenses against prompt-injection, human in the loop capabilities, etc. We’ll explore these in future posts, but the winning approach remains the same: Build small primitives, compose them, measure the result. That’s the whole doctrine.", "url": "https://wpnews.pro/news/evaluating-your-agentic-harnesses", "canonical_source": "https://data4sci.com/blog/evaluating-your-agentic-harnesses", "published_at": "2026-08-12 14:30:11+00:00", "updated_at": "2026-08-12 14:42:10.425879+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["d4sci_harness.py", "Anthropic", "Claude", "Col. John Boyd", "OODA loop"], "alternates": {"html": "https://wpnews.pro/news/evaluating-your-agentic-harnesses", "markdown": "https://wpnews.pro/news/evaluating-your-agentic-harnesses.md", "text": "https://wpnews.pro/news/evaluating-your-agentic-harnesses.txt", "jsonld": "https://wpnews.pro/news/evaluating-your-agentic-harnesses.jsonld"}}