{"slug": "i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t", "title": "I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.", "summary": "A developer built a tool to measure whether multi-agent scaffolding improves coding task performance, only to find that adding a planner, two drafters, and a judge made results worse (80% vs. 95%) at 22× the cost ($0.692 vs. $0.031) and 8× the latency (18.3s vs. 2.2s). Statistical analysis showed the 20-task suite lacked resolution to distinguish the harnesses, with only 3 of 20 tasks carrying information and none reaching significance. The developer warns that small test suites produce 'confident nonsense' and emphasizes that scoring should be deterministic, not model-graded.", "body_md": "I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.\n\nI never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.\n\nOne sweep. Twenty coding tasks, three harness shapes, real models, $0.99.\n\n| harness | calls/task | score | cost | latency |\n|---|---|---|---|---|\n| one drafter | 1 | 95% |\n$0.031 | 2.2s |\n| planner → drafter | 2 | 90% | $0.264 | 9.1s |\n| planner → two drafters → judge | 4 | 80% |\n$0.692 | 18.3s |\n\nAdding the scaffolding made it **worse** and cost **22× more**. The four-call panel beat the single drafter on **zero** of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.\n\nHere is what the tool actually said about that:\n\nOnly 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so\n\nthis suite cannot decidebetween them — that is a limit of the suite, not a finding about the harnesses.The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.\n\n95% versus 80% *looks* like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.\n\nSo the honest reading is narrower and more useful:\n\nThose are three different statements. Most eval tooling collapses them into a ranking.\n\nTwenty tasks move the score in five-point steps. That is the whole reason the test cannot decide: the resolution of the instrument is coarser than the effect being measured. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.\n\nThis is the same lesson my drift board taught me last month, when four \"regressions\" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.\n\nA harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.\n\nScoring is deterministic. Predicates execute the generated code and return a verdict; **no model grades anything**, so a score that moves means the system moved rather than the judge having a different day. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.\n\nThe comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.\n\nYour key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.\n\nOne vendor detail worth writing down, because it cost me an hour: `api.openai.com`\n\nanswers the CORS preflight with the right headers and then omits `access-control-allow-origin`\n\non the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what `anthropic-dangerous-direct-browser-access`\n\nis for. Testing only the preflight with `curl -X OPTIONS`\n\nshows success and is misleading.\n\nHarness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.\n\n$0.99 and about five minutes, to find out that the architecture I had been assuming was better is — on this evidence — not better, and definitely more expensive.\n\nI would rather know.\n\n**Tool:** [https://github.com/egnaro9/never-touch-ai](https://github.com/egnaro9/never-touch-ai) — draw a harness and sweep it; it runs free on mock substrates with no key at all.\n\n**Raw result:** [ results/sweep_2026-07-25.json](https://github.com/egnaro9/never-touch-ai/blob/main/results/sweep_2026-07-25.json) — every number above is computed from it, so you can check them.\n\n**Deeper write-up:** [the field note](https://github.com/egnaro9/never-touch-ai/blob/main/docs/field-note-first-result.md) — the graph execution model, why it's a sign test, and the two bugs the live run surfaced.\n\n**Built by** Erik Hill · [https://egnaro9.github.io](https://egnaro9.github.io)", "url": "https://wpnews.pro/news/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t", "canonical_source": "https://dev.to/agentdev9/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasnt-do", "published_at": "2026-07-25 20:06:31+00:00", "updated_at": "2026-07-25 20:31:50.807649+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "ai-research"], "entities": ["OpenAI", "Anthropic", "promptfoo", "LangSmith", "Braintrust"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t", "markdown": "https://wpnews.pro/news/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t.md", "text": "https://wpnews.pro/news/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t.txt", "jsonld": "https://wpnews.pro/news/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasn-t.jsonld"}}