# I built a tool to prove my multi-agent harness was worth it. It told me it wasn't.

> Source: <https://dev.to/agentdev9/i-built-a-tool-to-prove-my-multi-agent-harness-was-worth-it-it-told-me-it-wasnt-do>
> Published: 2026-07-25 20:06:31+00:00

I spend most of my time on agentic systems, and I had absorbed the same idea everyone else has: a planner improves things, and a panel of drafters with a judge improves them further. It sounds obviously true. More thinking, more review, better answers.

I never measured it. So I built something that could, pointed it at my own setup, and it disagreed with me.

One sweep. Twenty coding tasks, three harness shapes, real models, $0.99.

| harness | calls/task | score | cost | latency |
|---|---|---|---|---|
| one drafter | 1 | 95% |
$0.031 | 2.2s |
| planner → drafter | 2 | 90% | $0.264 | 9.1s |
| planner → two drafters → judge | 4 | 80% |
$0.692 | 18.3s |

Adding the scaffolding made it **worse** and cost **22× more**. The four-call panel beat the single drafter on **zero** of twenty tasks and lost three. Nothing errored — 0% failure rate across all sixty runs. It just did worse work, slower, for twenty-two times the money.

Here is what the tool actually said about that:

Only 3 tasks separated them. Even a clean sweep of 3 could not clear p<0.05, so

this suite cannot decidebetween them — that is a limit of the suite, not a finding about the harnesses.The panel costs 22× more and the suite cannot decide between them — on this evidence the extra spend buys nothing.

95% versus 80% *looks* like a decisive result. It isn't. Seventeen of the twenty tasks were ties, so only three carried any information, and three discordant tasks cannot reach significance even if one side sweeps all of them. A leaderboard would have printed the two numbers and let me conclude the panel is worse. That would have been a stronger claim than the data supports.

So the honest reading is narrower and more useful:

Those are three different statements. Most eval tooling collapses them into a ranking.

Twenty tasks move the score in five-point steps. That is the whole reason the test cannot decide: the resolution of the instrument is coarser than the effect being measured. The fix is not better statistics, it is more tasks — which is why the tool lets you bring your own suite and tells you, as you paste it, how many points each task is worth.

This is the same lesson my drift board taught me last month, when four "regressions" turned out to be rate limits and single-question noise. Small suites produce confident nonsense.

A harness config is data — roles, models, prompts, and a topology graph. You draw the shape or paste the JSON; each is a view of the other. Declare the axes you want to vary and it runs the matrix.

Scoring is deterministic. Predicates execute the generated code and return a verdict; **no model grades anything**, so a score that moves means the system moved rather than the judge having a different day. There is an assistant in the page, and it is allowed to read scores and explain them — never to produce one.

The comparison is per-task and paired, not two averages. Both shapes run the same twenty tasks, so the question is how many tasks one won, which has far more power at this sample size than comparing means. The test is an exact sign test: no normality assumption, no variance assumption, ties excluded because they carry no direction.

Your key stays in the browser. The backend receives sanitized traces and refuses anything key-shaped at its boundary; the page shows you the exact bytes it posts and tells you to check your own Network tab rather than believe the panel.

One vendor detail worth writing down, because it cost me an hour: `api.openai.com`

answers the CORS preflight with the right headers and then omits `access-control-allow-origin`

on the actual response, so a browser-direct call is discarded no matter how valid the key is. Anthropic opts in deliberately — that is what `anthropic-dangerous-direct-browser-access`

is for. Testing only the preflight with `curl -X OPTIONS`

shows success and is misleading.

Harness and prompt-comparison tooling is not a new category — promptfoo, LangSmith, Braintrust and others do model and prompt comparison, several with far more surface area than this. The narrow thing here is an intersection: browser-BYOK, plus deterministic no-LLM-judge grading, plus a comparison that reports when it cannot decide.

$0.99 and about five minutes, to find out that the architecture I had been assuming was better is — on this evidence — not better, and definitely more expensive.

I would rather know.

**Tool:** [https://github.com/egnaro9/never-touch-ai](https://github.com/egnaro9/never-touch-ai) — draw a harness and sweep it; it runs free on mock substrates with no key at all.

**Raw result:** [ results/sweep_2026-07-25.json](https://github.com/egnaro9/never-touch-ai/blob/main/results/sweep_2026-07-25.json) — every number above is computed from it, so you can check them.

**Deeper write-up:** [the field note](https://github.com/egnaro9/never-touch-ai/blob/main/docs/field-note-first-result.md) — the graph execution model, why it's a sign test, and the two bugs the live run surfaced.

**Built by** Erik Hill · [https://egnaro9.github.io](https://egnaro9.github.io)