#
I spent 9.3M tokens on a 109-agent code audit so you don't have to
The short version: I pointed a swarm of AI agents at a ~5k-line codebase to hunt for things worth fixing. The pipeline was parallel subsystem mappers → 8 "finder" lenses → dedup → adversarial verification of every finding → a ranking panel → synthesis.
It worked. 32 verified findings, a clean top-10, no complaints about the output. But it cost ~9.3M tokens (call it $46 of API time) and somewhere north of two-thirds of that was me lighting money on fire. Here's the autopsy.
#
Where the tokens actually went
| Stage |
Agents |
What went wrong |
| Verify |
86 of 109 (79%!) |
Killed only 2 of 34 findings — I paid the full cost of re-reading the code 86 times for a 6% hit rate |
| Map |
9 |
Redundant. The finders re-read the code anyway (correctly), so the map was a tax I paid twice |
| Find |
8 lenses |
~30% overlap between them (48 raw findings collapsed to 34 after dedup) |
| Prompts |
— |
JSON.stringify(x, null, 2) — yes, the pretty-printing — quietly bloated every downstream prompt by 30-40% |
The kicker: cache reads were 77% of all tokens. Every agent spawns with a fresh context and re-reads the same files from scratch. That's just the cost of fanning out wide — which means you'd better fan out where it pays.
#
The one mistake that mattered
I verified everything before I ranked anything. Adversarial verification is the single most expensive stage per unit of value, and most findings never even make the final cut. So I was paying premium prices to fact-check findings that were destined for the cutting-room floor.
Flip the order:
Same output. ~70% fewer agents. That's the whole lesson, really — the rest is footnotes.
#
Footnotes (the actual rules, in order of how much they save you)
Rank first, verify after. Only spend adversarial verification on the findings that'll actually appear in the deliverable. #
Match the paranoia to the stakes. A wrong finding in an internal audit costs someone a few minutes of reading — one refuter is plenty. Save the full 3-lens panel (code-truth / impact / approach) for finalists or claims that trigger real action. #
Batch verification by file. 34 findings lived in ~10 files. One verifier checking every finding against the same file reads it once instead of ten times. Never do per-finding-per-lens fan-out — that's the expensive way to learn this lesson. #
Skip the mappers on small repos (<10k LOC). One agent can swallow the whole thing, and the finders re-read it anyway, so a map phase is overhead twice over: once to build it, once to staple it onto every finder prompt. #
Six finder lenses, tops — with explicit "you don't cover X, that's the other agent's job" boundaries. Overlap isn't free; every duplicate has to swim through dedup and verification before it dies. #
Compact your JSON. JSON.stringify(x)
, never null, 2
. At agent-prompt scale, pretty-printing is just padding nobody reads. #
Send the cheap models to do the chores. Dedup, evidence-checking, code-truth verification — none of it needs the frontier model. #
Set a token budget up front and have the orchestrator check what's left before each fan-out, scaling the fleet down instead of barreling ahead open-loop.
#
What I'd keep, no notes
The code-truth veto in verification earned its keep — it caught 2 plausible-but-wrong findings that would've embarrassed me in the final report. Adversarial checking is worth every token on the things that survive ranking. #
Structured output schemas on every agent: zero parse failures across all 109. Boring, reliable, great. #
Hard caps on finder output (max 6 each) kept the funnel from ballooning. #
Resumable runs with cached results — I stopped mid-run and picked it back up later for basically nothing.
#
TL;DR
Fan out to find. Converge before you verify. Breadth is for discovery; rigor is for the survivors. Do it backwards and you'll end up — as I did — with 79% of your agents diligently auditing things no human will ever read.