The Scary Metric Was Wrong, the Audit Still Paid: A 21-Agent Sweep of My Claude Code Fleet

A developer running a fleet of 21 Claude Code agents conducted an audit that found 8 confirmed issues, including a silent cost regression in a built-in sub-agent and prompt-cache determinism as a major token-cost lever. The audit refuted 4 of 12 shortlisted candidates, including one where cited sources contradicted the claim. The developer emphasizes that verifying findings by re-fetching sources is critical to the audit's value.

Originally published on hexisteme notes. I run a small fleet of Claude Code agents that coordinate research, narrative, app-dev, and trading work through a shared task bus. Every so often I stop building on it and turn a scan on it instead — not "what should the fleet build next" but "is the fleet itself wasting money or lying to me." This time the scan was bigger than usual: 21 agents, about 2.4 million tokens, roughly 21 minutes wall clock. It found 79 raw candidate findings, verified them down with an adversarial pass to 8 confirmed, 2 partially rescued, and 4 refuted. And it surfaced one number that looked like the whole fleet was broken. That number turned out to be wrong. The audit was still worth running, because chasing the wrong number down is what found the three things that actually were broken. The setup was itself a small lesson in delegation discipline: cheap inventory work went to Haiku sub-agents, research went to Sonnet, the one step that needed to hold five research threads in its head at once went to Opus, and every candidate finding got re-checked by a separate Sonnet verifier before it was allowed to graduate to "confirmed." The rule for confirmation was strict and mechanical: the verifier had to actually re-fetch the cited source and match the claim against the live document, not trust the first agent's paraphrase. That rule is why 4 of the 12 shortlisted candidates got refuted outright — including one where the cited source directly contradicted the claim a recommendation to migrate a service off one transport to fix a bug report, when the bug's own reproduction steps happened on the target transport, meaning the migration wasn't the fix at all . Two more refuted candidates were just old news wearing a new hat: one was already-adopted behavior being re-pitched as a fresh recommendation, the other was a rename that got blamed for an unrelated outage with no supporting evidence. Verifying before adopting is not overhead here — it's the entire value of running the audit instead of just reading the raw findings. Four of the confirmed findings are worth generalizing past this one fleet. A silent cost regression in a built-in sub-agent. The CLI's built-in exploration sub-agent — the fast, cheap one meant for locating code and answering "where is X" — stopped being hard-pinned to the cheap model tier as of a recent CLI version, and now silently inherits whatever model the main session is running. On a flagship-model main session, every exploration call quietly bills at flagship rates instead of the cheap tier it was designed for. The verifier confirmed the regression was already active on this machine's installed version. The fix is not a code change, it's a discipline change: exploration calls now carry an explicit cheap-model override every time, because the default silently stopped being cheap. Prompt-cache determinism is the single biggest token lever nobody audits. Prompt caching only works on an exact prefix match, scoped to a specific model, with a short time-to-live — and a single byte of difference anywhere before the cache breakpoint invalidates everything after it. The usual silent killers are boring: a timestamp or a UUID generated fresh and dropped into a system prompt, JSON serialized without sorted keys so field order drifts between calls, tool lists that get reordered, or a system prompt that gets edited mid-session. None of these throw an error. They just quietly turn every call into a full-price cache miss. Since a cache read runs at roughly a tenth of the price of a fresh input token, and long-lived worker sessions re-send large system prompts on every turn, this is the largest single token-cost lever available for any harness running multi-turn workers — bigger than model choice, bigger than most prompt trimming. The audit's actual deliverable here wasn't a recommendation, it was a checklist: strip nondeterminism from every worker's system-prompt assembly, then assert in a real call that cache-read tokens are actually nonzero, because "I removed the timestamp" and "caching is actually working" are two different claims. A pricing promo cliff, compounded by a tokenizer change. The vendor's pricing page confirmed a promotional rate on the current model generation expires on a fixed date later this year, after which the per-token price rises. That alone is a plannable step function. What makes it worse is that the same model generation uses a different tokenizer than its predecessor, and produces roughly 30% more tokens for the same input text. Combine a price-per-token increase with a token-count increase on the same text, and a cost baseline that isn't recalibrated on that date will be silently wrong by something like 1.5x going forward — not because usage changed, but because the ruler changed. The fix was cheap: one dated line in a monitoring cadence, "recompute the token-count baseline on the current model's tokenizer on this date, don't reuse the old generation's counts." Turning the routing table from a reminder into an enforced rule. My own operating notes have long had a table saying which task types should go to which model tier — cheap tier for mechanical work, mid tier for standard implementation, top tier for architecture and hard judgment calls. A table in a docs file is advisory; nothing stops an agent from ignoring it under pressure. The audit found two harness features that make it enforceable instead of advisory: per-agent effort settings in a sub-agent's own configuration file so a cheap-tier worker runs at low reasoning effort no matter what the ambient session effort is , and a permission-rule syntax that can explicitly deny an agent from spawning an expensive-model sub-agent at all. Applied together, a mechanical worker is now structurally incapable of quietly escalating itself to the flagship model — the table stopped being something an agent could ignore and became something the harness enforces. Buried in the same inventory pass was the scariest line in the whole report: the fleet's shared task bus had a lifetime completion rate of 12% — about 1,055 tasks issued, 127 results ever collected — and every single one of those 127 collected results appeared to have come from a generic fallback handler rather than a real worker. Read at face value, that's a fleet that has been quietly failing at its one job for months while a fallback silently papered over it. I diagnosed it the same day, in a separate pass, and the number was an illusion — but not a boring one. The "every result is a fallback" claim was a schema misreading: the field the audit was checking for provenance simply didn't exist in the version-1 result schema being inspected, so its absence looked like "no real worker," when in fact all 127 results were genuine worker output that had just never been tagged with a from field to begin with. The "workers unreachable" events — 252 of them — were all artifacts of a benchmark run from weeks earlier that had never been cleaned out of the same log stream the audit was reading; the current seven registered workers were all healthy. And the 12% figure itself was contamination: 88% of the total task backlog turned out to be leftover test-bench debris counted as if it were real production backlog. Once that debris was quarantined out, the true completion rate on the real backlog was 98.4%. A separate stretch of apparent silence turned out to be neither a crash nor a hang — it was the fleet correctly refusing new work because a weekly session budget had actually been exhausted, which is itself corroborating evidence for the cost-regression problem in Act 1: something has been burning through top-tier-model budget fast enough to hit the ceiling. None of that makes the diagnosis a wash. Chasing the false alarm down surfaced three real defects that the alarm had simply pointed near, not at: Audit metrics in an agent system are not neutral measurements of the system — they are artifacts produced by the same harness they're measuring, inheriting its schema drift, its stale test data, and its logging gaps. A "12% completion rate" and a "98.4% completion rate" can both come out of the same event log, depending entirely on whether you trust the aggregation without checking it. The discipline isn't "don't trust scary numbers" — it's verify the metric with the same rigor you'd verify any other claim before you react to it. And a false alarm, investigated properly instead of dismissed once it's debunked, is not wasted effort: the investigation is what turns up the crash bug, the silent permission gap, and the pointless daily spend that the alarm was only ever pointing near. More notes at hexisteme.github.io/notes.