We run ~100 LLM agents unattended on local models. Last week we found one
document that had been rewritten 245 times in 5 days — every attempt
rejected. A sibling document: 225 times. Combined, about 470 wasted
generations, all burned on the same two files.
Here is the autopsy, with the actual numbers.
Our pipeline is simple: a producer agent writes a document, a reviewer agent
checks it against a contract (minimum length, required sections, no
placeholder junk), and rejected work goes back with fix instructions.
The rejected document was a key-management (KMS) implementation spec —
4,452 characters, perfectly on-topic. The reviewer's verdict:
"The request was a 3-line email triage response (LOCK / VERDICT / REASON),
but the answer is a long KMS spec. Rewrite as3 lines only."
One problem. We grepped the document: the words "LOCK", "VERDICT", and the
name of the triage service appear zero times in it. The reviewer had
invented the request.
Two contracts collided:
No output can satisfy both. So the producer failed the contract, got
re-queued, produced again, failed again — 245 times. Our retry cap counted
reviews, but a contract-failed output never reaches review. The give-up
mechanism existed; it just watched the wrong counter.
Our review prompt contained the artifact body (first 4,000 chars) and the
output format. It never contained the original request. We asked a model
"does this match the request?" without telling it what the request was.
A model asked to judge against information it doesn't have will
hallucinate that information. Ours did, confidently, 245 times' worth.
Bonus failure: we truncated long documents to 4,000 characters before
review without saying so, and reviewers marked them "thin — cut off
mid-sentence." The cut was ours, not the producer's.
We audited all 2,038 reviews on file for concrete terms (product names,
format tokens) that appear in the review but nowhere in the reviewed document. Result:
That's the uncomfortable lesson: a 0.2% hallucination rate produced 470
wasted runs, because nothing ever gave up. Low rate × infinite retries =
unbounded damage. The rate is not the risk; the loop is.
Each fix ships with a test we deliberately broke to confirm it fails.
The checker that catches broken outputs in this story (empty text, language
leakage, placeholder junk, contract violations) is free on npm:
honto-contract — it passed 600 downloads last week, so somebody besides us finds this useful now.
The unattended-operation checklist and three of our watchdog templates are
free (email-gated):
[Unattended-Operation Kit](https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=harness-kit)
The full set of 7 production templates (cron registry, silent-zero watch,
heartbeat, output contracts — the exact ones in this story) is
US$59. Honest note: we have no customers yet. Everything above is exactly what we
run on ourselves, measured on our own failures.