# Our AI reviewer invented a request. Our producer retried 245 times.

> Source: <https://dev.to/gxcafellc/our-ai-reviewer-invented-a-request-our-producer-retried-245-times-465f>
> Published: 2026-08-24 00:31:03+00:00

We run ~100 LLM agents unattended on local models. Last week we found one

document that had been rewritten **245 times in 5 days** — every attempt

rejected. A sibling document: 225 times. Combined, about 470 wasted

generations, all burned on the same two files.

Here is the autopsy, with the actual numbers.

Our pipeline is simple: a producer agent writes a document, a reviewer agent

checks it against a contract (minimum length, required sections, no

placeholder junk), and rejected work goes back with fix instructions.

The rejected document was a key-management (KMS) implementation spec —

4,452 characters, perfectly on-topic. The reviewer's verdict:

"The request was a 3-line email triage response (LOCK / VERDICT / REASON),

but the answer is a long KMS spec. Rewrite as3 lines only."

One problem. We grepped the document: the words "LOCK", "VERDICT", and the

name of the triage service **appear zero times** in it. The reviewer had

invented the request.

Two contracts collided:

No output can satisfy both. So the producer failed the contract, got

re-queued, produced again, failed again — 245 times. Our retry cap counted

**reviews**, but a contract-failed output never reaches review. The give-up

mechanism existed; it just watched the wrong counter.

Our review prompt contained the artifact body (first 4,000 chars) and the

output format. **It never contained the original request.** We asked a model

"does this match the request?" without telling it what the request was.

A model asked to judge against information it doesn't have will

hallucinate that information. Ours did, confidently, 245 times' worth.

Bonus failure: we truncated long documents to 4,000 characters before

review without saying so, and reviewers marked them "thin — cut off

mid-sentence." The cut was ours, not the producer's.

We audited all 2,038 reviews on file for concrete terms (product names,

format tokens) that appear in the review but **nowhere in the reviewed
document**. Result:

That's the uncomfortable lesson: a 0.2% hallucination rate produced 470

wasted runs, because nothing ever gave up. Low rate × infinite retries =

unbounded damage. The rate is not the risk; the loop is.

Each fix ships with a test we deliberately broke to confirm it fails.

The checker that catches broken outputs in this story (empty text, language

leakage, placeholder junk, contract violations) is free on npm:

[honto-contract](https://www.npmjs.com/package/honto-contract) — it passed

600 downloads last week, so somebody besides us finds this useful now.

The unattended-operation checklist and three of our watchdog templates are

free (email-gated):

[Unattended-Operation Kit](https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=harness-kit)

The full set of 7 production templates (cron registry, silent-zero watch,

heartbeat, output contracts — the exact ones in this story) is

[US$59](https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=templates-pro).

Honest note: we have no customers yet. Everything above is exactly what we

run on ourselves, measured on our own failures.
