{"slug": "our-ai-reviewer-invented-a-request-our-producer-retried-245-times", "title": "Our AI reviewer invented a request. Our producer retried 245 times.", "summary": "A developer's team running ~100 unattended LLM agents on local models discovered a single document that was rewritten 245 times in 5 days, with a sibling document rewritten 225 times, totaling about 470 wasted generations. The root cause was a reviewer agent hallucinating a request that never existed, combined with a retry mechanism that counted reviews instead of contract failures, allowing the loop to run unbounded. The developer's audit of 2,038 reviews found a 0.2% hallucination rate, but the real risk was the infinite retry loop, not the rate itself.", "body_md": "We run ~100 LLM agents unattended on local models. Last week we found one\n\ndocument that had been rewritten **245 times in 5 days** — every attempt\n\nrejected. A sibling document: 225 times. Combined, about 470 wasted\n\ngenerations, all burned on the same two files.\n\nHere is the autopsy, with the actual numbers.\n\nOur pipeline is simple: a producer agent writes a document, a reviewer agent\n\nchecks it against a contract (minimum length, required sections, no\n\nplaceholder junk), and rejected work goes back with fix instructions.\n\nThe rejected document was a key-management (KMS) implementation spec —\n\n4,452 characters, perfectly on-topic. The reviewer's verdict:\n\n\"The request was a 3-line email triage response (LOCK / VERDICT / REASON),\n\nbut the answer is a long KMS spec. Rewrite as3 lines only.\"\n\nOne problem. We grepped the document: the words \"LOCK\", \"VERDICT\", and the\n\nname of the triage service **appear zero times** in it. The reviewer had\n\ninvented the request.\n\nTwo contracts collided:\n\nNo output can satisfy both. So the producer failed the contract, got\n\nre-queued, produced again, failed again — 245 times. Our retry cap counted\n\n**reviews**, but a contract-failed output never reaches review. The give-up\n\nmechanism existed; it just watched the wrong counter.\n\nOur review prompt contained the artifact body (first 4,000 chars) and the\n\noutput format. **It never contained the original request.** We asked a model\n\n\"does this match the request?\" without telling it what the request was.\n\nA model asked to judge against information it doesn't have will\n\nhallucinate that information. Ours did, confidently, 245 times' worth.\n\nBonus failure: we truncated long documents to 4,000 characters before\n\nreview without saying so, and reviewers marked them \"thin — cut off\n\nmid-sentence.\" The cut was ours, not the producer's.\n\nWe audited all 2,038 reviews on file for concrete terms (product names,\n\nformat tokens) that appear in the review but **nowhere in the reviewed\ndocument**. Result:\n\nThat's the uncomfortable lesson: a 0.2% hallucination rate produced 470\n\nwasted runs, because nothing ever gave up. Low rate × infinite retries =\n\nunbounded damage. The rate is not the risk; the loop is.\n\nEach fix ships with a test we deliberately broke to confirm it fails.\n\nThe checker that catches broken outputs in this story (empty text, language\n\nleakage, placeholder junk, contract violations) is free on npm:\n\n[honto-contract](https://www.npmjs.com/package/honto-contract) — it passed\n\n600 downloads last week, so somebody besides us finds this useful now.\n\nThe unattended-operation checklist and three of our watchdog templates are\n\nfree (email-gated):\n\n[Unattended-Operation Kit](https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=harness-kit)\n\nThe full set of 7 production templates (cron registry, silent-zero watch,\n\nheartbeat, output contracts — the exact ones in this story) is\n\n[US$59](https://gxcafe.co.jp/harness-kit/?utm_source=devto&utm_medium=article&utm_campaign=templates-pro).\n\nHonest note: we have no customers yet. Everything above is exactly what we\n\nrun on ourselves, measured on our own failures.", "url": "https://wpnews.pro/news/our-ai-reviewer-invented-a-request-our-producer-retried-245-times", "canonical_source": "https://dev.to/gxcafellc/our-ai-reviewer-invented-a-request-our-producer-retried-245-times-465f", "published_at": "2026-08-24 00:31:03+00:00", "updated_at": "2026-08-24 00:43:34.579500+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "mlops", "developer-tools"], "entities": ["honto-contract", "npm", "gxcafe.co.jp"], "alternates": {"html": "https://wpnews.pro/news/our-ai-reviewer-invented-a-request-our-producer-retried-245-times", "markdown": "https://wpnews.pro/news/our-ai-reviewer-invented-a-request-our-producer-retried-245-times.md", "text": "https://wpnews.pro/news/our-ai-reviewer-invented-a-request-our-producer-retried-245-times.txt", "jsonld": "https://wpnews.pro/news/our-ai-reviewer-invented-a-request-our-producer-retried-245-times.jsonld"}}