{"slug": "don-t-let-an-unattended-model-rewrite-its-own-prompt", "title": "Don't let an unattended model rewrite its own prompt", "summary": "A developer outlined an \"envelope\" design for unattended LLM pipelines in which the model may only choose among human-approved plans, with failure feedback carried forward as typed template messages rather than a growing transcript. The approach caps prompt length, bounds attempts and model calls, and dead-letters records that exhaust the approved plan menu, backed by a breaker that pauses batches on rising dead-letter rates. A pilot on receipt field extraction using CORD v1 and a local qwen3-8k quant showed the design's structural properties but made no claim that output quality improved.", "body_md": "Unattended LLM pipelines fail in ways a person would normally fix by editing the prompt. If the system rewrites that prompt itself, the rewrite is new logic that nobody has reviewed. In a chat window that is tolerable, because a person reads the next turn. In a batch job it is not. The consumer of the output is code, the document in the window is often large, and the person shows up later, on an exception queue.\n\nThis note describes a narrower rule. At runtime the system may only choose among plans a person approved once for that version of the task. When an attempt fails, the feedback it carries forward is a short list of typed template messages, not a growing transcript of the failure. The note also says what a small receipt-extraction pilot did and did not show.\n\nThis is a design to build on. It is not a finished system, and it is not a claim that the outputs got better.\n\nCall the approved package an **envelope**. It holds the invariants, the allowed step graph, the list of plans the compiler may pick, the library of failure templates, and the numeric limits: attempts, time, prompt length, and how many times the same failure may repeat. People sign that package once per version. They do not sign each prompt the job emits.\n\nA run on one record looks like this.\n\nA separate breaker watches the batch. A rising dead-letter rate, or too many silent failures in a sample of accepted outputs, pauses the batch until a person decides to resume.\n\nThe prompt layout is deliberate. Stable instructions sit in the system role. The document stays byte-for-byte the same on every attempt. Only the plan, the ledger, and a short note change, and each of those is drawn from the envelope.\n\nThree properties follow from the controller, provided the envelope signature checks out and the deployment assumptions hold. They say nothing about whether the answer is good.\n\n**The prompt length is capped.** Each block has a token cap fixed in the envelope. The sum does not depend on the attempt number. A conversational retry that appends the last output and the last error gets longer every turn. This design refuses to send a prompt that would break the cap.\n\n**Every record ends.** There is a finite attempt limit, a timeout on every call, and a finite fallback menu that cannot re-enter the main loop. The record is accepted or it is dead-lettered. The number of model calls is bounded.\n\n**The plan, the note, and the invariants stay inside the envelope.** A plan must be one of the approved ones. A note must be an instance of an approved template. The invariant block is the one that was signed. Content inside the document is a different problem: the validator and the data-marking rules have to deal with it, and this design does not claim to have solved prompt injection.\n\nHumans still have work. They approve envelope versions, they clear dead letters, and they sample accepted outputs. That sample is the only way to see failures the validator is too weak to catch.\n\nThe pilot is receipt field extraction on CORD v1, a public dataset under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) ([Park et al.](https://huggingface.co/datasets/naver-clova-ix/cord-v1), [NeurIPS 2019 workshop](https://huggingface.co/datasets/naver-clova-ix/cord-v1);). Each record is a photographed retail receipt. The model saw text reconstructed from the word boxes, not the image. The task was nine amount fields plus the menu lines. A record was “clean” when every reference value appeared verbatim in that text.\n\nThe validator was intentionally loose. An amount passed if, after stripping currency marks and thousands separators, the number occurred somewhere in the receipt. A menu name passed if it occurred as a substring. A record can pass that check and still put the tax into the total. Arithmetic checks were not implemented.\n\nThe model was a local qwen3-8k quant at temperature 0.3, with an 8192-token context. Three conditions shared the envelope, the model, and the temperature: the controller, a single call, and a naive retry that repeated the same prompt with no failure directive. Seeds were 0, 1, and 2.\n\nEnvelope v2 told the model to copy values as printed, and it capped generation at 512 tokens. On the first 100 clean training receipts the controller dead-lettered the same six records on every seed, and rescued none of them. The directive added on the second attempt said the same thing as the first prompt, so the model did the same thing again.\n\nWe then read those six failures and wrote envelope v3. Two were truncated JSON: the receipt’s menu did not fit in 512 tokens. The other four were ambiguous instructions: a unit price marked with @, a line that prints both unit price and line total, or a value copied with its label (Total Rp 5.000 instead of 5.000). v3 raised the generation cap to 1024 and tightened the templates. That change was a new version, with a new hash, approved before the v3 runs.\n\nOn the same 100 training receipts, v3 accepted every record. Five of the six old failures were accepted on the second attempt, on every seed, after the typed directive was injected. The sixth, a truncated menu, passed on the first attempt once the generation cap was raised; no retry was required. One of the five, train-00103, is a cascade: the larger cap exposed a verbatim menu error that the truncation had hidden, and the directive then addressed that error on the second attempt. Naive retry, using the same v3 envelope and no directive, did not rescue those five records. A repeated identical prompt was not enough.\n\nThat pattern is consistent with a simple lesson. The ledger helps when the approved template tells the model something the first attempt did not. It does not show that the typed wording is what matters, as opposed to any extra hint. The two comparisons that would show that — a retry that carries only the check id, and a retry that appends the full failing output — were not run.\n\nOn 84 held-out clean test receipts, which were not used to write v3, the controller’s dead-letter rate was 2.4–3.6 percent (2 or 3 records out of 84). Single-shot and naive retry were both 7.1–9.5 percent (6 to 8 records). Naive retry’s dead-letter set matched single-shot on every seed. A paired exact test of controller against naive retry gave p-values from 0.0625 to 0.125. None of them is significant at 0.05, and with only four or five discordant pairs the test is too small to rule a real difference in or out. Two test records stayed dead letters on every controller seed. They are candidates for another envelope version, not evidence that v3 generalized cleanly.\n\nPrompt length stayed small: about 388 tokens on the first attempt, and about 506 to 653 on the second, inside an 8192-token window. The length cap was not under pressure. A conversational baseline that appends the whole failure was not run, so this pilot does not show that the envelope wins a length contest against that style of retry. The receipts are short relative to the problem the design is aimed at.\n\nAcceptance is not accuracy. On the training sample, a reference comparison disagreed with the CORD annotation on 11 of 14 accepted records for v2 (Wilson 95 percent interval about 52–92 percent) and on 10 of 14 for v3 (about 45–88 percent). A person reviewing the same 14 v3 records marked 8 wrong (about 33–79 percent). On the test set, the first 14 accepted records from one controller seed, taken in run order and not as a random sample, all disagreed with the annotation on at least one field (the interval for 14 of 14 still starts near 78 percent). No person reviewed that test sample. The intervals are wide. A permissive validator will still report a high accept rate on outputs that a reference, or a person, rejects.\n\nThe controller can be built so that length, termination, and envelope membership do not depend on the model behaving. On these receipts, repeating the same prompt did not repair the systematic failures. A more specific approved template did repair the ones we had already inspected, and the held-out difference pointed the same way without clearing a significance bar.\n\nThe accept rates are validator passes, not extraction quality. The adherence comparison and the pre-set cap on human effort were never run as defined. Mean inputs were a few hundred tokens, so the long-document case is still open.\n\nThe expensive part is writing the envelope. A narrow one blocks a legitimate fix. A loose one permits behavior the task owner did not intend. If a fixed prompt plus the same validator does as well as plan-switching inside the envelope, the runtime choice of plan is not the method. We did not time envelope authoring, and we did not freeze the plan and vary only the ledger.\n\nThe question left is practical. When is an approved envelope worth more than a fixed prompt and a strict validator? The pilot says the retry is only as good as the template a person was willing to sign. That is the part I trust enough to publish, and the part I most want checked before it goes out.", "url": "https://wpnews.pro/news/don-t-let-an-unattended-model-rewrite-its-own-prompt", "canonical_source": "https://dev.to/swaraj_shekhar/dont-let-an-unattended-model-rewrite-its-own-prompt-1dem", "published_at": "2026-11-04 03:03:22+00:00", "updated_at": "2026-11-04 03:22:25.999734+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "mlops", "ai-safety"], "entities": ["CORD v1", "qwen3-8k", "Park et al.", "NeurIPS 2019"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/don-t-let-an-unattended-model-rewrite-its-own-prompt", "markdown": "https://wpnews.pro/news/don-t-let-an-unattended-model-rewrite-its-own-prompt.md", "text": "https://wpnews.pro/news/don-t-let-an-unattended-model-rewrite-its-own-prompt.txt", "jsonld": "https://wpnews.pro/news/don-t-let-an-unattended-model-rewrite-its-own-prompt.jsonld"}}