You can never backfill a read receipt: a blog comment fixed our AI pipeline

A developer's AI pipeline was fixed after a blog commenter, pm25coder, pointed out that provenance records could be gamed by consumers that record source IDs without reading content. The team implemented read receipts at hand-off, content fingerprints at enqueue, and re-derivable citations, ensuring unread inputs show up as mismatch counts. The developer also confessed that their first audit version skipped entries lacking a hash, allowing forged citations to pass, and emphasized testing watchdogs by faking failures.

Last week we published a post-mortem: our AI reviewer hallucinated a request that didn't exist, and our producer retried the same document 245 times. A reader, pm25coder, left a comment that turned out to be the best code review we've ever received. This post is what we shipped because of it, same day. Our fix at the time was provenance: record which artifact each output was made from. The comment pointed at the hole: "Which artifact it was made from" can still be gamed by the exact failure it exists to catch — a consumer that records the source id without reading the content. ... Make the consumer cite what it actually read. ... You can never backfill a read receipt, so it has to be written when the reading happens. He was right, and we checked: our approval queue recorded the path of every deliverable. Nothing anywhere proved the enqueuer had ever opened the file. An empty or deleted file could sit in the human approval queue looking exactly like a real one. 1. Read receipts at hand-off. When the reviewer receives an excerpt, we now write {sha256 excerpt , excerptChars, fullChars} at the moment of hand-off. Written at read time, because it cannot be written later. 2. Fingerprints at enqueue. A deliverable enters the approval queue only after the enqueuer actually reads it: content hash and length are stored on the queue entry. Unreadable or empty → it never enters the queue, and the reason is archived where a human can see it. 3. Re-derivable citations. The queue entry also quotes the reviewer's verdict line verbatim . The nightly audit re-extracts that line from the review file and compares. A citation that cannot be re-derived from the artifact means someone wrote it without reading — and it now shows up as a mismatch count , not as an absence. That was the commenter's core point: an unread input should be a number on a chart, not a blank space. Worth confessing: our first version skipped any queue entry that lacked a hash — if entry.sha256 continue; — which meant a forged citation without a hash sailed through the very audit built to catch forged citations. We only found it because we test every check by faking the failure it should catch, and the fake passed. If you build watchdogs: break them on purpose before you trust them. Ours needed it, again. The output-contract checker from this story is free on npm: honto-contract https://www.npmjs.com/package/honto-contract . Our unattended-operation checklist and three watchdog templates are free email-gated : Unattended-Operation Kit https://gxcafe.co.jp/harness-kit/?utm source=devto&utm medium=article&utm campaign=harness-kit The full production set — now including the payment gate we extracted this week — is on the same page https://gxcafe.co.jp/harness-kit/?utm source=devto&utm medium=article&utm campaign=templates-pro . Honest note: still no customers. Everything above is what we run on ourselves, measured on our own failures. And thanks again, pm25coder — comments like yours are why we write these.