Our site, presentofai.com, publishes AI industry analysis daily with no human in the writing loop: agents ingest news and company filings into an event timeline, score them, and synthesize digests and long form articles. This post is about the part nobody plans for on day one: the verification pipeline we had to build after the writing pipeline embarrassed us.
If you are shipping LLM-generated content to the public, here is the architecture that stopped the bleeding, and the specific bugs that forced each layer. After every render, a judge model checks the draft against the source events it was built from: wrong attribution, merged or split entities, date errors, dek-vs-body contradictions, load bearing claims resting on a single source, number errors. Any high severity finding triggers exactly one revision pass, grounded only in the source events.
Why one pass and not a loop? Because we watched each regeneration fix the flagged error and introduce a new one, always in the hardest to verify detail: a bill's sponsors, two similar bills merged into one, a date that was actually the date reporting confirmed the event rather than the date it happened. Unbounded self-revision does not converge, it wanders.
The critic can only see the source events. If the error is IN your source data, the critic faithfully reproduces it. So a second stage extracts every load bearing claim (who, what mechanism, when, why, number) with a neutral search query for each, runs a fresh news search per claim, reads two or three independent articles, and rules each claim supported, wrong, contested or unverified.
This layer caught an invented attribution that had survived five prior review rounds: the draft credited a named former official with a specific quoted phrase, and the fresh search showed he had co-signed a group letter with different wording. The phrase belonged to someone else.
One rule keeps this layer honest: the model's world knowledge may flag a claim, but only retrieved sources may authorize a rewrite. If the model is sure something is wrong but the sources are silent, the claim gets hedged as contested or unverified, never rewritten from memory. A fabricated correction is still a fabrication.
Engineering note: firing all claim searches concurrently got us rate limited into uselessness, and the failure mode was silent because no sources means no findings means the draft passes. If your safety layer can fail open, cap the concurrency and add bounded retries, or it will pass everything exactly when it is broken.
The real fix was moving verification to the source layer. A rolling audit re-reads each timeline event's original source article and has a judge check for the recurring failure classes: speculation stated as fact, contested claims stated flat, a later confirmation date used as the action date, over-claims like "first known instance", and re-reports of an action that is already in the database under an earlier date. It rewrites summaries in place, merges duplicates via redirects, and flags mismatched sources.
That last class is the sneaky one. News wires re-report the same government action for weeks. Different headlines, different dates, same action. Our judge missed them because both title and date differed, so one June order became three events, and an article built on that timeline claimed an escalation that never happened.
Prompt instructions against duplication helped until they did not. What held was a code guard: reject any proposed article whose event-overlap with an existing one exceeds a threshold, measured against the smaller of the two sets. Our first version measured against the candidate's own list, and the model defeated it on the next run by padding its list until the ratio dropped under the bar.
The generalized lesson: never let the thing being validated control the denominator, and treat prompts as suggestions to a system that optimizes around them. Enforcement lives in code.
All of this runs behind presentofai.com, which tracks AI industry events, digests and pattern analyses daily. If you want the beginner-facing version, we also published a step by step guide to building your first agent for teams without engineers, with real costs and failure modes.