You can't prompt an LLM out of inventing statistics, but you can make an invented number impossible to publish. In my Research-to-LinkedIn Automation project, which turns a researcher's new papers into LinkedIn posts, the model extracts and drafts, while deterministic code decides what counts as true. Every "stated" claim needs a verbatim quote that is actually in the paper, any number without one is thrown away, and every numeral in the final draft must trace back to the source. If code says no, the draft fails, whatever the model thinks.
Posting about someone's research under their name means one wrong number is a public misquote of their own work. A vague sentence is embarrassing. A fabricated "37% reduction" is a correction request. So the design treats invented statistics as the highest-severity failure, and spends its strictest checks there.
Extraction is schema-driven (Zod plus the provider's structured-output mode), so there's no free-text parsing. Each claim carries a provenance:
STATED`` evidence must be a verbatim quote.INFERRED`` ABSENT
Two structural choices do most of the work:
After the model answers, code checks every STATED claim:
function containsEvidence(source: string, evidence: string): boolean {
const norm = (t: string) => t.toLowerCase().replace(/\s+/g, " ").trim();
return norm(evidence).length > 0 && norm(source).includes(norm(evidence));
}
It's plain substring matching after normalising case and whitespace. That's crude, and that's the point: a quote either appears in the paper or it doesn't, and no model gets a vote.
The drafting model writes from the extraction, not the paper, so errors can creep back in. The draft then goes through two independent checks in parallel, and both must pass:
A. An LLM fact-checker labels each claim SUPPORTED, HEDGED_OK, UNSUPPORTED or CONTRADICTED, and flags overstatement and anything that reads like medical advice.
B. Deterministic checks, which override the LLM:
12.5% or 3×) must appear in the verified important numbers or literally in the source text. One unmatched number fails the draft.
The rule from the design doc: if a check can be decided in code, code decides it. An LLM judge is good at nuance and bad at reliably noticing that 0.37 became 37%.
Being honest about the gaps matters as much as the checks:
Related: writing tool errors that AI agents understand, and building an OAuth 2.1 MCP server.
Written by Ali Ahmed · Original article