{"slug": "how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral", "title": "How to stop an LLM from inventing numbers: evidence containment and numeral tracing", "summary": "Ali Ahmed built a research-to-LinkedIn automation pipeline that uses deterministic code, not the LLM, to decide what counts as true: every STATED claim must carry a verbatim quote found in the source paper via normalized substring matching, and every numeral in the final draft must trace back to a verified number or the source text, with a single unmatched figure failing the draft. An LLM fact-checker runs in parallel but is overridden by the code checks, following the design rule that \"if a check can be decided in code, code decides it.", "body_md": "You can't prompt an LLM out of inventing statistics, but you can make an invented number **impossible to publish**. In my [Research-to-LinkedIn Automation](https://github.com/QuantumBreakz/Linkedin-Automation) project, which turns a researcher's new papers into LinkedIn posts, the model extracts and drafts, while **deterministic code decides what counts as true**. Every \"stated\" claim needs a verbatim quote that is actually in the paper, any number without one is thrown away, and every numeral in the final draft must trace back to the source. If code says no, the draft fails, whatever the model thinks.\n\nPosting about someone's research under their name means one wrong number is a public misquote of their own work. A vague sentence is embarrassing. A fabricated \"37% reduction\" is a correction request. So the design treats **invented statistics as the highest-severity failure**, and spends its strictest checks there.\n\nExtraction is schema-driven (Zod plus the provider's structured-output mode), so there's no free-text parsing. Each claim carries a **provenance**:\n\n`STATED`` evidence` must be a verbatim quote.`INFERRED`` ABSENT`\nTwo structural choices do most of the work:\n\nAfter the model answers, code checks every `STATED` claim:\n\n``` js\nfunction containsEvidence(source: string, evidence: string): boolean {\n  const norm = (t: string) => t.toLowerCase().replace(/\\s+/g, \" \").trim();\n  return norm(evidence).length > 0 && norm(source).includes(norm(evidence));\n}\n```\n\nIt's plain substring matching after normalising case and whitespace. That's crude, and that's the point: a quote either appears in the paper or it doesn't, and no model gets a vote.\n\nThe drafting model writes from the extraction, not the paper, so errors can creep back in. The draft then goes through **two independent checks in parallel**, and both must pass:\n\n**A. An LLM fact-checker** labels each claim `SUPPORTED`, `HEDGED_OK`, `UNSUPPORTED` or `CONTRADICTED`, and flags overstatement and anything that reads like medical advice.\n\n**B. Deterministic checks**, which **override** the LLM:\n\n`12.5%` or `3×`) must appear in the verified important numbers or literally in the source text. One unmatched number fails the draft.\nThe rule from the design doc: **if a check can be decided in code, code decides it.** An LLM judge is good at nuance and bad at reliably noticing that 0.37 became 37%.\n\nBeing honest about the gaps matters as much as the checks:\n\nRelated: [writing tool errors that AI agents understand](https://aliahmed.si/blog/agent-tool-error-messages/), and [building an OAuth 2.1 MCP server](https://aliahmed.si/blog/oauth-mcp-server-crm/).\n\nWritten by Ali Ahmed · [Original article](https://aliahmed.si/blog/stop-llm-inventing-numbers/)", "url": "https://wpnews.pro/news/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral", "canonical_source": "https://dev.to/ali_ahmed_44ee99051e327ce/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral-tracing-28l9", "published_at": "2026-11-05 22:37:12+00:00", "updated_at": "2026-11-05 22:53:01.398645+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Ali Ahmed", "LinkedIn", "Zod"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral", "markdown": "https://wpnews.pro/news/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral.md", "text": "https://wpnews.pro/news/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral.txt", "jsonld": "https://wpnews.pro/news/how-to-stop-an-llm-from-inventing-numbers-evidence-containment-and-numeral.jsonld"}}