Throughout the AI System Design series we follow Mailmind, a fictional AI email assistant that reads an inbox, triages messages, and drafts replies. It is our running example.
Before you read: Why Models Make Things Up and Prompt Engineering for Agents, Production Grade.
Last time Mailmind learned to find its memories: plain metadata, search by meaning, and graph traversal. Now it can find facts. But what happens when it invents them?
Mailmind catches hallucinations with a layered detection system. Deterministic checks compare dates, amounts and IDs against the source with plain code. Structured citations make every fact carry a verbatim quote and a message ID. Grounding verification checks claims against the source emails. Confidence scores decide which claims get extra scrutiny. Consistency checks run an extraction twice and compare. A cross-verification agent double-checks high-stakes actions. Together, these layers catch most plausible fictions before they become wrong replies.
Even with perfect retrieval, an LLM (large language model) can confidently state that a flight departs at 3:15 PM when no email contains that time. The real challenge is catching those inventions before a user books a taxi for the wrong time.
A hallucination is a confident, plausible statement that has no support in the user's data. The model is not lying. It is completing a pattern, and the most likely next token is not always the true one; Why Models Make Things Up explains why. Think of an intern too embarrassed to say "I don't know," so they invent an answer with a straight face.
Researchers often split hallucinations into two kinds. Intrinsic ones contradict the source: the email says $480 and the summary says $540. Extrinsic ones cannot be verified from the source at all: a refund amount that appears in no email (Ji et al. 2022, Survey of Hallucination in Natural Language Generation).
In Mailmind, hallucinations come in four flavours, and each calls for a different detection layer:
| Type | What goes wrong | Mailmind example | Best detection layer |
|---|---|---|---|
| Factual | States something no email says | "The vendor agreed to refund you" when nobody agreed | Grounding check against the thread |
| Entity | Confuses people or companies | Credits a message to the wrong sender | Check sender and IDs against the email headers |
| Citation | Points at a message that says something else | Quotes msg_884 for a fact that is in msg_901, or nowhere | Verbatim quote match, plus a check that the quote supports the claim |
| Temporal | Gets a date or time wrong | A flight at 3:15 PM that was moved to 4 PM | Normalize dates and times, then compare exactly |
Grounding verification takes the claims the agent makes and checks them against the original source text. If no supporting sentence exists, the claim gets flagged. It is the most direct defence for claims about meaning.
Mailmind runs it after extraction. Say the agent drafts a reply that says "The vendor agreed to refund you by Friday." A verification prompt asks: "Is the claim 'the vendor agreed to refund you' supported by this thread? Answer supported, contradicted or not found, and quote the sentence that decides it." If the model cannot find support, the claim is marked ungrounded. This catches a common pattern: a statement that sounds right but was never written. A grounding check costs an extra model call, so batch it: one call can check every claim in a reply at once. Numbers, dates and IDs skip this step and go to the cheaper deterministic checks described below.
Confidence scoring asks the model to rate its own certainty for each extracted fact. Low-confidence facts go to review before they spread. It is cheap and fast, so it works as a first-pass triage.
The problem is that models can be confidently wrong. A hallucinated flight time might get a score of 0.95 because the pattern looks exactly like a real itinerary. Confidence is a signal, not a verdict. Mailmind uses it to decide which claims get the more expensive checks: a claim with low confidence goes straight to a "needs review" queue, while high-confidence claims still pass through the other layers. Confidence scoring alone would let too many fictions through. It is the triage nurse, not the surgeon.
A cross-verification agent is a second, independent model call that checks the primary agent's output against the source. It answers the same question with a different prompt and no access to the first agent's reasoning, which keeps it from simply agreeing with the first answer.
Mailmind uses this for scheduling actions. When the primary agent extracts a meeting time from a thread, a verification agent receives only the raw emails and the proposed time. It must confirm or reject the extraction. If the two disagree, the action is blocked and the discrepancy goes to the user. This doubles the model cost for high-stakes operations, but a wrong meeting time costs far more than an extra API (application programming interface) call. The verification agent is the skeptical colleague who checks your work before you hit send.
A related research method, Chain-of-Verification, has a model plan verification questions about its own draft, answer them independently, and only then write its final answer (Dhuliawala et al. 2023, Chain-of-Verification Reduces Hallucination in Large Language Models).
Consistency checking runs the same extraction twice with slightly varied prompts and compares the results. This is the idea behind SelfCheckGPT: when a model knows a fact, repeated samples tend to agree, and when it is making something up, the samples tend to diverge and contradict each other (Manakul et al. 2023, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models). If the two runs disagree on a fact, treat it as suspect.
Mailmind uses this for batch processing of order confirmations and refunds. The system extracts the refund amount from a thread with two different prompt phrasings. If both runs return $540, the fact is considered stable. If one says $540 and the other $520, both are flagged. The method needs no labelled answers and works even when no human has reviewed the thread.
It has a blind spot. Consistent wrong answers pass. A wrong fact the model learned in training, or an email it misreads the same way twice, repeats on every run. So consistency checks complement checks against the source; they do not replace them.
Mailmind requires every extracted fact to include a verbatim quote from the source email and the message ID it came from. The output schema rejects any claim that lacks a citation. This is not a suggestion in the prompt. It is a hard constraint enforced by the parser.
When the agent drafts a reply that includes a meeting time, the structured output must look like: {"time": "3:15 PM", "quote": "Let's meet at 3:15 PM", "message_id": "msg_abc123"}. If the quote does not appear in the referenced message, the claim is discarded. This shifts the burden of proof onto the agent. It cannot invent a time and quietly slip it into the reply.
A matching quote proves the text exists. It does not prove that the text supports the claim, or that it is the latest word: a later message may move the meeting from 3:15 PM to 4 PM. For high-stakes facts, also check the newest messages in the thread before acting.
For structured facts like dates, dollar amounts and tracking numbers, deterministic checks are faster, cheaper and more reliable than any model. If "$540" appears nowhere in the thread, no amount of model reasoning can make it true.
Exact text matching alone gives false alarms, though. The email might say "$540.00", "USD 540" or "540 dollars", and a date might be written as "this Friday". So normalize first: parse amounts to numbers, and parse dates to calendar dates, resolving words like "Friday" against the date the email was sent. Then compare. Here is a Python sketch of the amount and citation checks:
import re
from decimal import Decimal
MONEY = re.compile(r"(?:\$|USD\s?)(\d[\d,]*(?:\.\d+)?)|(\d[\d,]*(?:\.\d+)?)\s?(?:dollars|USD)", re.I)
def amounts_in(text):
"""'$540.00', 'USD 540' and '540 dollars' all become Decimal('540')."""
return {Decimal((a or b).replace(",", "")) for a, b in MONEY.findall(text)}
def verify_amount(claimed, thread_text):
return Decimal(str(claimed)) in amounts_in(thread_text)
def normalize(text):
return " ".join(text.lower().replace("’", "'").split())
def verify_citation(claim, messages):
msg = messages.get(claim["message_id"])
return msg is not None and normalize(claim["quote"]) in normalize(msg["body"])
Mailmind runs deterministic checks before any model-based verification. When the agent claims a refund of $540 and no amount in the thread normalizes to 540, the claim is rejected at once, with no model call needed. This catches the most embarrassing class of hallucination: a number that was never in the inbox. Save the model for semantic claims like "the sender is angry." For anything that can be matched with code, use code. Deterministic checks are the bouncer at the door. They don't negotiate.
The layers are not a single line that every fact walks through. Facts are routed by type and by stakes. Every fact first needs its quote and message ID. Structured facts (dates, amounts, IDs) go to the deterministic check. Semantic claims go to the model-based grounding check. Only high-stakes actions, or facts with low confidence, add cross-verification. Any failure goes to a human.
flowchart TD
A["Extract facts with quote and message ID"] --> B{"What kind of fact?"}
B -->|"Date, amount, ID"| C["Deterministic check: normalize, then compare"]
B -->|"Semantic claim"| D["LLM grounding check"]
C -->|pass| E{"High stakes or low confidence?"}
D -->|pass| E
E -->|No| F["Use the fact in the reply"]
E -->|Yes| G["Cross-verification agent"]
G -->|agrees| F
C -->|fail| H["Human review"]
D -->|fail| H
G -->|disagrees| H
The layers build on each other. Prompt-level grounding instructions reduce hallucinations at generation time. Schema-level citations keep unsupported claims from leaving the agent. Routing by type keeps the expensive checks for where they matter. Spot-check audits sample a fraction of processed threads to measure the system's real hallucination rate. Every caught hallucination goes into the regression test set, and recurring patterns become rules in the prompt. The defence is not a single wall. It is a series of tripwires, each catching what the previous one missed.
These checks have relatives elsewhere on the site. Grounding and Citations: Answers You Can Check covers citations in retrieval systems, Evaluating RAG: Retrieval Metrics That Predict Answer Quality shows how to measure faithfulness at scale, and Guardrails: Input, Process, Output places checks like these in the output layer of a full guardrail design.
| Technique | What it checks | Relative cost | Best for |
|---|---|---|---|
| Deterministic checks | Normalized exact match in the source | Near zero | Dates, amounts, IDs |
| Structured citations | Schema-enforced quote and message ID | Minimal | All extractions |
| Grounding verification | Claim against the source text | One model call (batch all claims) | Semantic claims |
| Confidence scoring | The model's own certainty | Minimal | Deciding what to check harder |
| Consistency checking | Two extractions compared | Two model calls | Detection without labels |
| Cross-verification agent | A second model checks the output | One extra model call | High-stakes actions |
{
"type": "comparison",
"title": "Deterministic checks vs LLM verification",
"caption": "Illustrative, not measured. For structured facts like dates and dollar amounts, deterministic checks are faster and cheaper.",
"before": {
"label": "LLM verification",
"points": [
"Cost: one model call per batch of claims",
"Latency: seconds",
"Catches: semantic mismatches"
]
},
"after": {
"label": "Deterministic checks",
"points": [
"Cost: negligible",
"Latency: milliseconds",
"Catches: wrong or missing dates, amounts and IDs"
]
}
}
Q: How do you detect a hallucination when you don't have the source text?
If the source is unavailable, you cannot ground the claim. In that case, Mailmind marks the fact as unverifiable and either refuses to use it or asks the user to confirm. No source means no certainty, so the system does not guess.
Q: Does confidence scoring actually work, or is it just a number?
It works as a weak signal. Models are often overconfident on hallucinations, so a high score does not guarantee truth. A low score is a useful hint worth extra checking, not proof. Mailmind uses low confidence to trigger deeper verification, never to approve facts.
Q: Why not just use a second LLM to verify everything?
Cost and latency. Running a verification agent on every extracted fact would double the model spend and slow every reply. Mailmind reserves cross-verification for high-stakes actions, where the cost of an error exceeds the cost of the extra call.
Q: Can't the model just hallucinate the citation quote too?
It can. That's why code checks the quote against the source email, after normalizing case and spacing. If the model invents a quote that isn't in the thread, the deterministic check catches it. The citation is not a promise. It is a testable claim.
Q: How do you measure the real hallucination rate in production?
Spot-check audits. Mailmind samples a random fraction of processed threads and has a human review the extracted facts against the original emails. That gives a true error rate. The system also tracks how often each verification layer flags a fact, but the audit is the ground truth.
Mailmind processes 10,000 email threads in a week. Spot-check audits show that roughly 6% of threads contain at least one wrong fact, but you don't know which ones. Your logs also show that threads with low confidence scores are about three times as likely to contain an error. You cannot have a human review all 10,000. How do you find the bad threads without reviewing everything?
Answer: Run the deterministic checks on every thread first. They cost almost nothing, so there is no reason to ration them: does each claimed date, amount and ID appear in the source once both sides are normalized? Flag every thread that fails. Then rank the rest by risk using signals you already log: low confidence scores (which your logs tie to three times the error rate), runs that took many ReAct iterations, very long threads, new sender domains, and threads with several similar numbers close together. Send the riskiest slice, as many as your budget allows, through the grounding check and cross-verification. Only threads that fail an automated check go to a human. Keep a small random spot-check across all threads, including those that passed, to measure how many errors this triage actually catches and to tune the ranking. Going forward, store confidence scores, quotes and message IDs with every extraction so the triage runs on its own.
Next: Context Window Management for Big Jobs. It covers chunking, map-reduce and retrieval: how an agent handles work that never fits in one context window.
If you want this kind of breakdown every week, how real systems actually work under the hood, subscribe to Internals Decoded at internalsdecoded.com.
Originally published at Internals Decoded. AI internals, explained conversationally.