Let your LLM say I don't know instead of guessing

A developer writing on Chain of Thought argues that applications should let language models abstain rather than guess, citing Scaled Cognition CTO Dan Klein's claim that enterprise audits reveal real hallucination rates roughly five times what customers notice. The post recommends grounding answers in retrieved context with per-claim citations, permitting explicit "I don't know" responses when facts are missing, and tracking separate evaluation counts for answered, abstained, supported and unsupported answers.

Your user asks a factual question. The model returns a polished paragraph. Nothing in the stack threw an error. That is exactly when you are most exposed. A language model generates the most plausible next words, not the most true ones, so a wrong answer can look identical to a right one. That mechanism is what people call AI hallucination https://chainofthought.show/glossary/ai-hallucination/ : same token prediction, missing or weak facts, fluent fiction at the end. Dan Klein https://chainofthought.show/guests/dan-klein/ , co-founder and CTO of Scaled Cognition, put the internals this way on Chain of Thought: we label outputs after the fact, but the model never splits “truth” from “guess” on its own. If they're right, we call them truth. And if they're wrong, we call them hallucination. But the model does not internally distinguish between those. Dan Klein, co-founder and CTO of Scaled Cognition I’d stop treating confidence in the UI as a proxy for correctness. Treat every fluent sentence as a claim you still owe evidence for. Obvious mistakes get caught in QA. Klein’s audit story from enterprise work is worse: when his team audits enterprise systems, he says, the real hallucination rate is often about five times what the customer has noticed, because the rest are fluent enough to pass. His examples are a refund policy that is real and plausible but not yours, or yours but for a customer with a different status. Looks right. It looks right and is right are not the same. It is very hard to detect hallucinations when they are packaged up sufficiently fluently. Chip Huyen made the data angle plain on Chain of Thought ep 5: missing or wrong context pushes the model toward invention. Models are more likely to hallucinate when it does not have access to the necessary and correct information to answer questions. Chip Huyen, author of AI Engineering So the design job is to give the model the facts at answer time, allow silence when facts are missing, and verify what still matters. A smarter model does not remove that work. Pre-training shows the model fluent text, not labeled false statements. Rare facts like a specific birthday do not repeat as patterns, so the model still produces something that sounds like an answer. Benchmarks make the incentive worse: “I don’t know” often scores like a wrong answer, while a random guess sometimes hits. OpenAI’s 2025 write-up on why language models hallucinate argues for penalizing confident errors more than honest uncertainty. Anthropic’s interpretability work on Claude describes a default to decline on unrecognized entities, with a “known entity” signal that can flip that off and invite a guess anyway. For your app, that means instructions that forbid uncertainty fight the same scoreboards that trained the model to guess. If your eval sheet marks abstain as a miss, you are training your team to ship guessers. Grounding https://chainofthought.show/glossary/grounding/ means building the reply from material you put in context at question time, ideally with citations per claim. That cuts invented detail because the model has less room to freestyle. It does not guarantee truth: sources can be stale, retrieval can miss the doc, and the model can still stretch past what the sources say. Many production issues that look like “model bugs” are retrieval or document problems first. General models also carry a knowledge cutoff https://chainofthought.show/glossary/knowledge-cutoff/ : training frozen at a date, then released later. Ask about anything newer and you get “I don’t know” or a plausible guess. The standard fix is the same as grounding: fetch current or private text into the context window before generation RAG, search, or both . My default pattern for factual features: Step 4 is where AI evaluation and reliability https://chainofthought.show/topics/evaluation/ work pays off: separate counts for answered , abstained , answered supported and answered unsupported . Label this as illustration, not production code from any guest or vendor. Sketch: abstain if retrieval is weak; answer only from chunks MIN SCORE = 0.42 tune on your corpus def answer user question: str, retrieve, llm, faithfulness check : chunks = retrieve user question, top k=8 best = max c.score for c in chunks , default=0.0 if best < MIN SCORE: return {"status": "abstain", "message": "No trusted source found."} prompt = { "instruction": "Answer using ONLY the sources. " "If they do not support an answer, reply ABSTAIN." , "sources": c.text for c in chunks , "question": user question, } draft = llm.complete prompt if "ABSTAIN" in draft: return {"status": "abstain", "message": "Model declined."} if not faithfulness check draft, chunks : return {"status": "abstain", "message": "Failed source check."} return {"status": "ok", "text": draft, "sources": chunks} The abstain path is a first-class HTTP response, not a thrown exception buried in logs. Users should see “we don’t know” when you mean it. The longer explainer, with the episode clips, is on Chain of Thought https://chainofthought.show/ai-decoded/why-do-llms-hallucinate/ . It draws on this episode https://chainofthought.show/podcast/54-why-llms-are-plausibility-engines-not-truth-engines-dan-klein/ . Subscribe to the Chain of Thought newsletter https://newsletter.chainofthought.show/subscribe?utm source=devto&utm medium=referral&utm campaign=newsletter for new episodes and write-ups like this one. Drafted with AI assistance from the episode transcripts.