# Let your LLM say I don't know instead of guessing

> Source: <https://dev.to/conorbronsdon/let-your-llm-say-i-dont-know-instead-of-guessing-2dog>
> Published: 2026-11-01 20:30:07+00:00

Your user asks a factual question. The model returns a polished paragraph. Nothing in the stack threw an error. That is exactly when you are most exposed. A language model generates the most plausible next words, not the most true ones, so a wrong answer can look identical to a right one. That mechanism is what people call [AI hallucination](https://chainofthought.show/glossary/ai-hallucination/): same token prediction, missing or weak facts, fluent fiction at the end.

[Dan Klein](https://chainofthought.show/guests/dan-klein/), co-founder and CTO of Scaled Cognition, put the internals this way on Chain of Thought: we label outputs after the fact, but the model never splits “truth” from “guess” on its own.

If they're right, we call them truth. And if they're wrong, we call them hallucination. But the model does not internally distinguish between those.

*Dan Klein, co-founder and CTO of Scaled Cognition*

I’d stop treating confidence in the UI as a proxy for correctness. Treat every fluent sentence as a claim you still owe evidence for.

Obvious mistakes get caught in QA. Klein’s audit story from enterprise work is worse: when his team audits enterprise systems, he says, the real hallucination rate is often about five times what the customer has noticed, because the rest are fluent enough to pass. His examples are a refund policy that is real and plausible but not yours, or yours but for a customer with a different status.

Looks right. It looks right and is right are not the same. It is very hard to detect hallucinations when they are packaged up sufficiently fluently.

Chip Huyen made the data angle plain on Chain of Thought ep 5: missing or wrong context pushes the model toward invention.

Models are more likely to hallucinate when it does not have access to the necessary and correct information to answer questions.

*Chip Huyen, author of AI Engineering*

So the design job is to give the model the facts at answer time, allow silence when facts are missing, and verify what still matters. A smarter model does not remove that work.

Pre-training shows the model fluent text, not labeled false statements. Rare facts (like a specific birthday) do not repeat as patterns, so the model still produces something that sounds like an answer. Benchmarks make the incentive worse: “I don’t know” often scores like a wrong answer, while a random guess sometimes hits. OpenAI’s 2025 write-up on why language models hallucinate argues for penalizing confident errors more than honest uncertainty. Anthropic’s interpretability work on Claude describes a default to decline on unrecognized entities, with a “known entity” signal that can flip that off and invite a guess anyway.

For your app, that means instructions that forbid uncertainty fight the same scoreboards that trained the model to guess. If your eval sheet marks abstain as a miss, you are training your team to ship guessers.

[Grounding](https://chainofthought.show/glossary/grounding/) means building the reply from material you put in context at question time, ideally with citations per claim. That cuts invented detail because the model has less room to freestyle. It does not guarantee truth: sources can be stale, retrieval can miss the doc, and the model can still stretch past what the sources say. Many production issues that look like “model bugs” are retrieval or document problems first.

General models also carry a [knowledge cutoff](https://chainofthought.show/glossary/knowledge-cutoff/): training frozen at a date, then released later. Ask about anything newer and you get “I don’t know” or a plausible guess. The standard fix is the same as grounding: fetch current or private text into the context window before generation (RAG, search, or both).

My default pattern for factual features:

Step 4 is where [AI evaluation and reliability](https://chainofthought.show/topics/evaluation/) work pays off: separate counts for `answered`, `abstained`, `answered_supported` and `answered_unsupported`.

Label this as illustration, not production code from any guest or vendor.

```
# Sketch: abstain if retrieval is weak; answer only from chunks

MIN_SCORE = 0.42  # tune on your corpus

def answer(user_question: str, retrieve, llm, faithfulness_check):
    chunks = retrieve(user_question, top_k=8)
    best = max((c.score for c in chunks), default=0.0)
    if best < MIN_SCORE:
        return {"status": "abstain", "message": "No trusted source found."}

    prompt = {
        "instruction": (
            "Answer using ONLY the sources. "
            "If they do not support an answer, reply ABSTAIN."
        ),
        "sources": [c.text for c in chunks],
        "question": user_question,
    }
    draft = llm.complete(prompt)
    if "ABSTAIN" in draft:
        return {"status": "abstain", "message": "Model declined."}

    if not faithfulness_check(draft, chunks):
        return {"status": "abstain", "message": "Failed source check."}

    return {"status": "ok", "text": draft, "sources": chunks}
```

The abstain path is a first-class HTTP response, not a thrown exception buried in logs. Users should see “we don’t know” when you mean it.

The longer explainer, with the episode clips, is on [Chain of Thought](https://chainofthought.show/ai-decoded/why-do-llms-hallucinate/). It draws on [this episode](https://chainofthought.show/podcast/54-why-llms-are-plausibility-engines-not-truth-engines-dan-klein/).

[Subscribe to the Chain of Thought newsletter](https://newsletter.chainofthought.show/subscribe?utm_source=devto&utm_medium=referral&utm_campaign=newsletter) for new episodes and write-ups like this one.

*Drafted with AI assistance from the episode transcripts.*