Andrej Karpathy said it plainly in 2023: language models do not know they are wrong. They have no internal signal that flags uncertainty. They generate the most probable continuation of whatever you gave them, and they do it with the same confidence whether the output is correct or completely fabricated.
That is not hallucination. That is how the architecture works.
The word "hallucination" implies the model drifted on its own - that it wandered into fiction unprompted. That framing lets you off the hook. The more accurate framing does not.
Large language models are next-token predictors. At every step, the model produces a probability distribution over the entire vocabulary and samples from it. The output that emerges is the sequence that seemed most likely given everything before it. There is no lookup table, no database of facts it checks against. It is pattern completion operating at scale.
When the model produces something wrong, it is not because it had a moment of confusion. It is because the probability distribution it built from your prompt pointed toward that output. The wrong answer was the most likely answer given the input you provided.
This distinction matters because it changes where you look when things go wrong. If the model hallucinated, there is nothing you can do - it is a flaw in the system. If the model guessed badly because you gave it a vague prompt, that is your problem to fix and you can fix it right now.
I benchmark AI models professionally at Datawise. We run structured evaluations across dozens of tasks. The pattern that shows up most consistently: outputs that look wrong are almost always responding to inputs that were underspecified. The model gave a reasonable answer to the question that was actually asked, not the question the engineer thought they asked.
These are different questions:
The first prompt has six implied decisions the model has to guess. The second prompt has none.
The difficulty of writing a specific prompt is the difficulty of knowing what you actually need. If you cannot write the specific prompt, you do not yet know what you need. That is useful information - it means you should stop prompting and start thinking.
What makes this genuinely tricky is that LLMs produce wrong outputs with the same fluency and confidence as correct ones. The prose sounds authoritative. The code looks clean. There is no stutter, no hedge, no signal that says "I am filling in a gap here."
This is where experience matters. A junior engineer reads the output and trusts it because it looks right. A senior engineer reads the output and asks: where did I leave room for interpretation? Every ambiguous word in the prompt is a decision the model made without you. Every missing constraint is a place where probability took over.
When an output is wrong, the most common response is to resubmit with a slightly different wording and hope for a different result. Sometimes that works. More often it does not, because the problem was not the phrasing - it was the missing context.
Retrying without fixing the specification is the AI equivalent of restarting a service without checking the logs. You might get lucky. You have not fixed anything.
When an AI output is wrong, read your prompt before you rewrite it. Ask where you left room for interpretation. Add the missing constraints. Be specific about inputs, outputs, error handling, dependencies, and edge cases before you ask for the implementation.
A useful habit: before submitting a prompt, reread it as if you were a new engineer joining the project with no context. What would you have to guess? Everything you would have to guess is a place the model will guess too.
The model is not lying to you. It is showing you the shape of what you did not specify. Once you see it that way, the fix is always the same.
Write tighter prompts.