I've been digging into the mechanics of token selection and the Mixture of Experts (MoE) architecture, and it becomes clear that hallucination isn't a bug—it's a predictable mathematical byproduct of how these models function.
The math behind word selection #
When you hit "send" on a prompt, the model doesn't just "know" the answer. At every single step of the generation process, the model produces a probability distribution across its entire vocabulary. It's essentially handing you a ranked list of every possible next token and assigning a percentage to each.
The way we choose from that list is called the sampling strategy. If you always pick the #1 most likely word, you are using Greedy Decoding. It's efficient and logical, but it's incredibly boring. If you used greedy decoding for everything, every AI-written story would eventually loop into the same repetitive phrases.
To add variety, we use Temperature. Think of temperature as a way to reshape that probability distribution before we pick a word:
Low Temperature (< 1.0): This sharpens the distribution. The most likely words get even more weight, and the unlikely ones get crushed. This makes the model focused and predictable—perfect for coding or factual extraction.High Temperature (> 1.0): This flattens the distribution. The gap between the "likely" and "unlikely" words shrinks, giving the model a higher chance of picking something unexpected. This is great for creative writing but is exactly where hallucinations start to ramp up.
I ran a quick Python simulation to show how these probabilities look in a raw state. Here is a mock distribution representing what a model might output after the prompt "the cat sat on the":
import numpy as np
np.random.seed(7)
vocabulary = ["mat", "roof", "moon", "table", "keyboard", "president"]
probabilities = np.array([0.45, 0.20, 0.15, 0.12, 0.06, 0.02])
print("Vocabulary and their probabilities:")
for word, prob in zip(vocabulary, probabilities):
print(f" {word:12s} {prob:.2f}")
Running that gives us a clear hierarchy:
Vocabulary and their probabilities:
mat 0.45
roof 0.20
moon 0.15
table 0.12
keyboard 0.06
president 0.02
If we apply a simple greedy decoding function to this:
def greedy_decode(vocabulary, probabilities):
best_index = np.argmax(probabilities)
return vocabulary[best_index]
for i in range(5):
print(f"Attempt {i+1}: {greedy_decode(vocabulary, probabilities)}")
The output is always identical:
Attempt 1: mat
Attempt 2: mat
Attempt 3: mat
Attempt 4: mat
Attempt 5: mat
This illustrates why purely deterministic models feel robotic. To get a "human" flow, we need to introduce sampling randomness.
Why hallucination is a feature, not a bug #
This brings us to the uncomfortable truth: hallucinations are a direct consequence of this sampling process. When we increase temperature to make a model more creative, we are explicitly telling it to consider less probable tokens.
A hallucination happens when the model follows a high-probability path of syntax (the sentence sounds grammatically perfect) but a low-probability path of factuality. Because the model is essentially a sophisticated autocomplete, it prioritizes the "flow" of the next token based on its training data. If the most statistically "likely" next word in a sentence structure is a factually incorrect noun, the model will grab it without hesitation.
The rise of Mixture of Experts (MoE) #
While the sampling process handles the output, the internal architecture is changing how we scale these models. We are moving away from "dense" models (where every parameter is used for every prompt) toward Mixture of Experts (MoE).
In a dense model, if you ask a math question or a poetry question, the entire brain fires. In an MoE architecture—which powers some of the biggest names in the industry—the model is divided into specialized sub-networks (the "experts"). A "router" mechanism looks at your prompt and decides which specific experts are best suited to handle it.
This allows for massive parameter counts (trillions of weights) while keeping the actual computational cost (FLOPs) per token relatively low, because only a fraction of the model is actually "awake" for any given word. This architecture is the current frontier for making high-performance LLM agents viable in real-world deployment.
Next OpenAI just dropped GPT-5.6 Sol and it changes the vision game →