Understanding the mechanics of LLM generation explains why

A technical analysis of large language model (LLM) generation explains that hallucination is a predictable mathematical byproduct of token sampling, not a bug. The piece details how temperature settings reshape probability distributions, with high temperature increasing creativity but also hallucination risk, and describes the shift from dense models to Mixture of Experts (MoE) architecture. The author, who ran a Python simulation, notes that greedy decoding always picks the most likely token, leading to repetitive outputs, while sampling randomness introduces variability.

Understanding the mechanics of LLM generation explains why I've been digging into the mechanics of token selection and the Mixture of Experts MoE architecture, and it becomes clear that hallucination isn't a bug—it's a predictable mathematical byproduct of how these models function. The math behind word selection When you hit "send" on a prompt, the model doesn't just "know" the answer. At every single step of the generation process, the model produces a probability distribution across its entire vocabulary. It's essentially handing you a ranked list of every possible next token and assigning a percentage to each. The way we choose from that list is called the sampling strategy. If you always pick the 1 most likely word, you are using Greedy Decoding . It's efficient and logical, but it's incredibly boring. If you used greedy decoding for everything, every AI-written story would eventually loop into the same repetitive phrases. To add variety, we use Temperature . Think of temperature as a way to reshape that probability distribution before we pick a word: Low Temperature < 1.0 : This sharpens the distribution. The most likely words get even more weight, and the unlikely ones get crushed. This makes the model focused and predictable—perfect for coding or factual extraction. High Temperature 1.0 : This flattens the distribution. The gap between the "likely" and "unlikely" words shrinks, giving the model a higher chance of picking something unexpected. This is great for creative writing but is exactly where hallucinations start to ramp up. I ran a quick Python simulation to show how these probabilities look in a raw state. Here is a mock distribution representing what a model might output after the prompt "the cat sat on the": python import numpy as np np.random.seed 7 A mock probability distribution over a tiny vocabulary vocabulary = "mat", "roof", "moon", "table", "keyboard", "president" probabilities = np.array 0.45, 0.20, 0.15, 0.12, 0.06, 0.02 print "Vocabulary and their probabilities:" for word, prob in zip vocabulary, probabilities : print f" {word:12s} {prob:.2f}" Running that gives us a clear hierarchy: Vocabulary and their probabilities: mat 0.45 roof 0.20 moon 0.15 table 0.12 keyboard 0.06 president 0.02 If we apply a simple greedy decoding function to this: python def greedy decode vocabulary, probabilities : best index = np.argmax probabilities return vocabulary best index for i in range 5 : print f"Attempt {i+1}: {greedy decode vocabulary, probabilities }" The output is always identical: Attempt 1: mat Attempt 2: mat Attempt 3: mat Attempt 4: mat Attempt 5: mat This illustrates why purely deterministic models feel robotic. To get a "human" flow, we need to introduce sampling randomness. Why hallucination is a feature, not a bug This brings us to the uncomfortable truth: hallucinations are a direct consequence of this sampling process. When we increase temperature to make a model more creative, we are explicitly telling it to consider less probable tokens. A hallucination happens when the model follows a high-probability path of syntax the sentence sounds grammatically perfect but a low-probability path of factuality . Because the model is essentially a sophisticated autocomplete, it prioritizes the "flow" of the next token based on its training data. If the most statistically "likely" next word in a sentence structure is a factually incorrect noun, the model will grab it without hesitation. The rise of Mixture of Experts MoE While the sampling process handles the output, the internal architecture is changing how we scale these models. We are moving away from "dense" models where every parameter is used for every prompt toward Mixture of Experts MoE . In a dense model, if you ask a math question or a poetry question, the entire brain fires. In an MoE architecture—which powers some of the biggest names in the industry—the model is divided into specialized sub-networks the "experts" . A "router" mechanism looks at your prompt and decides which specific experts are best suited to handle it. This allows for massive parameter counts trillions of weights while keeping the actual computational cost FLOPs per token relatively low, because only a fraction of the model is actually "awake" for any given word. This architecture is the current frontier for making high-performance LLM agents viable in real-world deployment. Next OpenAI just dropped GPT-5.6 Sol and it changes the vision game → /en/threads/7526/