{"slug": "what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax", "title": "What Actually Happens When an LLM Generates a Token? A Deep Dive into Softmax", "summary": "An engineer's deep dive explains the mechanics of how large language models generate tokens, focusing on the softmax function that converts raw logits into a probability distribution. The post breaks down the mathematical steps, showing how exponentiation and normalization turn scores like 8.2 for 'Paris' into probabilities such as 66.5%, and clarifies that LLMs are probabilistic in output but deterministic in their internal scoring.", "body_md": "**21:38. Berlin. Tuesday evening.**\n\nI had just come home from a meetup with a question in my head.\n\nIs an LLM probabilistic or not?\n\nBecause, well... somehow yes. But also somehow no.\n\nWhat actually happens between \"the model predicts a token\" and \"the model produces a token\"?\n\nIn summary, an LLM produces a probability distribution over possible next tokens. A decoding strategy then turns that distribution into an actual token. But that explanation hides most of the interesting engineering and actually very interesting mathematics. To fully understand what happens (and to rediscover the beauty of e !), it helps to follow the process one step at a time.\n\nWhen an LLM processes a prompt, it does not simply decide:\n\n```\n\"The next token is Paris.\"\n```\n\nFirst of all, the model takes the input and produces a **score for every possible next token, based on its learned weights**.\n\nSuppose the context is:\n\nThe capital of France is ...\n\nConceptually, the model might produce something like:\n\n```\nParis      8.2\nLondon     6.1\nBerlin     5.4\nPizza     -2.3\n```\n\nThese scores are called **logits**.\n\nA logit is a raw score produced by the model for a possible next token.\n\n**How do those scores become probabilities?** That's where **Softmax** enters.\n\nThe problem with raw logits is that they are not probabilities. They can be positive, negative, large, or small. Within a probability distribution, all probabilities are positive, and all probabilities add up to 100%.\n\nFor example:\n\n```\nParis      70 %\nLondon     20 %\nBerlin     10 %\n```\n\nOur logits don't do that. So we need a transformation that turns them into something that can be interpreted as probabilities. That's what Softmax does.\n\nThe Softmax function is:\n\nAt first glance, the formula might look a little bit intimidating. But it's actually pretty simple, so stay with me. Simply put, the formula does two things in order to transform the logits into probabilities:\n\nFirst, it transforms every logit into a positive value. In addition, it amplifies the relative differences between the values so that the probability distribution gets clearer. That's what the exponential function e^x is for. Finally, the fraction normalizes these values by dividing each one by the sum of all exponentiated logits.\n\nThe result is a probability distribution where all values are positive and add up to 1.\n\nLet's go down the rabbit hole step by step.\n\nThe little\ni\nsimply identifies the **token we're currently looking at**.\n\npi\nmeans **the probability assigned to token\ni\n.**\n\nIf we're calculating the probability of Paris, we write: pi=pParis\n\nNothing mysterious. It's just what we're interested in - the probability.\n\nzi\nis the corresponding **logit** for token\ni\n.\n\nSo if the logit of Paris is 2, we write: zParis=2\n\nStill straightforward.\n\nLook at:\n\nThe symbol\n∑\nsimply means: Add things together. And\nj\nis the **index we use to go through all the tokens we're summing over**.\n\nIn this case:\n\nTake every possible token, calculate ezj , and add all the results together.\n\nIn engineering language this would be:\n\nStart with zero, iterate over all possible next tokens, calculate ezj for each one, and add the result to the total.\n\n```\ntotal = 0\n\nfor j in tokens:\n    total += exp(z[j])\n```\n\nSuppose we have only three tokens:\n\n```\nParis    z = 2\nLondon   z = 1\nBerlin   z = 0\n```\n\nThen:\n\nThat's it.\n\nThe denominator in the Softmax formula is simply the sum of all exponentiated logits.\n\nIn our example, the result would be\n\nNow calculate the probability of Paris (z=2):\n\nSo the probability of Paris is approximately **66.5%**.\n\nThen London (z=1):\n\nSo the probability of London is approximately **24.5%**.\n\nThen Berlin (z=0):\n\nSo the probability of Berlin is approximately **9.0%**.\n\nNotice what happens: You get pretty differentiated results and at the same time, the scores turned to a probability. That is Softmax.\n\nExponentiate every score. Add all those values together. Then divide each individual value by the total.\n\nAnd now the results form a probability distribution.\n\nThe formula only looks intimidating because mathematics is extremely good at packing an entire paragraph into one line.\n\nLook at the following table:\n\n| Token / probability |\nWith exponential\npi=∑jezjezi\n|\nWithout exponential\npi=∑jzjzi\n|\n|---|---|---|\n| pParis | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …\\mathbf{66.5%} | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …\\mathbf{66.7%} |\n| pLondon | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …\\mathbf{24.5%} | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …\\mathbf{33.3%} |\n| pBerlin | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: … \\mathbf{9.0%} | ParseError: KaTeX parse error: Unexpected end of input in a macro argument, expected '}' at end of input: …=\\mathbf{0.0%} |\nSum |\n100.0% |\n100.0% |\n\nIn this particular example, Paris and London still look relatively similar, but Berlin immediately drops to 0% without ex . More importantly, directly normalizing logits only works as a valid probability construction when the logits are non-negative. Real logits can also be negative.\n\nSo the exponential isn't just decoration. It gives us a transformation with the properties we need: positive values and a way of amplifying relative differences before normalization. And that brings us to e .\n\nYou might remember that e≈2.718281828...\n\nYou might also remembered that it had something to do with exponential functions and logarithms. Let's explore why it matters.\n\ne\nnaturally appears when we describe **continuous exponential growth**. And it has one remarkably convenient mathematical property:\n\nThe rate at which ex changes - f′(x) - is always and exactly its current value f(x) . Crazy.\n\nTake this table with x , the values f(x) and their exponential growth f′(x) :\n\n| x | f(x)=ex | f′(x)=ex |\n|---|---|---|\n| 0 | f(0)=1 | f′(0)=1 |\n| 1 | f(1)=2.718 | f′(1)=2.718 |\n| 2 | f(2)=7.389 | f′(2)=7.389 |\n\nThe larger the function becomes, the faster it grows — at a rate proportional to its current value.\n\nDid you notice? The equation is mathematically elegant. Computers, however, have finite-precision numbers. But exponentials grow fast. Very fast. If a logit is 10000, just calculating e10000 is not a nice thing to do to your processor with floating-point arithmetic.\n\nSo real implementations use a mathematically equivalent, numerically stable form of Softmax:\n\nWhy is this allowed?\n\nBecause subtracting the same constant from every logit does not change the resulting Softmax probabilities:\n\nfor any constant (c).\n\nLet's explain this from an engineering perspective: In practice, we want to keep the numbers inside the exponential function as small as possible. Large exponents can cause numerical overflow and make the calculation unnecessarily difficult. So we subtract the **largest logit from every logit**.\n\nIf the largest logit is max(z) , we calculate:\n\nzi′=zi−max(z)\n\nThis makes the largest logit exactly zero:\n\nmax(z)−max(z)=0\n\nAnd because the largest logit was subtracted from all the others, every other adjusted logit is negative:\n\nzi−max(z)≤0\n\nNow look at what happens when we exponentiate them.\n\nThe largest exponent is:\n\ne0=1\n\nAnd all the other exponentials are between 0 and 1:\n\n0<ezi−max(z)<1\n\nSo instead of calculating potentially huge numbers like e1000 , we are always working with values between 0 and 1.\n\n**But why are we allowed to do this?**\n\nBecause - and now we come to the already mentioned mathematical background - subtracting the same constant from every logit does not change the resulting Softmax probabilities:\n\nsoftmax(z)=softmax(z−c)\n\nfor any constant c .\n\nIn our case, we simply choose:\n\nc=max(z)\n\nSo we haven't changed the probabilities we're calculating.\n\n**We've just shifted all logits down so that the calculation stays numerically small and safe.**\n\nAnd since we had already come this far down the rabbit hole: ln is - simply spoken - the mathematical opposite of e : If ex is the operation that takes us from x to exponential growth, ln(x) takes us back again.\n\nFor example, if:\n\nthen:\n\n**Back to the LLM. Back to Softmax.**\n\nLet's do a quick recap. We started with:\n\n```\nParis       2\nLondon      1\nBerlin      0\n```\n\nSoftmax applies ex :\n\nThen it normalizes:\n\nAnd there it is:\n\n```\nParis      66.5 %\nLondon     24.5 %\nBerlin      9.0 %\n```\n\nThe model started with scores (logits). Softmax transformed them. The exponential function turned the relative differences into positive weights. Normalization turned those weights into proportions. And now we have a probability distribution.\n\n**The LLM hasn't randomly chosen anything yet.**\n\nWe have only calculated the distribution.\n\nTemperature doesn't itself introduce randomness. It changes the shape of the probability distribution **before we sample from it**.\n\nThe temperature-adjusted Softmax is:\n\nThe\nT\nstands for **temperature**. It modifies the logits **before** the exponential transformation.\n\nSuppose we have:\n\n```\nParis     2\nLondon    1\n```\n\nThe effect of different temperature values is:\n\n| Temperature T | Paris: 2/T | London: 1/T | Difference |\n|---|---|---|---|\n| 0.5 | 4 | 2 | 2 |\n| 1 | 2 | 1 | 1 |\n| 2 | 1 | 0.5 | 0.5 |\n\n* T=1 means that the logits are not changed by temperature: zi/T=zi .\n\nThen ex acts on these values. Low temperature amplifies the differences between logits. High temperature reduces them. The resulting distribution becomes respectively sharper or flatter.\n\nThe lower the temperature, the larger the difference between the adjusted logits becomes. The exponential transformation then amplifies this difference, resulting in a **sharper probability distribution**. In other words, the probability becomes more concentrated on the token with the higher logit.\n\nVice versa, the higher the temperature, the smaller the difference between the adjusted logits becomes. After the exponential transformation, this results in a **flatter probability distribution**, meaning that the probability is distributed more evenly across the possible tokens.\n\nAnd this is where the role of\ne\nbecomes especially useful: Temperature doesn't directly manipulate the final percentages. It just changes the scores **before the exponential transformation**, where differences can be amplified or compressed.\n\nThe formula says: zi/T\n\nSo if (T=0), aren't we dividing by zero?\n\nYes. We are. But remember Limes, your friend and helper when it comes to division by zero. Because the thing is, the formula is not defined at (T=0). Mathematically, the interesting thing is the **limit as (T) approaches zero from above**:\n\nAgain, suppose we have\n\n```\nParis     2\nLondon    1\n```\n\nThe effect of temperature values approaching zero is:\n\n| Temperature T | Paris: 2/T | London: 1/T | Difference | Paris probability | London probability |\n|---|---|---|---|---|---|\n| 1∗ | 2 | 1 | 1 | 73.1 | 26.9 |\n| 0.5 | 4 | 2 | 2 | 88.1 | 11.9 |\n| 0.1 | 20 | 10 | 10 | 99.995 | 0.005 |\n| 0.01 | 200 | 100 | 100 | ≈100 | ≈0 |\n\nT=1 means that the logits are not changed by temperature: zi/T=zi .\n\nSo the table makes the effect of temperature very visible:\n\nThe values become enormous. But Softmax doesn't care about their absolute size. It cares about their **relative size**.\n\nFor Softmax with temperature T :\n\nWe divide the numerator **and** denominator by\ne2/T\n. This gives:\n\nNow consider the limit: T→0+\n\nThen: T1→∞ and therefore: −T1→−∞\n\nExplanation: As T approaches zero, the value of T becomes smaller and smaller. And because there is a minus sign in front, the exponent becomes increasingly negative.\n\nTherefore: e−1/T→0\n\nExplanation: For a negative exponent, ex produces a value between 0 and 1 . The more negative the exponent becomes, the closer the result gets to zero.\n\nwhich leaves:\n\nFor Softmax with temperature T :\n\nWe divide the numerator **and** denominator by\ne1/T\n. This gives:\n\nNow consider the limit: T→0+\n\nThen: T1→∞\n\nExplanation: As T approaches zero, the value of T becomes smaller and smaller. The exponent therefore becomes increasingly positive.\n\nTherefore:\n\nExplanation: For a positive exponent, ex produces a value greater than 1 . The larger the exponent becomes, the larger the result gets. As the exponent approaches infinity, the value of ex also approaches infinity.\n\nwhich leaves:\n\nAnd there we have the complete result:\n\np1→1 and p2→0\n\nThe important trick is that we divide by the exponential term corresponding to the token we are currently calculating.\n\nFor p1 , dividing by e2/T gives us e−1/T , which approaches 0 .\n\nFor p2 , dividing by e1/T gives us e1/T , which approaches ∞ .\n\nThat makes it explicit why the probability of the larger logit approaches 1 , while the probability of the smaller logit approaches 0 .\n\nAnd there it is: While the absolute numbers themselves don't matter, their **ratio** is becoming decisive.\n\nSoftmax doesn't care how enormous the numbers are. It cares how enormous they are relative to one another.\n\nThis is where the mathematical model and the practical API diverge slightly.\n\nAt (T=0), the Softmax equation itself isn't defined. In practice, a system may interpret `temperature = 0`\n\nas a request for greedy or otherwise non-sampling decoding rather than literally dividing by zero.\n\nConceptually:\n\nrather than:\n\nTemperature zero isn't \"Softmax with zero.\"\n\nAt T=0 , the Softmax formula itself is not defined because we would be dividing by zero.\n\nIn practice, `temperature = 0`\n\nis usually used to tell the system: **Just choose the token with the highest probability.**\n\nThere is one small catch. What if two tokens have exactly the same highest logit?\n\n```\nParis      2\nLondon     2\nBerlin     1\n```\n\nNow Paris and London are tied.\n\nIf we look at what happens as T gets closer and closer to zero, neither one becomes more likely than the other. They both keep the same probability.\n\nThere is no mathematical way for Softmax itself to choose between two exactly equal maximum logits. So the **tie has to be resolved by the decoding algorithm**, not by Softmax.\n\nIf there is no clear winner, the decoding system needs a rule for resolving the tie.\n\nFor example, a system could define a deterministic tie-breaking rule such as:\n\n```\n1. Find the highest logit.\n2. If several tokens have the same highest logit,\n   choose the token with the smallest token ID.\n```\n\nThen:\n\n```\nParis      2   ← token ID 471\nLondon     2   ← token ID 892\nBerlin     1\n```\n\nwould always select Paris.\n\nAt this point, we have a probability distribution.\n\nSuppose:\n\n```\nParis      70 %\nLondon     20 %\nBerlin     10 %\n```\n\nSampling means: **Draw a token according to this distribution.**\n\n| Run | Result |\n|---|---|\n| One run | → Paris |\n| Another | → Paris |\n| Another | → London |\n| Another | → Paris |\n| Another | → Berlin |\n\nThe distribution itself has not changed. The concrete draw can.\n\nA probabilistic model does not necessarily produce a probabilistic output.\n\nSo the variability comes from the sampling process itself: Temperature changes the distribution. Sampling is the random selection.\n\nYou can think about it like this:\n\n```\nProbability distribution\n          ↓\n     Temperature\n          ↓\n\"How strongly should\n the options differ?\"\n          ↓\n       Sampling\n          🎲\n          ↓\n        Token\n```\n\nOr, less formally: **Temperature changes the weights. Sampling rolls the dice.**\n\nWhat exactly does the computer do when we sample?\n\nAs it normally doesn't have access to some magical source of true randomness, a concrete implementation can use a **pseudo-random number generator**, or PRNG. A pseudo-random generator produces a sequence of values from a mathematical algorithm and an initial state, commonly represented by a seed.\n\nAs a consequence, the sampling process can be probabilistic in its behavior while the concrete sequence of pseudo-random numbers is deterministic if the relevant state is fixed.\n\nIf the relevant conditions are identical — including the model parameters, input, generation settings, and random state or seed — the same pseudo-random sequence can be generated again.\n\nSo there are two different perspectives here.\n\nFrom the perspective of the model or user:\n\nSampling behaves probabilistically.\n\nFrom the perspective of a fixed implementation with a fixed pseudo-random state:\n\nThe concrete sequence can be deterministic and reproducible.\n\n\"Random\" does not automatically mean \"irreproducible.\" And \"probabilistic\" does not automatically mean \"nondeterministic at every level of the system.\" **The implementation matters.**\n\nNow the terminology starts to fall into place.\n\nTop-ksays: Only consider the (k) most likely tokens.\n\nFor example, with `top-k = 3`\n\n, only the **three most likely tokens** remain candidates for sampling: **Paris, London, and Berlin**.\n\n| Token | Probability | Top-k = 3 |\n|---|---|---|\n| Paris | 50% | ✓ |\n| London | 25% | ✓ |\n| Berlin | 15% | ✓ |\n| Madrid | 7% | — |\n| Rome | 3% | — |\n\nTop-pworks differently: It keeps thesmallest set of tokens whose cumulative probability mass is at least (p), after sorting candidates by probability in descending order.\n\nFor example, with `top-p = 0.8`\n\n, we keep adding the most likely tokens until the **cumulative probability reaches at least 80%**.\n\n| Token | Probability | Cumulative probability | Top-p = 0.8 |\n|---|---|---|---|\n| Paris | 50% | 50% | ✓ |\n| London | 25% | 75% | ✓ |\n| Berlin | 15% | 90% | ✓ |\n| Madrid | 7% | 97% | — |\n| Rome | 3% | 100% | — |\n\nHere, Paris + London gives us 75%, so we need Berlin to reach 90%.\n\nTherefore, **Paris, London, and Berlin remain candidates**.\n\nTop-k fixes the **number of candidates**. Top-p fixes the **probability mass**. Both can constrain the sampling space.\n\nThe model produces the scores. Softmax turns them into a probability distribution. Temperature reshapes it. Top-k and Top-p can constrain the candidates. And sampling selects what comes next.\n\nWhat initially sounded like one operation — *the LLM generates the next token* — is actually a chain of very different operations.\n\n**Once you follow a single token through that chain, the process becomes much less mysterious.**\n\nThere is no single \"generation\" step. There is a sequence of mathematical transformations and a decoding decision at the end.", "url": "https://wpnews.pro/news/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax", "canonical_source": "https://dev.to/younic/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax-4g48", "published_at": "2026-08-21 22:23:15+00:00", "updated_at": "2026-08-21 22:43:55.323561+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax", "markdown": "https://wpnews.pro/news/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax.md", "text": "https://wpnews.pro/news/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax.txt", "jsonld": "https://wpnews.pro/news/what-actually-happens-when-an-llm-generates-a-token-a-deep-dive-into-softmax.jsonld"}}