Next-Token Prediction: How an AI Actually Writes Text (Not Magic — Just Probability)

📺 Prefer to watch?

[90-second YouTube Short]· 💬[Telegram]

Originally published on software-engineer-blog.com.

Ask an AI the same question twice. Get two different answers. That's not a glitch you tolerate — it's the entire mechanism working exactly as designed.

Start below the buzzword: a language model never sees a finished sentence. It only ever answers one tiny question, over and over — given everything written so far, what's the next chunk of text?

One-line mental model: the model outputs a probability over every possible next token → it samples from that distribution instead of always grabbing the top score → the winning token gets glued onto the text → the exact same question runs again from scratch, one token at a time.

Say DraftPal, a writing assistant, is finishing: "The cat sat on the ___." It doesn't know the ending. It computes one probability for every possible next token it knows about:

mat    → 41%
chair  → 19%
floor  → 12%
...    → (thousands more, trailing to ~0%)

That's it. That's the entire "intelligence" at this step — a ranked list over the whole vocabulary, built fresh from the text so far.

Here's the detail that explains half the "weird" behavior people notice about LLMs: the model does not deterministically pick mat

because it's the highest score. It samples — a weighted die roll across that entire distribution. 41% wins most of the time. Sometimes chair

wins instead. Same model, same prompt, different word — because the die was rolled, not read off a table.

Whatever wins gets glued onto the text, and the whole question — "given everything so far, what's next?" — runs again from scratch, now one token longer. One token, one roll, repeat. That loop, run a few hundred times, is what writes an entire reply.

tokens = tokenize(prompt)
while not done:
    distribution = model(tokens)      # probability over every next token
    next_token = sample(distribution) # NOT always argmax
    tokens.append(next_token)

Add four words of context before the same question — "write this like a horror story" — and the exact same probability computation comes back totally different: mat

collapses under 1%, coffin

jumps to 99%. Nothing about the model changed. The input context changed, so the distribution it computes changed.

This one mechanism quietly explains two things developers run into constantly:

The determinism/variety trade-off. Force the model to always take the top slot (greedy decoding, or "temperature 0") and answers get boringly identical every run — useful when you need reproducibility, e.g. structured extraction. Leave sampling on and you get natural variety, at the cost of never getting the exact same output twice.

The compute cost is per token, not per reply. Every single token — not the whole response — costs one full forward pass through the model. A 500-token answer is roughly 500 times more expensive than a 1-token answer, not "a bit more." This is also why streaming feels slow on long outputs: you're watching the loop happen in real time.

No going back — the seed of a hallucination. Once a token is glued onto the context, it is never revised. The model doesn't get to reconsider token 40 after generating token 41. So one confident wrong guess early on doesn't get corrected — the next question is now "given everything so far, including that wrong guess, what's next?" — and the model builds forward on its own mistake. That's the actual mechanical origin of a hallucination: not "the model lied," but "the model committed to a token and the loop only moves forward."

If you've ever looked at an inference dashboard, two metrics show up everywhere: TTFT (time-to-first-token) and TPOT (time-per-output-token). This loop is exactly what they're measuring.

That's also why batching exists as a serving technique: since each loop iteration is bottlenecked on the model's weights into the GPU's compute units rather than on the arithmetic itself, serving frameworks pack multiple users' next-token requests into the same forward pass so one expensive weight-load produces many tokens at once. And it's why response length is the single biggest lever on cost and latency in any LLM product — you are quite literally paying for the number of times the loop above has to run.

Not a sentence writer. A next-token predictor, running in a loop — one probability chart, one weighted roll, one token glued on, repeated until it samples a stop.

Two devs run the same prompt through the same model. One gets "…sat on the mat," the other gets "…the windowsill." Neither is wrong. That's not inconsistency — that's the mechanism.

Want the full walkthrough with the running example built out end to end?

[Watch the long-form video.]Or the[90-second cut]if you just want the core loop.