Language Modeling as Next-Token Prediction

Language modeling is the task of predicting the next token in a sequence, a formulation that the chain rule of probability shows is sufficient to capture all statistical regularities in language. This approach, used by modern LLMs like GPT and LLaMA, traces back to Claude Shannon's 1948 information theory and his experiments estimating English entropy at 0.6–1.3 bits per character.

Language Modeling as Next-Token Prediction Welcome to Arc 2. In Arc 1 we built up the mathematical tools: vectors, matrices, probability distributions, cross-entropy, gradients, optimizers, floating point. Now we get to use them. This arc covers how people modeled language before transformers, because every design choice in modern LLMs is a reaction to something that came before. If you skipped Arc 1, you can probably still follow along, but the cross-entropy post /blog/cross-entropy-and-loss and the probability distributions post /blog/probability-distributions are the most directly relevant prerequisites. So. What is a language model? A language model is a system that assigns a probability to the next token, given all the tokens that came before it. Given a sequence of words or subwords, or bytes , it outputs a probability distribution over what comes next. Every GPT, every LLaMA, every Claude, every n-gram model from the 1990s does some version of this. The architectures are wildly different, but the objective is the same: predict the next token from context. BERT and its descendants use a related but distinct objective, masked language modeling, where you predict held-out tokens from their bidirectional surroundings rather than predicting strictly left-to-right. We'll come back to that in Arc 4. That objective is doing more work than it looks like. The Chain Rule: Why This Works Why does predicting the next token let you model entire sequences? Because of a fact from probability theory that we covered in Post 1.3 /blog/probability-distributions : the chain rule. Any joint probability over a sequence of tokens can be decomposed into a product of conditionals: Or more compactly: This is an identity, not an approximation. It's true for any distribution over sequences, no matter how complex. And it says something important: if you can model each conditional accurately, you can reconstruct the full joint distribution over any sequence. The chain rule guarantees that a next-token predictor is, in principle, sufficient to capture any pattern in language. Here's what that factorization looks like in practice, for the first few words of a sentence: tP w t| w <t 1| ∅ 2| the 3| the cat Each step is a conditional distribution over the vocabulary. Multiply the chosen probabilities together and you get P sequence . The task "predict the next word" sounds like autocomplete, like the suggestion bar on your phone. The chain rule tells us it's actually a universal density estimator for sequences. If your model is expressive enough and you train it on enough data, minimizing the next-token prediction loss should in theory learn every statistical regularity in the training distribution. Syntax, semantics, world knowledge, reasoning patterns. All of it shows up as patterns in what comes next. Shannon's Guessing Game Claude Shannon understood this in 1948. His paper "A Mathematical Theory of Communication" is where entropy, cross-entropy, and most of information theory come from. But he also played a revealing game. In a 1951 follow-up paper, he had human subjects try to guess the next letter in English text, one character at a time. People could predict the next letter with high accuracy, especially after seeing a few characters of context. After "the United Sta", the next letter is almost certainly "t". After a space following a period, it's probably a capital letter. After "q", it's almost certainly "u". Shannon used these experiments to estimate the entropy of English at roughly 0.6 to 1.3 bits per character. If you already have enough context, the average English character carries only about 1 bit of genuine surprise. The rest is redundant, predictable from what came before. That redundancy is what language models exploit. A perfect language model would compress English text to about 1 bit per character. A bad one wastes bits on things it should have predicted. The gap between the model's predictions and reality is exactly what cross-entropy measures, as we covered in Post 1.4 /blog/cross-entropy-and-loss . Here's an animated version of Shannon's game, one letter at a time. Watch the distribution get peakier and the "bits this step" number fall as the context grows. After "the q" the next letter is essentially forced, so the entropy drops to near zero. After a space, we're back near 3-4 bits, because any word could start next. The running average settles in the neighborhood Shannon estimated from his human subjects. Per-letter entropy over a short sentence. The distribution starts wide, collapses when context determines the next letter after 't', after 'q' , and reopens whenever a new word starts. The running average creeps toward Shannon's ≈1-bit estimate as context accumulates. If you want to feel what "bits of entropy" actually mean in the abstract, here's the distribution-to-entropy gauge from Arc 1 again. Drag the concentration slider or press play and watch it breathe : a uniform distribution over 5 outcomes has bits of entropy, and a one-hot distribution has zero. Everything a language model does is a push along that axis, from "I have no idea what comes next" toward "I know exactly which token comes next." Entropy as a function of distribution shape. Peaky distributions carry less entropy because the outcome is more predictable. Shannon's 1-bit-per-character estimate means that after enough context, English text is about as predictable as a mild coin flip per character. Perplexity: Making the Numbers Interpretable We introduced perplexity briefly in the cross-entropy post, but let's revisit it. It's been the standard evaluation metric for language models since the 1970s IBM speech-recognition days, and the intuition is cleaner than cross-entropy. Perplexity is just the exponentiated cross-entropy: where is the cross-entropy of the model with respect to the true data distribution , measured on some held-out test set. In practice, since we compute cross-entropy using natural log in PyTorch, you'll often see: where "loss" is the average cross-entropy loss per token. The intuition is that perplexity is the effective number of tokens the model is choosing between at each step. If a model has perplexity 10 on English text, it's as uncertain, on average, as if it were uniformly guessing among 10 equally likely next tokens. Slide the cross-entropy below to see where different regimes land: Perplexity is just 2^ cross-entropy . 1 bit ↔ perplexity 2 coin flip ; 8 bits ↔ perplexity 256 barely better than random over a small vocab . Some rough landmarks to calibrate your intuition: | Model | Approx. Perplexity | What it means | |---|---|---| | Uniform random 50k vocab | 50,000 | Knows nothing | | Unigram word frequencies only | ~1,000 | Knows which words are common | | Trigram + Kneser-Ney | ~80-150 | Knows short local patterns | | LSTM LM 2016-era | ~30-50 | Captures medium-range dependencies | | GPT-2 1.5B 2019 | ~37 on WikiText-103 | Better long-range coherence | | Frontier closed models 2026 | not publicly comparable | Internal pretraining loss is almost certainly far below GPT-2-era models, but a clean apples-to-apples perplexity number depends on tokenizer, benchmark, context length, and whether the model is base or post-trained, and frontier vendors generally don't publish those numbers | A uniform random model over a 50,000-word vocabulary has perplexity 50,000. It has no idea what comes next. A model with perplexity 30 has narrowed it down to "about 30 plausible options" at each position. That's a lot of learned structure. Perplexity for the latest closed models isn't really a published number anymore; the concept still matters, but the comparable numbers stop somewhere around the open-weight LMs you can actually run a held-out evaluation on. A Worked Example Let me make this concrete. Consider the sentence: "the cat sat on the mat" Let's compute the probability of this sentence under three different models: a unigram model, a bigram model, and a hypothetical "good" model. Then we'll compare their perplexities. Unigram model. A unigram model ignores all context. It just knows how frequent each word is. Let's say in our training corpus: | Word | P word | |---|---| | the | 0.07 | | cat | 0.002 | | sat | 0.001 | | on | 0.03 | | mat | 0.0005 | The unigram probability of the sentence is: Per-token cross-entropy using log base 2 : Perplexity: Bigram model. A bigram model conditions on the previous word. With reasonable bigram probabilities from a corpus: | Bigram | Probability | |---|---| | the start | 0.15 | | cat given the | 0.01 | | sat given cat | 0.15 | | on given sat | 0.30 | | the given on | 0.25 | | mat given the | 0.005 | Per-token cross-entropy: Perplexity: The bigram model's perplexity dropped from 137 to 15 just by looking at the previous word. That single word of context made a huge difference. "sat" after "cat" is much more likely than "sat" in isolation. A "good" model. A stronger language model can assign higher probabilities because it uses the full context, not just the previous token: | Position | P w t | context | |---|---| | the | 0.15 | | cat | 0.04 | | sat | 0.25 | | on | 0.50 | | the | 0.60 | | mat | 0.20 | Per-token cross-entropy: Perplexity: Three models, same sentence, wildly different numbers. The viz below stacks them together. The blue bars show the probability each model assigns to the correct next token at every position, and the red bars show the per-token surprise . Watch the red bars shrink and flatten as the model improves. That shrinkage is where the perplexity drop comes from. Same sentence, three models. Autoplay cycles through unigram → bigram → good model. The red surprise bars collapse as the model gains context: the unigram bars are tall and ragged, the bigram bars are mixed, the good model's are short and even. Running perplexity falls from ≈137 to ≈4.7. Here's what those probability assignments look like side by side at a single position. The blue bars show how a "good" model distributes probability at position 5 predicting "the" after "cat sat on" , and the red bars show what a bigram model does with only the previous word "on": Predicting the 5th word after 'the cat sat on'. The good model is very confident in 'the'. The bigram model, which can only see 'on', spreads probability more widely. The good model assigns 60% to "the" because it has seen the full phrase "the cat sat on" and knows what typically follows. The bigram model can only see "on" and has to guess more broadly. That difference in confidence is exactly what perplexity captures. Implementation: A Minimal Bigram LM Here's a complete bigram language model with perplexity evaluation. This is the simplest possible language model that actually conditions on context. The whole thing fits on a screen. Training is just counting pairs, and evaluation just looks up conditional probabilities, averages the log-probs, and exponentiates. The +1 in the numerator is add-one Laplace smoothing, the crudest way to handle unseen bigrams. We'll see much better smoothing in the next post /blog/ngrams-and-sparsity . Why Not Just Use Accuracy? Why train language models with cross-entropy loss on the predicted distribution , rather than just scoring accuracy "did the model guess the right word" ? Two reasons. First, accuracy is not differentiable. It's a step function: right or wrong, 1 or 0. No gradient to flow backward, so you can't train a neural network with it. Second, accuracy throws away information about confidence. If the true next word is "mat" and Model A assigns it 51% while Model B assigns it 99%, accuracy scores both as perfect. Cross-entropy sees the difference. It rewards high confidence in the correct answer and heavily penalizes confident wrong answers because . That makes it a much richer training signal. The viz below morphs between the two models. Both are "correct" under an accuracy metric. The barely-winning distribution carries over 1 bit of loss per token; the confident one is near zero. That gap is the gradient signal that makes training possible. Two models, both 100% accurate on the top class. The confident one left state loses ~0.01 bits per token; the barely-winning one right state loses over a full bit. Accuracy would call these equivalent. Cross-entropy sees a chasm. The Sufficiency Argument One more point about next-token prediction, because it's easy to understate. The chain rule decomposition above means that any probability distribution over sequences of tokens can be exactly represented as a product of next-token conditionals. No loss of generality. You're not approximating anything by decomposing a sequence distribution into per-position predictions. So the task "predict the next token as well as possible" is sufficient to learn everything learnable about the distribution. Syntax, grammar, facts about the world, reasoning patterns, stylistic conventions, code structure. If any of these regularities help predict what token comes next in a large enough corpus, a sufficiently powerful model trained on next-token prediction will learn to exploit them. This is why GPT-style models can do so many things without being explicitly trained for any of them. Translation? It helps predict what comes after "Translate the following French to English:". Arithmetic? It helps predict what comes after "What is 347 + 829? The answer is". Code completion? Same principle. The diversity of capabilities emerges because the training data contains diverse patterns, and next-token prediction incentivizes the model to internalize all of them. Here's that idea made literal. Same objective, same model, different context prefixes. Watch the peak of the next-token distribution land on the "right answer" for each task, no fine-tuning required. One objective argmax next-token probability , five very different contexts. The same system that picks 'dog' after a French-to-English prompt picks '42' after an arithmetic prompt and '1' after a base-case return. Many capabilities are induced by the context prefix rather than by a separate task-specific head. Of course, "sufficiently powerful" and "large enough corpus" are doing a lot of work in those sentences. The objective is not nearly as narrow as "autocomplete" makes it sound: in principle, next-token prediction can represent any distribution over sequences. As a sequence-modeling objective, it is far more general than it first appears. In practice, the bottlenecks move to data, capacity, optimization, architecture, context, and post-training. Universality of the decomposition is not the same as sufficiency for robust reasoning, truthfulness, agency, alignment, or out-of-distribution generalization. Those are real constraints that the rest of the series will keep coming back to. Misconceptions "Next-token prediction is just autocomplete." Your phone's autocomplete uses a small n-gram or shallow neural model that predicts the most likely next word. GPT-4 is also predicting the next token. Same objective in both cases. What differs is model expressiveness and training data scale. At sufficient scale, "autocomplete" becomes a system that has internalized enough statistical structure of language to do translation, reasoning, and code generation. "Perplexity tells you how good a model is at real tasks." Perplexity measures how well the model's distribution matches the test data distribution. That's correlated with downstream task performance, but the correlation is noisy. A model can have excellent perplexity and still generate bland, repetitive text. A model with slightly worse perplexity might generate more interesting and useful responses after RLHF fine-tuning. Perplexity is the canonical metric for evaluating next-token modeling quality on a held-out text distribution. It's a useful proxy for base-model quality, but not a complete measure of general capability, which also depends on data mix, reasoning robustness, tool use, instruction following, calibration, code and math benchmarks, and post-training behavior. We'll dig into this more in Arc 8 when we cover evaluation. "You need a different objective for each NLP task." This was the standard assumption before the GPT era. If you wanted a model for sentiment analysis, you trained a sentiment classifier. If you wanted translation, you trained a seq2seq model. GPT-2 and then GPT-3 showed that a single model trained on next-token prediction could do many tasks via prompting, without task-specific training. The field moved from "one objective per task" to "one objective, many tasks." That shift flows directly from the sufficiency of next-token prediction. What's next The next post covers n-gram models and the curse of sparsity , what happens when you try to do next-token prediction with nothing but counting, why the count table is almost entirely zeros no matter how much data you throw at it, and the smoothing tricks that kept statistical NLP alive until neural models arrived. Additional reading and watching - Shannon, C.E. 1948 . A Mathematical Theory of Communication https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf . Bell System Technical Journal , 27 3 , 379-423. The foundational paper for information theory, entropy, and the mathematical framework for thinking about language as a probabilistic source. - Shannon, C.E. 1951 . Prediction and Entropy of Printed English https://www.princeton.edu/~wbialek/rome/refs/shannon 51.pdf . Bell System Technical Journal , 30 1 , 50-64. Shannon's human guessing-game experiments that estimated the entropy of English at ~1.0-1.3 bits per character. - Kaplan, J., et al. 2020 . Scaling Laws for Neural Language Models https://arxiv.org/abs/2001.08361 . Showed that loss scales as a power law with model size, dataset size, and compute, with predictable returns. - Hoffmann, J., et al. 2022 . Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556 . The "Chinchilla" paper. Showed that previous models were significantly undertrained relative to their size, and that data and parameters should scale together. - Wei, J., et al. 2022 . Emergent Abilities of Large Language Models https://arxiv.org/abs/2206.07682 . Documents capabilities that appear suddenly at certain model scales, though see Schaeffer et al. 2023 , "Are Emergent Abilities of Large Language Models a Mirage?" for a counter-argument. - Radford, A., et al. 2019 . Language Models are Unsupervised Multitask Learners https://cdn.openai.com/better-language-models/language models are unsupervised multitask learners.pdf . The GPT-2 paper. Showed that a single next-token-prediction model could do reading comprehension, translation, and summarization zero-shot. - Brown, T., et al. 2020 . Language Models are Few-Shot Learners https://arxiv.org/abs/2005.14165 . The GPT-3 paper. Demonstrated that in-context learning from next-token prediction scaled smoothly with model size across a wide range of benchmarks. - Jelinek, F., Mercer, R. L., Bahl, L. R., & Baker, J. K. 1977 . Perplexity — a measure of the difficulty of speech recognition tasks https://pubs.aip.org/asa/jasa/article-abstract/62/S1/S63/642598 . JASA . The original introduction of perplexity as an evaluation metric for language models in the speech recognition community.