# Eron on X: "LLMs do one thing: predict the next word > Source: > Published: 2026-08-20 11:28:26+00:00 Every time you send a message to ChatGPT, Claude or Gemini the same thing happens. The model looks at everything you wrote and makes one decision: what word probably comes next. Then it does it again. And again. Until the response is complete. That's it. That's the whole trick. And somehow from that one absurdly simple task repeated billions of times on an incomprehensible amount of text something emerges that can write code, explain quantum physics, translate poetry and hold a conversation that feels remarkably like talking to a person. Here's what's actually happening inside. The number that puts everything in perspective GPT-3 trained on enough text that if a human tried to read all of it without stopping - no sleep, no breaks, twenty four hours a day - it would take over 2,600 years. Larger models since then trained on significantly more. This matters because the model didn't learn language the way you learned it. Nobody sat down and wrote rules. Nobody explained grammar or meaning or context. The model was shown an almost incomprehensible volume of text and given one instruction: get better at predicting what comes next. Everything it knows about language, facts, reasoning, tone and personality emerged from that single optimization target applied at a scale that is genuinely difficult to hold in your head. What a parameter actually is The model's behavior is determined entirely by numbers. Hundreds of billions of them. These are called parameters or weights and they're the only thing that makes one model different from another. At the start of training every parameter is set to random values. The model produces pure gibberish. Then training begins. The process works like this. Take a piece of text. Feed all of it except the last word into the model. Look at what the model predicted would come next. Compare that to the actual last word. Measure how wrong the model was. Then adjust every one of those hundreds of billions of parameters slightly in the direction that would have made the model less wrong. Do this for trillions of examples. The math involved is staggering. If you could perform one billion calculations per second and you tried to replicate the computation required to train the largest language models it would take you over 100 million years. This is only possible because of specialized chips called GPUs that run enormous numbers of operations simultaneously. After training is complete those hundreds of billions of parameters encode - in a form that nobody fully understands - something about how language works, what things mean, how concepts relate and what kinds of responses are appropriate in what kinds of situations. How a single prediction becomes a conversation When you send a message to a chatbot here is exactly what happens. The system takes the full context of your conversation, adds a description of the AI's role and purpose, and feeds all of it into the model. The model looks at this entire block of text and predicts the most likely next word. That word gets added to the text. Then the whole thing runs again. And again. Each prediction is probabilistic. The model doesn't pick the single most likely next word every time. It samples from a distribution. The most likely words get picked most often but occasionally a less likely word gets selected. This is why asking the same question twice gives you different answers and why responses feel natural rather than mechanical. Pure maximum-likelihood prediction produces stilted repetitive text. Sampling from the distribution produces something that reads like a person wrote it. The temperature setting you sometimes see in AI tools controls this distribution. Low temperature makes the model more deterministic and focused. High temperature makes it more varied and creative and eventually incoherent if pushed too far. The architecture that made everything possible Before 2017 language models processed text word by word from start to finish. Each word was handled sequentially. This created a fundamental problem with long texts because information from the beginning of a passage had to travel through every intermediate step to influence the end. In 2017 a team at Google published a paper called Attention Is All You Need. The architecture they introduced - the transformer - changed everything. Transformers don't read text sequentially. They take in the entire input at once and process it in parallel. Every word can directly attend to every other word simultaneously. The beginning of a long passage influences the end as directly as words that appear right next to each other. The key mechanism is called attention. Every word in the input gets represented as a long list of numbers. Then attention lets every word examine every other word and update its own representation based on the context around it. The word bank means something different in the phrase river bank than in the phrase bank account. Attention allows the model to distinguish between these meanings by examining the surrounding words and adjusting what the number representation of bank actually encodes. This runs through many layers. Each layer refines the representations based on context. By the final layer the goal is that each word's representation encodes whatever information might be useful for predicting what comes next - including information pulled from across the entire input, however long it is. What the network is actually made of A neural network is a system of connected mathematical operations loosely inspired by how neurons work in the brain. The basic unit is a neuron. A neuron holds a number. Its value is determined by taking a weighted sum of the values of the neurons connected to it from the previous layer, adding a constant called a bias and passing the result through a function that keeps the output within a certain range. The weights on those connections are the parameters. During training the weights get adjusted to make the network better at its task. In the context of image recognition you can see this concretely. A network trained to recognize handwritten digits starts by processing individual pixel values. The first hidden layer might learn to detect edges. The next layer combines edges into shapes. The final layer combines shapes into digit identities. Nobody programmed any of this. It emerged from training. Language models work at a much larger scale with much more complex structure but the same basic principle applies. The model learns internal representations that are useful for its task. Those representations aren't designed. They emerge. The training that makes it useful Pre-training on internet text teaches the model to predict language. But predicting random internet text is not the same as being a useful assistant. A model trained only on pre-training data might produce text that continues whatever pattern it was given - including producing harmful content, wrong information or responses that are technically fluent but completely unhelpful. To address this chatbots undergo a second training stage called reinforcement learning from human feedback. Human raters evaluate different responses and indicate which ones are better. This feedback gets used to further adjust the model's parameters to make it more likely to produce responses that humans prefer. This is why models refuse certain requests, try to be helpful rather than just fluent, and generally behave more like assistants than autocomplete systems. The pre-training gave them knowledge of language and the world. The second training stage shaped how they apply that knowledge. What the model actually learned Here is something that is genuinely strange about how these systems work. When a neural network trained to recognize handwritten digits is visualized - when you look at what patterns each neuron has learned to respond to - the result is not always what you'd expect. Instead of clearly learning edges and then shapes the network often learns patterns that don't have obvious interpretations. They work but not in the way you might have hoped. The same is true for language models at a much larger scale. Researchers can observe that the models do something that looks like reasoning, something that looks like analogy, something that looks like understanding context. But exactly how those behaviors emerge from the underlying mathematics is not well understood even by the people who built these systems. The models are not programmed to reason. They were trained to predict text. The reasoning - if that's what it is - emerged from that training at scale. This is both what makes these systems impressive and what makes them unreliable in ways that are hard to predict. The model that can explain a complex concept clearly might also confidently state something completely false. It learned what fluent helpful text looks like. Whether the content of that text is accurate is a different question. The thing nobody fully understands If you train a large language model on random labels - if you scramble all the correct answers before training - the model can still achieve perfect accuracy on the training data. It just memorizes. It doesn't generalize. If you train on correctly labeled data the model learns much faster and generalizes to examples it has never seen. The structure in the data makes the patterns easier to find. This tells you something important. These models are not simply memorizing. They are finding structure. What that structure corresponds to - whether it's something like understanding or something that merely resembles it - is a question that researchers are actively working on and haven't answered. What is clear is that next-word prediction at sufficient scale on sufficient data produces systems that can do things nobody explicitly programmed them to do. Write code. Translate languages. Summarize documents. Answer questions about topics that appear nowhere in their training data. Why this is the beginning not the end The transformer architecture from 2017 enabled parallel processing of text at scale. The scale of computation required to train the largest models is measured in operations that would take millions of years to perform by hand. The emergence of useful capabilities from next-word prediction alone surprised even the researchers who built these systems. None of this was obvious before it worked. The idea that predicting text could produce something capable of writing software or explaining science or engaging in coherent multi-turn conversation was not something anyone predicted with confidence. It worked because language contains within it an enormous amount of structure about the world. To predict what word comes next in a description of a physics experiment you need to understand something about physics. To predict the next word in a piece of code you need to understand something about what the code is doing. The model was never taught physics or programming directly. It learned whatever it needed to learn to get better at predicting text. One job. Predict the next word. Do it with hundreds of billions of parameters on trillions of examples. Something that feels like thinking falls out the other side. Nobody is entirely sure why. / If this was useful - follow, the next one drops here first.