Pretraining is the first stage of training a language model. Its main task is to predict the next token. Take “The capital of France is ___.” A pretrained model reads “The capital of France is” and predicts the next token, “Paris.”
The model makes a prediction, compares it with the real token, calculates the error, and updates its weights.
To pretrain a model, you repeat this step over and over, on a huge pile of text, code, and other data:
After pretraining, the model has picked up language, facts, code, and common reasoning patterns from that data.
Pretraining is only the first stage. Compare it with the two stages that usually follow:
| Stage | Goal | Data | Feedback signal |
|---|---|---|---|
| Pretraining | Learn general language patterns | Huge, mixed text and code | Next-token prediction error |
| Fine-tuning | Learn a specific task or format | Small, curated examples | Difference from a labeled output |
| RL | Learn a preferred behavior | The model’s own outputs | A reward score |
Pretraining teaches a model what patterns exist in its data. Feed it a lot of code, and it gets better at code. Feed it many tool-call examples, and tool use can become a natural part of its output.