Chinchilla is quoted as a rule about how many tokens to train on. That is the part that stopped being followed. The reason it stopped is a different piece of arithmetic that nobody published as a headline, and it is short enough to do here.
An empirical relationship between resources and pretraining loss. Across many training runs at different sizes, loss falls as a power law in parameters, in data and in compute — meaning that on log-log axes the points fall on a line, and that each constant multiple of a resource buys a constant subtraction from the loss.
Two consequences are worth internalising before any specific result. Diminishing returns are built in: the step from 1B to 10B parameters and the step from 10B to 100B buy the same amount of loss. And the quantity being predicted is loss on held-out text, which is not the same as anything you care about — a point the emergence debate is largely about.
Kaplan et al. (2020) fitted the first widely-used set of these laws and concluded that, given more compute, most of it should go into a bigger model. The generation of very large, comparatively lightly-trained models followed that advice.
Hoffmann et al. (2022) — the Chinchilla paper — re-ran the analysis with a learning-rate schedule matched to each run’s length, which the earlier work had not done, and reached a different conclusion: parameters and training tokens should scale roughly together. Their headline ratio is about twenty training tokens per parameter at the compute-optimal point. The paper’s demonstration was that a model trained this way outperformed a substantially larger, less-trained contemporary at the same training compute — which reframed the previous generation as undertrained rather than merely large.
The result has itself been scrutinised. Besiroglu and colleagues (2024) attempted to replicate the parametric fit in the Chinchilla paper and reported that the published coefficients were not consistent with the paper’s own reported data, with the reconstructed fit implying a somewhat different optimal ratio and tighter confidence intervals than stated. The qualitative conclusion — that the previous generation was badly undertrained and that data should scale with parameters — survives this; the precise ratio is softer than its constant quotation suggests. Treat “twenty tokens per parameter” as an order of magnitude, not a target.
Two approximations do all the work here, and both are standard. Label them as approximations: they ignore attention’s quadratic term, which is small relative to the matrix multiplies at typical sequence lengths, and they ignore everything about memory bandwidth and utilisation.
training FLOPs ~= 6 N D N = parameters, D = training tokens
(2 forward + 4 backward, per parameter per token)
inference FLOPs ~= 2 N per token generated
Chinchilla’s rule is D ≈ 20N, so a compute-optimal 70B model trains on about 1.4 trillion tokens: 6 × 7.0e10 × 1.4e12 ≈ 5.9e23 FLOPs.
Chinchilla answers: given a fixed training budget, what is the best loss I can reach? That is the right question for a research result and the wrong question for a product, because a served model pays 2N FLOPs per token for the rest of its life. Total lifetime cost is:
total ~= 6 N D_train + 2 N D_serve
Now compare two ways of reaching a given quality bar. Take the compute-optimal 70B above, and a 7B model trained far past its compute-optimal point — Meta stated publicly that Llama 3 was trained on over 15 trillion tokens, so that regime is real, not hypothetical.
N D_train training FLOPs FLOPs per served token
A (optimal) 70e9 1.4e12 5.9e23 2 x 70e9 = 1.4e11
B (over- 7e9 15e12 6.3e23 2 x 7e9 = 1.4e10
trained)
Training cost: comparable.
Serving cost: 10x apart, on every token, forever.
The assumption doing the work — and it must be labelled, because this page did not measure it — is that A and B reach comparable quality on the tasks you care about. They do not in general; overtraining buys less per token as it goes, and the two models will differ. But the direction is what matters, and the direction is unambiguous: once serving volume is large, moving down in parameters and up in training tokens converts a one-time training cost into a permanent reduction in cost per request. That is why small, heavily-overtrained models became the commercially interesting size, and it happened without anyone publishing a paper called “inference-optimal scaling”.
Sparsity attacks the same equation from the other side, by making the N in the serving term smaller than the N in the checkpoint — which is exactly what mixture of experts buys.