{"slug": "a-quick-ish-chinchilla-check", "title": "A quick(ish) Chinchilla check", "summary": "Giles Thomas, a developer, tested the Chinchilla scaling rule by comparing overtrained GPT-2 style models (trained on 40 tokens per parameter) against a model scaled up in parameters and tokens equally. The Chinchilla heuristic held, but the improvement was surprisingly small and possibly within noise. The experiment involved three 163M-parameter models and one larger model, with results suggesting that overtraining can be a viable strategy when model size is constrained.", "body_md": "I recently [overtrained a couple of GPT-2 style models](https://www.gilesthomas.com/2026/07/why-do-openai-gpt2-weights-beat-mine-3-overtraining),\ntraining them both on 40 tokens per parameter rather than the 20 per parameter\nthat is generally regarded as \"Chinchilla-optimal\".\n\nThe normal heuristic is that instead of doing that, you should scale up the number of tokens and the number of parameters equally -- so I would have been better off scaling up the model by and the token count by the same amount. By doing that, I should expect to get a better model in terms of loss on my held-back test set than I did with my 40-tokens-per-parameter models.\n\nMy training machine `poppy`\n\nwasn't doing anything, so I decided to give that a go.\nWould the Chinchilla rule-of-thumb hold up?\n\nAs you might expect, it did. But it was a surprisingly close-run thing, and could conceivably have been in the noise. Let's take a look.\n\nIf you already know all about the Chinchilla paper -- regular readers in particular must be sick and tired of it by now :-) -- then\n\n[click here to skip this section].\n\nIn [\"Training Compute-Optimal Large Language Models\"](https://arxiv.org/pdf/2203.15556),\nwhich is always called the Chinchilla paper after the name of the model they trained at\nthe end, the authors tried to work out the optimal number of tokens to train an LLM on\nbased on its number of parameters. In particular, they were pushing back on a trend\nthey were seeing at the time, where people were making models ever-larger, but not\nincreasing the amount of data they were training on.\n\nThe authors were all at Google DeepMind, and this was the kind of project that only a large lab could do: they trained \"over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens\". Their conclusion was \"for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled\".\n\nThey don't actually state an overall optimal number of tokens to train on in the paper, but in table 3 they provide an estimate of the optimal training FLOPs and tokens for models of various sizes, and it's approximately 20 tokens per parameter.\n\nThat number has become a heuristic, and people talk about a model as being trained for the Chinchilla-optimal number of tokens. Models that were trained on fewer tokens per parameter are referred to as \"undertrained\", and models that were trained on more as \"overtrained\".\n\nIt's worth noting that overtraining a model is not, in itself, a bad thing. If you have a model of a particular size and you continue training it past the Chinchilla-optimal number of tokens, it will -- in general -- get better. The point of the heuristic is that doing that is not the best way to spend whatever budget you have in terms of compute time. You'll get better results, as they say, by scaling the number of tokens and the number of parameters equally.\n\nBut let's say you're creating a model for specific target hardware -- say, a mobile device.\nYou have a hard restriction on how large the model can be -- the device has only so much\nRAM to hold it. So it might make sense to overtrain to get a better model. [1](https://www.gilesthomas.com/feed/rss.xml#fn-1)\n\nBut if you're not so limited in how many parameters you can use, then you should indeed scale the model up, and that's what I wanted to try. How would that work?\n\nA week or two back, I was investigating whether I could make my GPT-2 style models\n[better at a specific instruction-following task by overtraining them](https://www.gilesthomas.com/2026/07/why-do-openai-gpt2-weights-beat-mine-3-overtraining).\nThe details of that experiment aren't important here, but what it meant was that\nI had three GPT-2-style models, each of exactly the same size, roughly 163M parameters\n\n`jax-gpt2-chinchilla`\n\nhere.`jax-gpt2-2x-chinchilla`\n\n.`jax-gpt2-2-epoch-chinchilla`\n\nWhen I tested them against a held-back test set of sequences -- stuff that they'd never seen before -- they got results rather like you might expect:\n\n| Test loss | |\n|---|---|\n`jax-gpt2-2x-chinchilla` |\n3.324953 |\n`jax-gpt2-2-epoch-chinchilla` |\n3.326482 |\n`jax-gpt2-chinchilla` |\n3.418784 |\n\nA lower loss is better, and you can see that the longer-trained models were noticeably better than the Chinchilla-optimal one. The difference between them was tiny; they were trained starting with the same initial weights, and the training runs themselves were deterministic, but a difference of 0.05% in loss doesn't seem like it could be meaningful -- an extra batch for one or one fewer for the other could easily swap them around, you'd think.\n\nNow, these models each had 163,009,536 parameters -- they were the small-size model\nfrom the [GPT-2 paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf),\nmodified to not have QKV bias or weight-tying. `jax-gpt2-chinchilla`\n\nhad been trained\non 3,260,190,720 tokens (rounded up to fit into a round number of full batches), and\nthe other two on 6,520,381,440 tokens each -- double the amount (rounded up too).\n\nWhat I needed to do for my Chinchilla check was to try training a model that used the same amount of compute, scaling the parameters and the number of training tokens equally. Because training compute increases roughly linearly with both parameters and tokens, that would mean scaling both up by , giving us:\n\n...and thus 4,610,605,920 tokens.\n\nHow to scale the model up?\n\nIn the GPT-2 paper, they train four models:\n\n| Name | Parameters\n|\nLayers | `d_emb` |\nMHA heads\n|\n|---|---|---|---|---|\n| small | 124M | 12 | 768 | 12 |\n| medium | 345M | 24 | 1024 | 16 |\n| large | 762M | 36 | 1280 | 20 |\n| xl | 1542M | 48 | 1600 | 25 |\n\nI wanted to scale my own model up from 163M parameters to about 231M. Which of those numbers would I want to increase, and by how much?\n\nThe first thing that stands out is that the number of heads is *always* 1/64th of\nthe number of embedding dimensions. So that sorted that one out. I just needed to\nadjust the number of layers, and the number of embedding dimensions, but ensure that the\nlatter was a multiple of 64.\n\nI decided to see if I could fit some kind of curve to the relationship between\nthe number of parameters and the GPT-2 authors' choices. This was made a bit more\ncomplicated by one thing: they were using weight-tying, and I was not. That meant\nthat they re-used the embedding matrix at the start of the LLM as an output head\nat the end -- which is why they had 38M fewer parameters. Embeddings and the\noutput head make up a [surprisingly large percentage of the parameters](https://www.gilesthomas.com/2026/07/llm-parameter-counts) for\nsmall models like this -- about 47% without weight-tying, 23% with.\n\nI couldn't work out a solid way to scale things up and wound up doing some rather\nmessy [hacking around in a spreadsheet](https://docs.google.com/spreadsheets/d/1rgzoc7-bQfK43XQpp00htEgQ43b_ahtBwdrb_HfmIZk/edit?usp=sharing).\nI came up with two proposed model sizes that were within a couple of percentage\npoints of the right size:\n\n| Name | Layers | `d_emb` |\nMHA heads | Parameters | % diff |\n|---|---|---|---|---|---|\n`slightly-larger` |\n15 | 896 | 14 | 235,621,120 | +2.21% |\n`slightly-smaller` |\n14 | 896 | 14 | 225,978,368 | -1.97% |\n\nInterestingly, I found that because `d_emb`\n\ncould only change in increments/decrements\nof 64, it was a pretty coarse control -- my first attempt at making a `slightly-smaller`\n\nmodel changed it to the next step down, 832, but that led to a model that was\n9.25% too small.\n\nThat was an interesting first lesson. I'd previously been thinking of the Chinchilla rule as being something like \"don't double the tokens, just scale the model and the tokens equally\". But that \"just\" was wrong. Scaling a model is hard -- even with just two dials to fiddle with, like in this case, it was tricky to get something right -- and I can't say for sure that my choices were the right ones.\n\nAnyway, the next step was to double-check that these models would use the right amount of compute to train.\n\nAs I said earlier, the compute time scales roughly linearly with the number of parameters. Let's dig into that \"roughly\".\n\nDifferent kinds of parameters take different amounts of FLOPs to train, and scale differently with things like the embedding dimensions, sequence length, and so on.\n\nNow, for very large models, a lot of that comes out in the wash, but with tiny models like these where the embeddings make up such a large proportion of the parameters, it might matter.\n\nConveniently, in appendix F of the Chinchilla paper, they provide a set of formulae for estimating the\nnumber of training FLOPs for a normal dense LLM like these ones. I coded that up into\n[a script](https://github.com/gpjt/chinchilla-flop-calculator) that, given the JSON\nconfiguration files I was using for my models and training runs, would work out the\nnumber of FLOPs for a single epoch of training. It didn't take account of the fact that\nmy real training runs round the number of tokens up so that we do a round number of\nfull batches, but I felt that so long as the results weren't *very* close that wouldn't matter.\n\nI got these results (multiplying the two-epoch numbers by two):\n\n| Est. training FLOPs | |\n|---|---|\n`jax-gpt2-chinchilla` |\n3,544,967,596,946,227,200 |\n`jax-gpt2-2x-chinchilla` |\n7,089,935,193,892,454,400 |\n`jax-gpt2-2-epoch-chinchilla` |\n7,089,935,193,892,454,400 |\n`slightly-larger` |\n7,419,664,885,127,577,600 |\n`slightly-smaller` |\n6,804,429,215,367,168,000 |\n\nThe numbers were indeed different enough that I wasn't worried about the batch-rounding.\nAnd the good news was that `slightly-larger`\n\nand `slightly-smaller`\n\nwould indeed use\nslightly more and slightly less compute to train than the overtrained models --\nabout 4.6% more and 4% less respectively. A true Chinchilla-equivalent model would\nlie somewhere between them.\n\nIt was time to train some models!\n\nI kicked off the run for the `slightly-larger`\n\nmodel first. Because it was bigger than the\n163M models I'd been training, I couldn't\nfit such large batches into my VRAM; previously I'd been running with a batch size of\n6, and now I could only fit in a batch of 4. Luckily, though, I was using\n[gradient accumulation](https://www.gilesthomas.com/2026/04/llm-from-scratch-32k-interventions-training-our-best-model-locally-gradient-accumulation),\nso by bumping that up from 16 steps to 24 steps I could keep the same\noverall batch size and keep the training runs comparable.\n\nEven despite that, the training run ran out of VRAM about 60 hours in -- I'm guessing\ndue to VRAM fragmentation, as I did not have `TF_GPU_ALLOCATOR`\n\nset to `cuda_malloc_async`\n\n--\nbut I was able to restart from the most recent checkpoint and complete the run.\nAfter just less than four days total training time, it completed.\n\nWhen it was done, I copied the last checkpoint 4 over to my dev box,\n\n`perry`\n\n, and\nran my standard smoke test against it, asking\nit to complete \"Every effort moves you\" with 20 tokens, using greedy sampling. I got\nsomething reasonably coherent:\n\n```\nEvery effort moves you.\nI’m not sure what you’re thinking.\nI’m\n```\n\nNext, I converted the safetensors file -- which had been saved by my JAX code -- into a format compatible with my PyTorch code, because that's what I use for evals. I ran another smoke test (this one with temperature 1):\n\n```\nEvery effort moves you through the motions for your life, your soul, your body,\nand your soul’s happiness\n```\n\nVery spiritual. Next, it was time to work out the loss on my held-back test set:\n\n``` bash\ngiles@perry:~/Dev/ddp-base-model-from-scratch (main)$ uv run test_loss.py datasets/ ../jax-gpt2-from-scratch/runs/full-llm-full-train-with-mha-output-bias-larger-chinchilla/model.json ../jax-gpt2-from-scratch/runs/full-llm-full-train-with-mha-output-bias-larger-chinchilla/checkpoints/latest/pytorch-model.safetensors\nFetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 3485.09it/s]\n100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3200/3200 [07:11<00:00,  7.42it/s]\nLoss against our test dataset: 3.280028\n```\n\nWell, it was certainly better than the 3.324953 that the best of the overtrained models got -- but only by a bit over 1% better. Interesting!\n\nI decided to train the second model, `slightly-smaller`\n\n. This one crashed mid-way\nthrough with an error that I've seen before:\n\n```\njax.errors.JaxRuntimeError: INTERNAL: CUDA error: Failed to end stream capture: CUDA_ERROR_STREAM_CAPTURE_INVALIDATED: operation failed due to a previous error during capture [executable_name='jit_train_step']\n```\n\nI'm going to have to investigate that more in future, but for now, I just restarted from the checkpoint, and again after a bit less than four days, I had a model.\n\nThe JAX smoke test was solid:\n\n```\nEvery effort moves you forward.\nThe best way to get started is to start with a free trial.\nYou can\n```\n\n...and so was the PyTorch one:\n\n```\nEvery effort moves you forward in love with our products.\nI love the way it’s easy to use.\n```\n\nBoth quite commercial this time! It was time for the proper test loss eval:\n\n``` bash\ngiles@perry:~/Dev/ddp-base-model-from-scratch (main)$ uv run test_loss.py datasets/ ~/Dev/jax-gpt2-from-scratch/runs/full-llm-full-train-with-mha-output-bias-larger-chinchilla-2/model.json ~/Dev/jax-gpt2-from-scratch/runs/full-llm-full-train-with-mha-output-bias-larger-chinchilla-2/checkpoints/latest/pytorch-model.safetensors\nFetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 3151.24it/s]\n100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3200/3200 [06:45<00:00,  7.90it/s]\nLoss against our test dataset: 3.292937\n```\n\nSo, slightly worse than the 3.280028 from the larger model, better than the 3.324953 from the best overtrained one.\n\nTime to put this all together.\n\nHere's an updated version of the table from the start of this post; I've added in\nthe two new models, and the improvement they each had over `jax-gpt2-2x-chinchilla`\n\nin both absolute terms and as a percentage rounded to 3sf.\n\n| Test loss | Improvement | Improvement % | |\n|---|---|---|---|\n`slightly-larger` |\n3.280028 | 0.044925 | 1.35% |\n`slightly-smaller` |\n3.292937 | 0.032016 | 0.962% |\n`jax-gpt2-2x-chinchilla` |\n3.324953 | - | - |\n`jax-gpt2-2-epoch-chinchilla` |\n3.326482 | - | - |\n`jax-gpt2-chinchilla` |\n3.418784 | - | - |\n\nNow, unlike the overtrained models, prior to training these two new ones started with different\ninitial weights to the `jax-gpt2-chinchilla`\n\none -- after all, they had to, because\nthey had more of them!\n\nA while back, I did a [bit of analysis](https://www.gilesthomas.com/2026/04/llm-from-scratch-32i-interventions-what-is-in-the-noise)\nof how random variation in weight initialisation can change the resulting test loss.\nIt wasn't anything in-depth, but I trained three models with different explicit seeds\nset prior to the model initialisation, but with the same seed set before the training\nrun started 5.\n\nThose three models wound up with test losses of 3.681356, 3.673943, and 3.664345. Doing statistics with three data points is a bit flaky, but the cost of training models is so high that I'll leave the Proper Science to the likes of Google DeepMind and wing it :-)\n\nNow, piling statistical flakiness on statistical flakiness, we'll compare these. You'd normally expect about two thirds of results to be within one SD of the mean, 95.4% to be within two SDs, and 99.7% to be within three.\n\nThree SDs on that (yes, different, I know) distribution is 0.025587. That's smaller than both of the improvements that our Chinchilla-optimal runs had over the overtrained ones.\n\nSo what does that tell us? Well, perhaps not much given the statistical flakiness. But I think it is useful\ndirectionally. It suggests that we might be able to take these results seriously\nas an improvement, and that Chinchilla held: scaling up the model and the number of\ntokens evenly did give us a better model than just scaling up the number of tokens.\nIn particular, the fact that the loss for `slightly-smaller`\n\nwas lower -- even though\nit had 4% *less* compute spent on it than the overtrained models -- was encouraging.\n\nBut it's certainly far from a slam-dunk. A larger test, training lots of overtrained models and lots of Chinchilla-optimal ones, all with different random seeds, would give actual real serious data.\n\nNot worth it for me, and perhaps not for anyone.\n\nI wanted to do a quick sanity check of the Chinchilla heuristic of 20 tokens per parameter. I came up with results that were certainly in line with it -- perfectly so in terms of the ordering of the models I trained. But the effect was small enough that I could imagine that it was in the noise, especially given the small numbers of models I'm able to train. I'll chalk it up as a tentative success.\n\nIn addition, I learned one useful thing: when talking about scaling up a model to more parameters, you actually have to think quite hard about where you want to put those parameters. I wound up doing a rough curve-fit to the models in the GPT-2 paper, but I have no idea if that was optimal. At some point I should try to dig up some research into optimising embedding dimensions, numbers of layers, and so on. But not now, as I've a bunch of other stuff I want to investigate first.\n\nAnyway, I hope you found this experiment interesting, and as ever, comments and questions welcome below. Thanks for reading!\n\nI'm less familiar with arguments for under-training -- that is, for fewer than\n20 tokens per parameter. I've heard that these days, modern LLMs get a lot more\nreinforcement learning than they do pre-training, and perhaps that might mean that\nsome very big ones are undertrained prior to RL? I'm uncertain. It's unlikely\nto be raw lack of data; even for those of us outside the big labs,\n[FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) has 18.5T tokens.\nOn its own, that would be enough to train a 0.925T-parameter model, and given that\nyou can [apparently](https://arxiv.org/abs/2305.16264) do four epochs over the\nsame data before you start getting diminishing returns, that takes us up to\n3.7T. That's frontier-lab size, and I'm sure they have better datasets than FineWeb. [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-1)\n\nParameter counts are from the paper, apart from the \"small\" model, which is\nknown to be wrong -- I used my own calculation, and the result is in line with what I've\nseen elsewhere. [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-2)\n\nThe paper doesn't mention the number of heads; these numbers are from\n\"[Build a Large Language Model (from Scratch)](https://www.manning.com/books/build-a-large-language-model-from-scratch)\",\nand match up with the ones on [this Hugging Face page](https://huggingface.co/transformers/v2.2.0/pretrained_models.html). [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-3)\n\nRegular readers might have noticed that I'm ignoring what I've been calling the \"best\" checkpoint.\nI've come to the conclusion that because for my training script, \"best\" means best in\nterms of training loss, and the training loss changes based on what training\ndata the model has seen recently, it's actually not a very useful metric and\njust confuses things. At some point I'll probably re-introduce pre-checkpoint\nevals and use that for \"best\", which would be the right way to do it. [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-4)\n\nAt the time I was using dropout, so training runs were not deterministic without\na known seed. [↩](https://www.gilesthomas.com/feed/rss.xml#fnref-5)", "url": "https://wpnews.pro/news/a-quick-ish-chinchilla-check", "canonical_source": "https://www.gilesthomas.com/2026/08/chinchilla-check", "published_at": "2026-08-07 19:00:00+00:00", "updated_at": "2026-08-09 09:55:52.845965+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Giles Thomas", "Google DeepMind", "GPT-2", "Chinchilla"], "alternates": {"html": "https://wpnews.pro/news/a-quick-ish-chinchilla-check", "markdown": "https://wpnews.pro/news/a-quick-ish-chinchilla-check.md", "text": "https://wpnews.pro/news/a-quick-ish-chinchilla-check.txt", "jsonld": "https://wpnews.pro/news/a-quick-ish-chinchilla-check.jsonld"}}