Quadrillion-parameter transformers and how to train them

A new paper by Baron et al. (2026) reports a scaling law showing that when training data is held fixed, compute-optimal model size keeps growing with compute budget, reaching roughly 7,000x Chinchilla's 20-tokens-per-parameter ratio at 10^29 FLOPs on a 5T-token dataset. The law predicts loss from unique tokens, model size, epochs and ensemble members, adding an ensembling term to prior data-constrained scaling laws, and the authors say their hyperparameter law stays near-optimal at 16x the fitted compute. The work argues that in a multi-epoch, data-constrained regime, parameter counts should rise by orders of magnitude and that bigger models on repeated data need much more weight decay.

Quadrillion-parameter transformers and how to train them TL;DR Bigger models on repeated data need much more weight decay. Our hyperparameter law stays near-optimal at 16× the fitted compute. With scaled hyperparameters and ensembling, loss keeps falling with compute on fixed data. Prior recipes get worse. 1.Introduction The 1,000,000x sample efficiency gap between AI and humans is one of the central mysteries of the field, with no good explanation. One clue is model size: Chinchilla’s prescription for 30T tokens is a ~1.5T-parameter model, a tokens per parameter of 20:1 Hoffmann et al., 2022 ref-hoffmann-2022 . Humans, by contrast, have ~100T parameters counting each synapse as a weight and learn from just ~200M tokens, a tokens per parameter of 1:500,000. The tokens per parameter gap between AI and humans is thus 10,000,000x, and an important step forward for the sample efficiency gap may be to close it. On 30T tokens, we should be training quintillion-parameter neural networks. Our current networks are way too small, by seven orders of magnitude. Our new paper https://arxiv.org/abs/2610.37842 Baron et al., 2026 ref-paper makes progress towards this vision. Extrapolating its scaling law to a dataset of 5T tokens, the compute-optimal parameter count reaches ~7,000x the standard Chinchilla prescription at 10<sup 29</sup FLOPs. Given enough compute, we give prescriptions for training quadrillion-parameter networks. 2.Parameters are all you need Our new paper Baron et al., 2026 ref-paper finds that when the dataset is held fixed, the compute-optimal model size keeps growing with the compute budget. In this regime, we also want to increase the number of epochs and the number of ensemble members with the compute budget. By contrast, Chinchilla prescribes a fixed model size with a fixed dataset: 20 tokens per parameter. Extrapolating our paper’s prescriptions to 5T tokens, we find that the ensemble holds over 300 parameters per token at 10<sup 29</sup FLOPs, nearly 7,000x Chinchilla’s parameters-per-token ratio, and the count keeps growing with compute. We should be increasing parameter counts by orders of magnitude to achieve optimal performance in the multi-epoch regime. Compute is growing much faster than the supply of high quality language training data Villalobos et al., 2024 ref-villalobos-2024 , so language models will be increasingly trained in a multi-epoch data-constrained regime. To study this regime directly, we train transformers on fixed datasets, across a range of data sizes, model sizes, and epoch count, and fit how loss depends on each. Each run also uses snapshot ensembling, averaging the predictions of checkpoints saved during training Huang et al., 2017 ref-huang-2017 ; Mandal et al., 2026 ref-mandal-2026 , which lowers loss at no extra training compute. While snapshot ensembling is a bit unusual, we find that ensembling is an important technique to integrate in a first-class way within the data-constrained regime. Concretely, we fit a single scaling law that predicts the loss $L$ from four quantities: the number of unique tokens in the dataset $U$, the model size $N$, the number of epochs $R$, and the number of ensemble members $M$. The law extends the existing data-constrained scaling laws Muennighoff et al., 2023 ref-muennighoff-2023 ; Lovelace et al., 2026 ref-lovelace-2026 with one new term for ensembling: The first three terms come from Chinchilla: we have an irreducible cost, a term for the size of the model, and a term for the size of the dataset. The fourth term is an overfitting penalty following Lovelace et al. 2026 ref-lovelace-2026 , and the fifth is the gain from ensembling. It is zero for a single model and grows with each added model towards a ceiling. The constants and exponents are fit to our runs. 3.Larger models for more epochs Prior data-constrained scaling laws hold weight decay fixed across scales Lovelace et al., 2026 ref-lovelace-2026 . These pre-existing laws predict loss increasing with compute, and don't consider that we can use larger models with more regularization as part of the scaling law. Existing transfer rules don’t fix this problem. MuP covers learning rate, but not weight decay Yang et al., 2022 ref-yang-2022 . The weight decay heuristics contradict each other Wang and Aitchison, 2025 ref-wang-2025 ; Bergsma et al., 2025 ref-bergsma-2025 ; Qiu et al., 2025 ref-qiu-2025 , and none account for epochs. So we fit power laws for how these hyperparameters should scale in terms of unique data, model size, and epochs. We find that optimal weight decay grows steeply with epochs and with model size. With larger compute budgets and fixed datasets, we should train larger models for more epochs. This follows the finding from Kim et al. 2026 ref-kim-2026 , but goes further by giving concrete prescriptions for the training parameters. Furthermore the law extrapolates well. We find that it is near-optimal at 16x of the fitted compute, and within 0.01 nats of the optimal loss on every axis we test. We find that each prior heuristic would miss on at least one axis. Extrapolated to 5T tokens, the parameter count our data-constrained scaling law prescribes is ~7,000x larger than Chinchilla would predict at 10<sup 29</sup FLOPs, and the models train for several epochs each. 4. Fast Ensembling as a scaling axis While our models are much bigger than Chinchilla would prescribe, we would like to scale parameter counts further in order to improve sample efficiency. We explore ensembling through the snapshot ensembles of q0 Mandal et al., 2026 ref-mandal-2026 , and through deep ensembles Lakshminarayanan et al., 2017 ref-lakshminarayanan-2017 . With snapshot ensembling, we save checkpoints from different points in a single training run and average their predictions Huang et al., 2017 ref-huang-2017 ; Mandal et al., 2026 ref-mandal-2026 . Since the checkpoints come from one run, this costs no extra training compute. To make the checkpoints diverse, we train with a cyclic learning rate in the second half of training, which repeatedly raises and lowers the learning rate in one-epoch cycles. At the lowest learning rate of each cycle, we observe that the model converges to good solutions. We find that cycling the learning rate is better than holding the learning rate at its peak, and averaging the snapshots’ predictions lowers loss further. We hypothesize that this procedure works because of mode connectivity Garipov et al., 2018 ref-garipov-2018 : the good solutions are joined by low-loss paths, so short cycles can reach different strong models without much training. We also include deep ensembling in our scaling laws. Deep ensembling trains independent models with different seeds and averages their predictions Lakshminarayanan et al., 2017 ref-lakshminarayanan-2017 , so each member costs a full training run. The snapshots from each run are pooled into one ensemble. Unlike adding parameters, which increases overfitting, adding ensemble members reduces it, so we give ensembling its own term in the data-constrained scaling law. The term is zero for a single model, and its gain grows toward a ceiling as members are added. This law fits and extrapolates better than the alternatives, including on runs from Kim et al. 2026 ref-kim-2026 . We find that deep ensembling becomes compute-optimal when the compute budget is large relative to the dataset. Past this point, the optimal number of members grows roughly linearly with compute. So the prescription with limited data is to grow model size and epoch count first with scaled weight decay, including snapshot ensembling in the multi-epoch training. Then we allocate the extra compute to deep ensemble members. In this way, loss continues to improve with more compute, whereas prior recipes eventually get worse. 5.Conclusion In order to close the sample efficiency gap between AI and humans, we are motivated to train larger networks than today’s models. From comparison with the brain, we may want to increase the size of current models by seven orders of magnitude. In our latest paper Baron et al., 2026 ref-paper , we provide prescriptions that increase the size of the network by four orders of magnitude on limited datasets, and give prescriptions showing how to train them. We think we can improve data efficiency by going further, but it will require new algorithms, hardware and more. We’ll have more to share on this soon. References 1. Ethan Baron, Shmuel Berman, Anirudh Satheesh, Akshay Vegesna, Samip Dahal, Andrew Gordon Wilson. Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling https://arxiv.org/abs/2610.37842 . arXiv:2610.37842, 2026. 2. Shane Bergsma, Nolan Dey, Gurpreet Gosal, Gavia Gray, Daria Soboleva, Joel Hestness. Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training https://arxiv.org/abs/2505.13738 . arXiv:2505.13738, 2025. 3. Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, Andrew Gordon Wilson. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs https://arxiv.org/abs/1802.10026 . NeurIPS, 2018. 4. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre. Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556 . arXiv:2203.15556, 2022. 5. Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, Kilian Q. Weinberger. Snapshot Ensembles: Train 1, Get M for Free https://arxiv.org/abs/1704.00109 . ICLR, 2017. 6. Konwoo Kim, Suhas Kotha, Percy Liang, Tatsunori Hashimoto. Pre-training under Infinite Compute https://arxiv.org/abs/2509.14786 . ICLR, 2026. 7. Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles https://arxiv.org/abs/1612.01474 . NeurIPS, 2017. 8. Justin Lovelace, Christian Belardi, Srivatsa Kundurthy, Shriya Sudhakar, Kilian Q. Weinberger. Prescriptive Scaling Laws for Data Constrained Training https://arxiv.org/abs/2605.01640 . arXiv:2605.01640, 2026. 9. Bishwas Mandal, Shmuel Berman, Akshay Vegesna, Samip Dahal. q0: Primitives for Hyper-Epoch Pretraining https://arxiv.org/abs/2606.03938 . arXiv:2606.03938, 2026. 10. Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, Colin Raffel. Scaling Data-Constrained Language Models https://arxiv.org/abs/2305.16264 . NeurIPS, 2023. 11. Shikai Qiu, Zixi Chen, Hoang Phan, Qi Lei, Andrew Gordon Wilson. Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scales https://arxiv.org/abs/2512.05620 . arXiv:2512.05620, 2025. 12. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Marius Hobbhahn. Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data https://arxiv.org/abs/2211.04325 . ICML, 2024. 13. Xi Wang, Laurence Aitchison. How to Set AdamW's Weight Decay as You Scale Model and Dataset Size https://arxiv.org/abs/2405.13698 . ICML, 2025. 14. Greg Yang, Edward J. Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, Jianfeng Gao. Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer https://arxiv.org/abs/2203.03466 . NeurIPS, 2021.