Neural Scaling Law

Neural scaling laws are empirical statistical laws in machine learning that describe how neural network performance changes with model size, training dataset size, and training cost, with some models also scaling inference-time compute. These laws guide resource allocation in deep learning, particularly for large language models, where finetuning datasets are typically less than 1% the size of pretraining datasets.

Neural scaling law In machine learning https://en.wikipedia.org/wiki/Machine learning , a neural scaling law is an empirical scaling law https://en.wikipedia.org/wiki/Scaling law that describes how neural network https://en.wikipedia.org/wiki/Neural network machine learning performance changes as key factors are scaled up or down. These factors typically include the number of parameters, training dataset https://en.wikipedia.org/wiki/Training, validation, and test data sets size, 1 cite note-1 2 and training cost. Some models also exhibit performance gains by scaling inference https://en.wikipedia.org/wiki/Inference through increased test-time compute TTC , extending neural scaling laws beyond training to the deployment phase. 3 cite note-:8-3 Introduction edit /w/index.php?title=Neural scaling law&action=edit§ion=1 In general, a deep learning https://en.wikipedia.org/wiki/Deep learning model can be characterized by four parameters: model size, training dataset size, training cost, and the post-training error rate e.g., the test set error rate . Each of these variables can be defined as a real number https://en.wikipedia.org/wiki/Real number , usually written as respectively: parameter count, dataset size, computing cost, and loss https://en.wikipedia.org/wiki/Loss function . A neural scaling law is a theoretical or empirical statistical law https://en.wikipedia.org/wiki/Empirical statistical laws between these parameters. There are also other parameters with other scaling laws. Size of the model edit /w/index.php?title=Neural scaling law&action=edit§ion=2 In most cases, the model's size is simply the number of parameters. However, one complication arises with the use of sparse models, such as mixture-of-expert models https://en.wikipedia.org/wiki/Mixture of experts . 4 With sparse models, during inference, only a fraction of their parameters are used. In comparison, most other kinds of neural networks, such as transformer https://en.wikipedia.org/wiki/Transformer deep learning architecture models, always use all their parameters during inference. Size of the training dataset edit /w/index.php?title=Neural scaling law&action=edit§ion=3 The size of the training dataset is usually quantified by the number of data points within it. Larger training datasets are typically preferred, as they provide a richer and more diverse source of information from which the model can learn. This can lead to improved generalization performance when the model is applied to new, unseen data. 5 However, increasing the size of the training dataset also increases the computational resources and time required for model training. With the "pretrain, then finetune" method used for most large language models https://en.wikipedia.org/wiki/Large language model , there are two kinds of training dataset: the pretraining dataset and the finetuning dataset. Their sizes have different effects on model performance. Generally, the finetuning dataset is less than 1% the size of pretraining dataset. 6 cite note-:2-6 In some cases, a small amount of high quality data suffices for finetuning, and more data does not necessarily improve performance. 6 cite note-:2-6 Many scaling laws, due to their inherent diminishing returns nature, value data based on a submodular set function https://en.wikipedia.org/wiki/Submodular set function which was shown in a paper 7 on this topic. Cost of training edit /w/index.php?title=Neural scaling law&action=edit§ion=4 Training cost is typically measured in terms of time how long it takes to train the model and computational resources how much processing power and memory are required . It is important to note that the cost of training can be significantly reduced with efficient training algorithms, optimized software libraries, and parallel computing https://en.wikipedia.org/wiki/Parallel computing on specialized hardware such as GPUs https://en.wikipedia.org/wiki/Graphics processing unit or TPUs https://en.wikipedia.org/wiki/Tensor processing unit . The cost of training a neural network model https://en.wikipedia.org/wiki/Network model is a function of several factors, including model size, training dataset size, the training algorithm complexity https://en.wikipedia.org/wiki/Computational complexity theory , and the computational resources available. 5 In particular, doubling the training dataset size does not necessarily double the cost of training, because one may train the model for several times over the same dataset each being an " epoch https://en.wikipedia.org/wiki/Glossary of artificial intelligence epoch " . Performance edit /w/index.php?title=Neural scaling law&action=edit§ion=5 The performance of a neural network model is evaluated based on its ability to accurately predict the output given some input data. Common metrics for evaluating model performance include: 5 cite note-goodfellow-5 - Negative log-likelihood https://en.wikipedia.org/wiki/Likelihood function per token logarithm of perplexity https://en.wikipedia.org/wiki/Perplexity for language modeling https://en.wikipedia.org/wiki/Language model ; Accuracy https://en.wikipedia.org/wiki/Accuracy and precision , precision, recall https://en.wikipedia.org/wiki/Precision and recall , and F1 score https://en.wikipedia.org/wiki/F1 score for classification https://en.wikipedia.org/wiki/Classification tasks; Mean squared error https://en.wikipedia.org/wiki/Mean squared error MSE or mean absolute error https://en.wikipedia.org/wiki/Mean absolute error MAE for regression https://en.wikipedia.org/wiki/Regression analysis tasks; Elo rating https://en.wikipedia.org/wiki/Elo rating system in a competition against other models, such as gameplay https://en.wikipedia.org/wiki/Computer chess or 10 cite note-:6-10 preference by a human judge https://en.wikipedia.org/wiki/Reinforcement learning from human feedback . 11 cite note-11 Performance can be improved by using more data, larger models, different training algorithms, regularizing https://en.wikipedia.org/wiki/Regularization mathematics the model to prevent overfitting https://en.wikipedia.org/wiki/Overfitting , and early stopping https://en.wikipedia.org/wiki/Early stopping using a validation set. When the performance is a number bounded within the range of , such as accuracy, precision, etc., it often scales as a sigmoid function https://en.wikipedia.org/wiki/Sigmoid function of cost, as seen in the figures. Examples edit /w/index.php?title=Neural scaling law&action=edit§ion=6 Hestness, Narang, et al, 2017 edit /w/index.php?title=Neural scaling law&action=edit§ion=7 The 2017 paper 2 is a common reference point for neural scaling laws fitted by statistical analysis on experimental data. Previous works before the 2000s, as cited in the paper, were either theoretical or orders of magnitude smaller in scale. Whereas previous works generally found the scaling exponent to scale like , with , the paper found that . Of the factors they varied, only task can change the exponent . Changing the architecture optimizers, regularizers, and loss functions, would only change the proportionality factor, not the exponent. For example, for the same task, one architecture might have while another might have . They also found that for a given architecture, the number of parameters necessary to reach lowest levels of loss, given a fixed dataset size, grows like for another exponent . They studied machine translation with LSTM , generative language modelling with LSTM , ImageNet classification with ResNet , and speech recognition with two hybrid LSTMs complemented by either CNNs or an attention decoder architectures . Henighan, Kaplan, et al, 2020 edit /w/index.php?title=Neural scaling law&action=edit§ion=8 A 2020 analysis 12 studied statistical relations between over a wide range of values and found similar scaling laws, over the range of , , and over multiple modalities text, video, image, text to image, etc. . 12 cite note-:0-12 In particular, the scaling laws it found are Table 1 of 12 : - For each modality, they fixed one of the two , and varying the other one is varied along using , the achievable test loss satisfieswhere is the varied variable, and are parameters to be found by statistical fitting. The parameter is the most important one. - When is the varied variable, ranges from to depending on the model modality. This corresponds to the from the Chinchilla scaling paper. - When is the varied variable, ranges from to depending on the model modality. This corresponds to the from the Chinchilla scaling paper. - Given fixed computing budget, optimal model parameter count is consistently aroundThe parameter varies by a factor of up to 10 for different modalities. The exponent parameter varies from to for different modalities. This exponent corresponds to the from the Chinchilla scaling paper. - It's "strongly suggested" but not statistically checked that . This exponent corresponds to the from the Chinchilla scaling paper. The scaling law of was confirmed during the training of GPT-3 https://en.wikipedia.org/wiki/GPT-3 Figure 3.1 13 . Chinchilla scaling Hoffmann, et al, 2022 edit /w/index.php?title=Neural scaling law&action=edit§ion=9 One particular scaling law " Chinchilla scaling https://en.wikipedia.org/wiki/Chinchilla AI " states that, for a large language model https://en.wikipedia.org/wiki/Large language model LLM autoregressively trained for one epoch, with a cosine learning rate https://en.wikipedia.org/wiki/Learning rate schedule, we have: 15 where the variables are - is the cost of training the model, measured in floating-point operations FLOPs . - is the number of tokens in the training set. - is the average negative log-likelihood loss per token nats https://en.wikipedia.org/wiki/Nat unit /token , achieved by the trained LLM on the test dataset.- represents the loss of an ideal generative process on the test data - captures the fact that a Transformer language model with parameters underperforms the ideal generative process - captures the fact that the model trained on tokens underperforms the ideal generative process and the statistical parameters are - , meaning that it costs 6 FLOPs per parameter to train on one token. This is estimated by Kaplan et al. Note that training cost is much higher than inference cost, as training entails both forward and 16 cite note-kaplan-scaling-16 backward passes https://en.wikipedia.org/wiki/Backpropagation , whereas inference costs 1 to 2 FLOPs per parameter to infer on one token. - . Although Besiroglu et al. 17 claims that the statistical estimation is slightly off, and should be . The statistical laws were fitted over experimental data with . Since there are 4 variables related by 2 equations, imposing 1 additional constraint and 1 additional optimization objective https://en.wikipedia.org/wiki/Mathematical optimization allows us to solve for all four variables. In particular, for any fixed , we can uniquely solve for all 4 variables that minimizes . This provides us with the optimal for any fixed :Plugging in the numerical values, we obtain the "Chinchilla efficient" model size and training dataset size, as well as the test loss achievable:Similarly, we may find the optimal training dataset size and training compute budget for any fixed model parameter size, and so on. There are other estimates for "Chinchilla efficient" model size and training dataset size. The above is based on a statistical model of . One can also directly fit a statistical law for without going through the detour, for which one obtains:or as tabulated: | / FLOP | / FLOPs of training Gopher | || |---|---|---|---| | 400 million | 1.92e+19 | 1/29968 | 8.0 billion | | 1 billion | 1.21e+20 | 1/5706 | 20.2 billion | | 10 billion | 1.23e+22 | 1/2819 | 205.1 billion | | 67 billion | 5.76e+23 | 1 | 1.5 trillion | | 175 billion | 3.85e+24 | 6.7 | 3.7 trillion | | 280 billion | 9.90e+24 | 17.2 | 5.9 trillion | | 520 billion | 3.43e+25 | 59.5 | 11.0 trillion | | 1 trillion | 1.27e+26 | 221.3 | 21.2 trillion | | 10 trillion | 1.30e+28 | 22515.9 | 216.2 trillion | Discrepancy edit /w/index.php?title=Neural scaling law&action=edit§ion=10 The Chinchilla scaling law analysis for training transformer https://en.wikipedia.org/wiki/Transformer machine learning model language models suggests that for a given training compute budget , to achieve the minimal pretraining loss for that budget, the number of model parameters and the number of training tokens should be scaled in equal proportions, . This conclusion differs from analysis conducted by Kaplan et al., 16 which found that should be increased more quickly than , . This discrepancy can primarily be attributed to the two studies using different methods for measuring model size. Kaplan et al.: 18 cite note-18 - did not count the parameters in the token embedding layer, which when analyzed at smaller model sizes leads to biased coefficients; - studied smaller models than the Chinchilla group, magnifying the effect; - assumed that . Secondary effects also arise due to differences in hyperparameter tuning and learning rate schedules. Kaplan et al.: 19 cite note-19 - used a warmup schedule that was too long for smaller models, making them appear less efficient; - did not fully tuning optimization hyperparameters. Beyond Chinchilla scaling edit /w/index.php?title=Neural scaling law&action=edit§ion=11 As Chinchilla scaling has been the reference point for many large-scaling training runs, there had been a concurrent effort to go "beyond Chinchilla scaling", meaning to modify some of the training pipeline in order to obtain the same loss with less effort, or deliberately train for longer than what is "Chinchilla optimal". Usually, the goal is to make the scaling law exponent larger, which means the same loss can be trained for much less compute. For instance, filtering data can make the scaling law exponent larger. 20 cite note-20 Another strand of research studies how to deal with limited data, as according to Chinchilla scaling laws, the training dataset size for the largest language models already approaches what is available on the internet. 21 found that augmenting the dataset with a mix of "denoising objectives" constructed from the dataset improves performance. studies optimal scaling when all available data is already exhausted such as in rare languages , so one must train multiple epoches over the same dataset whereas Chinchilla scaling requires only one epoch . The Phi series of small language models were trained on textbook-like data generated by large language models, for which data is only limited by amount of compute available. 22 cite note-22 23 cite note-23 Chinchilla optimality was defined as "optimal for training compute", whereas in actual production-quality models, there will be a lot of inference after training is complete. "Overtraining" during training means better performance during inference. 24 cite note-24 LLaMA https://en.wikipedia.org/wiki/LLaMA models were overtrained for this reason. Subsequent studies discovered scaling laws in the overtraining regime, for dataset sizes up to 32x more than Chinchilla-optimal. 25 cite note-25 Broken neural scaling laws BNSL edit /w/index.php?title=Neural scaling law&action=edit§ion=12 A 2022 analysis 26 found that many scaling behaviors of artificial neural networks follow a smoothly broken power law https://en.wikipedia.org/wiki/Power law Smoothly broken power law functional form: in which refers to the quantity being scaled i.e. , , , number of training steps, number of inference steps, or model input size and refers to the downstream or upstream performance evaluation metric of interest e.g. prediction error, cross entropy https://en.wikipedia.org/wiki/Cross entropy , calibration error, AUROC https://en.wikipedia.org/wiki/Receiver operating characteristic , BLEU score percentage https://en.wikipedia.org/wiki/BLEU , F1 score https://en.wikipedia.org/wiki/F1 Score , reward, Elo rating https://en.wikipedia.org/wiki/Elo rating system , solve rate, or FID https://en.wikipedia.org/wiki/Fréchet inception distance score in zero-shot https://en.wikipedia.org/wiki/Zero-shot learning , prompted https://en.wikipedia.org/wiki/Prompt engineering , or fine-tuned https://en.wikipedia.org/wiki/Fine-tuning machine learning settings. The parameters are found by statistical fitting. On a log–log plot https://en.wikipedia.org/wiki/Log–log plot , when is not too large and is subtracted out from the y-axis, this functional form looks like a series of linear segments connected by arcs; the transitions between the segments are called "breaks", hence the name broken neural scaling laws BNSL . The scenarios in which the scaling behaviors of artificial neural networks were found to follow this functional form include large-scale vision https://en.wikipedia.org/wiki/Computer vision , language https://en.wikipedia.org/wiki/Natural language processing , audio, video, diffusion https://en.wikipedia.org/wiki/Diffusion model , generative modeling https://en.wikipedia.org/wiki/Generative model , multimodal learning https://en.wikipedia.org/wiki/Multimodal learning , contrastive learning https://en.wikipedia.org/wiki/Contrastive learning , AI alignment https://en.wikipedia.org/wiki/AI alignment , AI capabilities, robotics https://en.wikipedia.org/wiki/Robotics , out-of-distribution OOD generalization, continual learning, transfer learning https://en.wikipedia.org/wiki/Transfer learning , uncertainty estimation https://en.wikipedia.org/wiki/Uncertainty quantification / calibration https://en.wikipedia.org/wiki/Calibration statistics , out-of-distribution detection https://en.wikipedia.org/wiki/Anomaly detection , adversarial robustness https://en.wikipedia.org/wiki/Adversarial machine learning , distillation https://en.wikipedia.org/wiki/Knowledge distillation , sparsity, retrieval, quantization, pruning https://en.wikipedia.org/wiki/Pruning artificial neural network , fairness https://en.wikipedia.org/wiki/Fairness machine learning , molecules, computer programming/coding, math word problems, arithmetic, emergent abilities https://en.wikipedia.org/wiki/Large language model Emergent abilities , double descent https://en.wikipedia.org/wiki/Double descent , supervised learning https://en.wikipedia.org/wiki/Supervised learning , unsupervised https://en.wikipedia.org/wiki/Unsupervised learning / self-supervised https://en.wikipedia.org/wiki/Self-supervised learning learning, and reinforcement learning https://en.wikipedia.org/wiki/Deep reinforcement learning single agent and multi-agent https://en.wikipedia.org/wiki/Multi-agent reinforcement learning . The architectures for which the scaling behaviors of artificial neural networks were found to follow this functional form include residual neural networks https://en.wikipedia.org/wiki/Residual neural network , transformers https://en.wikipedia.org/wiki/Transformer machine learning model , MLPs https://en.wikipedia.org/wiki/Multilayer perceptron , MLP-mixers https://arxiv.org/abs/2105.01601 , recurrent neural networks https://en.wikipedia.org/wiki/Recurrent neural network , convolutional neural networks https://en.wikipedia.org/wiki/Convolutional neural network , graph neural networks https://en.wikipedia.org/wiki/Graph neural network , U-nets https://en.wikipedia.org/wiki/U-Net , encoder-decoder https://en.wikipedia.org/wiki/Transformer machine learning model Encoder-decoder architecture and encoder-only https://en.wikipedia.org/wiki/Transformer deep learning architecture encoder-only and decoder-only models, ensembles https://en.wikipedia.org/wiki/Ensemble learning and non-ensembles , MoE https://en.wikipedia.org/wiki/Mixture of experts mixture of experts and non-MoE models, and sparse pruned https://en.wikipedia.org/wiki/Pruning artificial neural network and non-sparse unpruned models. Inference scaling edit /w/index.php?title=Neural scaling law&action=edit§ion=13 Other than scaling up training compute, one can also scale up inference compute or "test-time compute" 3 . As an example, the Elo rating https://en.wikipedia.org/wiki/Elo rating system of AlphaGo https://en.wikipedia.org/wiki/AlphaGo improves steadily as it is allowed to spend more time on its Monte Carlo Tree Search https://en.wikipedia.org/wiki/Monte Carlo tree search per play. 27 cite note-27 : Fig 4For AlphaGo Zero https://en.wikipedia.org/wiki/AlphaGo Zero , increasing Elo by 120 requires either 2x model size and training, or 2x test-time search. Similarly, a language model for solving competition-level coding challenges, AlphaCode, consistently improved log-linearly in performance with more search time. 28 cite note-:5-28 29 cite note-29 For Hex https://en.wikipedia.org/wiki/Hex board game , 10x training-time compute trades for 15x test-time compute. 10 For for heads up Libratus https://en.wikipedia.org/wiki/Libratus no-limit https://en.wikipedia.org/wiki/Betting in poker No limit Texas hold 'em https://en.wikipedia.org/wiki/Texas hold 'em , and Cicero for , and many other abstract games of partial information, inference-time searching improves performance at a similar tradeoff ratio, for up to 100,000x effective increase in training-time compute. Diplomacy https://en.wikipedia.org/wiki/Diplomacy game 28 cite note-:5-28 In 2024, the OpenAI o1 https://en.wikipedia.org/wiki/OpenAI o1 report documented that o1's performance consistently improved with both increased train-time compute and test-time compute, and gave numerous examples of test-time compute scaling in mathematics, scientific reasoning, and coding tasks. 30 cite note-30 31 cite note-31 One method for scaling up test-time compute is process-based supervision , where a model generates a step-by-step reasoning chain to answer a question, and another model either human or AI provides a reward score on some of the intermediate steps, not just the final answer. Process-based supervision can be scaled arbitrarily by using synthetic reward score without another model, for example, by running Monte Carlo rollouts and scoring each step in the reasoning according to how likely it leads to the right answer. Another method is by revision models , which are models trained to solve a problem multiple times, each time revising the previous attempt. 32 cite note-32 Other examples edit /w/index.php?title=Neural scaling law&action=edit§ion=14 Vision transformers edit /w/index.php?title=Neural scaling law&action=edit§ion=15 Vision transformers https://en.wikipedia.org/wiki/Vision transformer , similar to language transformers, exhibit scaling laws. A 2022 research trained vision transformers, with parameter counts , on image sets of sizes , for computing in units of TPUv3-core-days . 33 cite note-33 After training the model, it is finetuned on ImageNet https://en.wikipedia.org/wiki/ImageNet training set. Let be the error probability of the finetuned model classifying ImageNet test set. They found . Neural machine translation edit /w/index.php?title=Neural scaling law&action=edit§ion=16 Ghorbani, Behrooz et al. 34 studied scaling laws for neural machine translation https://en.wikipedia.org/wiki/Neural machine translation specifically, English as source, and German as target in encoder-decoder Transformer https://en.wikipedia.org/wiki/Transformer machine learning model models, trained until convergence on the same datasets thus they did not fit scaling laws for computing cost or dataset size . They varied They found three results: - is a scaling law function of , where are encoder and decoder parameter count. It is not simply a function of total parameter count . The function has form , where are fitted parameters. They found that minimizes loss if is held fixed. - "saturates" that is, it reaches for smaller models when the training and testing datasets are "source-natural" than "target-natural". A "source-natural" data point means a pair of English-German sentences, and the model is asked to translate the English sentence into German, and the English sentence is written by a natural English writer, while the German sentence is translated from the English sentence by a machine translator. To construct the two kinds of datasets, the authors collected natural English and German sentences online, then used machine translation to generate their translations. 35 cite note-35 - As models grow larger, models trained on source-original datasets can achieve low loss but bad BLEU score https://en.wikipedia.org/wiki/BLEU . In contrast, models trained on target-original datasets achieve low loss and good BLEU score in tandem Figure 10, 11 . 34 cite note-:3-34 The authors hypothesize that source-natural datasets have uniform and dull target sentences, and so a model that is trained to predict the target sentences would quickly overfit. 36 trained Transformers for machine translations with sizes on dataset sizes . They found the Kaplan et al. 2020 scaling law applied to machine translation: . They also found the BLEU score scaling as . 16 cite note-kaplan-scaling-16 Transfer learning edit /w/index.php?title=Neural scaling law&action=edit§ion=17 Hernandez, Danny et al. 37 studied scaling laws for transfer learning https://en.wikipedia.org/wiki/Transfer learning in language models. They trained a family of Transformers in three ways: - pretraining on English, finetuning on Python - pretraining on an equal mix of English and Python, finetuning on Python - training on Python The idea is that pretraining on English should help the model achieve low loss on a test set of Python text. Suppose the model has parameter count , and after being finetuned on Python tokens, it achieves some loss . We say that its "transferred token count" is , if another model with the same achieves the same after training on Python tokens. They found for pretraining on English text, and for pretraining on English and non-Python code. Precision edit /w/index.php?title=Neural scaling law&action=edit§ion=18 Kumar et al. 38 study scaling laws for numerical precision in the training of language models. They train a family of language models with weights, activations, and KV cache in varying numerical precision in both integer and floating-point type to measure the effects on loss as a function of precision. For training, their scaling law accounts for lower precision by wrapping the effects of precision into an overall "effective parameter count" that governs loss scaling, using the parameterization . This illustrates how training in lower precision degrades performance by reducing the true capacity of the model in a manner that varies exponentially with bits. For inference, they find that extreme overtraining of language models past Chinchilla-optimality can lead to models being more sensitive to quantization, a standard technique for efficient deep learning. This is demonstrated by observing that the degradation in loss due to weight quantization increases as an approximate power law in the token/parameter ratio seen during pretraining, so that models pretrained on extreme token budgets can perform worse in terms of validation loss than those trained on more modest token budgets if post-training quantization is applied. Other work examining the effects of overtraining include Sardana et al. 39 and Gadre et al. 40 cite note-40 Densing laws edit /w/index.php?title=Neural scaling law&action=edit§ion=19 Xiao et al. 9 considered the parameter efficiency "density" of models over time. The idea is that over time, researchers would discover models that use their parameters more efficiently, in that models with the same performance can have fewer parameters. A model can have an actual parameter count , defined as the actual number of parameters in the model, and an "effective" parameter count , defined as how many parameters it would have taken a previous well-known model to reach he same performance on some benchmarks, such as MMLU https://en.wikipedia.org/wiki/MMLU . is not measured directly, but rather by measuring the actual model performance , then plugging it back to a previously fitted scaling law, such as the Chinchilla scaling law, to obtain what would be required to reach that performance , according to that previously fitted scaling laws. A densing law states that , where is real-world time, measured in days. See also edit /w/index.php?title=Neural scaling law&action=edit§ion=20 Large language model https://en.wikipedia.org/wiki/Large language model – Type of machine learning model Foundation model https://en.wikipedia.org/wiki/Foundation model – Artificial intelligence model paradigm Artificial general intelligence https://en.wikipedia.org/wiki/Artificial general intelligence – Type of AI with wide-ranging abilities Hilberg's hypothesis https://en.wikipedia.org/wiki/Hilberg's hypothesis – Power law growth of entropy of language or a stochastic process References edit /w/index.php?title=Neural scaling law&action=edit§ion=21 ↑ cite ref-1 Bahri, Yasaman; Dyer, Ethan; Kaplan, Jared; Lee, Jaehoon; Sharma, Utkarsh 2024 . "Explaining neural scaling laws" https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11228526 .. Proceedings of the National Academy of Sciences https://en.wikipedia.org/wiki/Proceedings of the National Academy of Sciences of the United States of America 121 27 e2311878121. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2102.06701 https://arxiv.org/abs/2102.06701 . Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2024PNAS..12111878B https://ui.adsabs.harvard.edu/abs/2024PNAS..12111878B . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1073/pnas.2311878121 https://doi.org/10.1073%2Fpnas.2311878121 . PMC https://en.wikipedia.org/wiki/PMC identifier 11228526 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11228526 . PMID https://en.wikipedia.org/wiki/PMID identifier 38913889 https://pubmed.ncbi.nlm.nih.gov/38913889 . 1 cite ref-:4 2-0 2 cite ref-:4 2-1 Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; Patwary, Md Mostofa Ali; Yang, Yang; Zhou, Yanqi 2017-12-01 . "Deep Learning Scaling is Predictable, Empirically". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1712.00409 https://arxiv.org/abs/1712.00409 cs.LG https://arxiv.org/archive/cs.LG . 1 cite ref-:8 3-0 2 cite ref-:8 3-1 Cobbe, Karl; Kosaraju, Vineet; Bavarian, Mohammad; Chen, Mark; Jun, Heewoo; Kaiser, Lukasz; Plappert, Matthias; Tworek, Jerry; Hilton, Jacob 2021-11-18 . "Training Verifiers to Solve Math Word Problems". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2110.14168 https://arxiv.org/abs/2110.14168 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-4 Rajbhandari, Samyam; Li, Conglong; Yao, Zhewei; Zhang, Minjia; Aminabadi, Reza Yazdani; Awan, Ammar Ahmad; Rasley, Jeff; He, Yuxiong 2022-06-28 . "DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale" https://proceedings.mlr.press/v162/rajbhandari22a.html . Proceedings of the 39th International Conference on Machine Learning . PMLR: 18332–18346. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2201.05596 https://arxiv.org/abs/2201.05596 . 1 cite ref-goodfellow 5-0 2 cite ref-goodfellow 5-1 3 cite ref-goodfellow 5-2 Goodfellow, I., Bengio, Y., & Courville, A. 2016 . Deep Learning. MIT Press. 1 cite ref-:2 6-0 2 cite ref-:2 6-1 Zhou, Chunting; Liu, Pengfei; Xu, Puxin; Iyer, Srini; Sun, Jiao; Mao, Yuning; Ma, Xuezhe; Efrat, Avia; Yu, Ping; Yu, Lili; Zhang, Susan; Ghosh, Gargi; Lewis, Mike; Zettlemoyer, Luke; Levy, Omer 2023-05-01 . "LIMA: Less Is More for Alignment". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2305.11206 https://arxiv.org/abs/2305.11206 cs.CL https://arxiv.org/archive/cs.CL . ↑ cite ref-7 Bilmes, Jeff; Bhatt, Gantavya; Das, Arnav May 28, 2026 . "How Much Is a Dataset Worth? Scaling Laws, the Vendi Score, and Matrix Spectral Functions". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2605.29448 https://arxiv.org/abs/2605.29448 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-8 "google/BIG-bench" https://github.com/google/BIG-bench/tree/main . Google. 2024-09-24. Retrieved 2024-09-25. 1 cite ref-:7 9-0 2 cite ref-:7 9-1 Xiao, Chaojun; Cai, Jie; Zhao, Weilin; Zeng, Guoyang; Lin, Biyuan; Zhou, Jie; Zheng, Zhi; Han, Xu; Liu, Zhiyuan https://en.wikipedia.org/wiki/ModelBest Liu Zhiyuan 2024-12-06 . "Densing Law of LLMs". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2412.04315 https://arxiv.org/abs/2412.04315 cs.AI https://arxiv.org/archive/cs.AI . 1 cite ref-:6 10-0 2 cite ref-:6 10-1 Jones, Andy L. 2021 . "Scaling Scaling Laws with Board Games". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2104.03113 https://arxiv.org/abs/2104.03113 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-11 LMSYS Chatbot leaderboard https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 1 cite ref-:0 12-0 2 cite ref-:0 12-1 3 cite ref-:0 12-2 Henighan, Tom; Kaplan, Jared; Katz, Mor; Chen, Mark; Hesse, Christopher; Jackson, Jacob; Heewoo, Jun; Brown, Tom B.; Dhariwal, Prafulla; Mann, Chris; Radford, Alec; Ramesh, Aditya; Ryder, Nick; Ziegler, Daniel M.; Schulman, John; Gray, Scott; Hallacy, Chris; Amodei, Dario; McCandlish, Sam 2020-10-27 . Scaling Laws for Autoregressive Generative Modeling . arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2010.14701 https://arxiv.org/abs/2010.14701 . OCLC https://en.wikipedia.org/wiki/OCLC identifier 1228442047 https://search.worldcat.org/oclc/1228442047 . ↑ cite ref-13 Brown, Tom B.; Mann, Benjamin; Ryder, Nick; Subbiah, Melanie; Kaplan, J.; Dhariwal, Prafulla; Neelakantan, Arvind; Shyam, Pranav; Sastry, Girish; Askell, Amanda; Agarwal, Sandhini; Herbert-Voss, Ariel; Krueger, Gretchen; Henighan, T.; Child, Rewon 2020-05-28 . "Language Models are Few-Shot Learners". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2005.14165 https://arxiv.org/abs/2005.14165 cs.CL https://arxiv.org/archive/cs.CL . ↑ cite ref-14 Besiroglu, Tamay 2024-04-17 . "Chinchilla Scaling: A Replication Attempt" https://epochai.org/blog/chinchilla-scaling-a-replication-attempt . Epoch AI . Retrieved 2024-09-24. ↑ cite ref-15 Hoffmann, Jordan; Borgeaud, Sebastian; Mensch, Arthur; Buchatskaya, Elena; Cai, Trevor; Rutherford, Eliza; Casas, Diego de Las; Hendricks, Lisa Anne; Welbl, Johannes; Clark, Aidan; Hennigan, Tom; Noland, Eric; Millican, Katie; Driessche, George van den; Damoc, Bogdan 2022-03-29 . "Training Compute-Optimal Large Language Models". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2203.15556 https://arxiv.org/abs/2203.15556 cs.CL https://arxiv.org/archive/cs.CL . 1 cite ref-kaplan-scaling 16-0 2 cite ref-kaplan-scaling 16-1 3 cite ref-kaplan-scaling 16-2 Kaplan, Jared; McCandlish, Sam; Henighan, Tom; Brown, Tom B.; Chess, Benjamin; Child, Rewon; Gray, Scott; Radford, Alec; Wu, Jeffrey; Amodei, Dario 2020 . "Scaling Laws for Neural Language Models". CoRR . abs/2001.08361. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2001.08361 https://arxiv.org/abs/2001.08361 . ↑ cite ref-17 Besiroglu, Tamay; Erdil, Ege; Barnett, Matthew; You, Josh 2024-04-15 . "Chinchilla Scaling: A replication attempt". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2404.10102 https://arxiv.org/abs/2404.10102 cs.AI https://arxiv.org/archive/cs.AI . ↑ cite ref-18 Pearce, Tim; Song, Jinyeop 2024 . "Reconciling Kaplan and Chinchilla Scaling Laws". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2406.12907 https://arxiv.org/abs/2406.12907 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-19 Porian, Tomer; Wortsman, Mitchell; Jitsev, Jenia; Schmidt, Ludwig; Carmon, Yair 2024-07-25 . "Resolving Discrepancies in Compute-Optimal Scaling of Language Models". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2406.19146 https://arxiv.org/abs/2406.19146 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-20 Sorscher, Ben; Geirhos, Robert; Shekhar, Shashank; Ganguli, Surya; Morcos, Ari S. 2023-04-21 . "Beyond neural scaling laws: beating power law scaling via data pruning". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2206.14486 https://arxiv.org/abs/2206.14486 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-21 Tay, Yi; Wei, Jason; Chung, Hyung Won; Tran, Vinh Q.; So, David R.; Shakeri, Siamak; Garcia, Xavier; Zheng, Huaixiu Steven; Rao, Jinfeng 2022-11-16 . "Transcending Scaling Laws with 0.1% Extra Compute". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2210.11399 https://arxiv.org/abs/2210.11399 cs.CL https://arxiv.org/archive/cs.CL . ↑ cite ref-22 Muennighoff, Niklas; Rush, Alexander; Barak, Boaz; Le Scao, Teven; Tazi, Nouamane; Piktus, Aleksandra; Pyysalo, Sampo; Wolf, Thomas; Raffel, Colin A. 2023-12-15 . "Scaling Data-Constrained Language Models" https://proceedings.neurips.cc/paper files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html . Advances in Neural Information Processing Systems . 36 : 50358–50376. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2305.16264 https://arxiv.org/abs/2305.16264 . ↑ cite ref-23 Li, Yuanzhi; Bubeck, Sébastien; Eldan, Ronen; Del Giorno, Allie; Gunasekar, Suriya; Lee, Yin Tat 2023-09-11 . "Textbooks Are All You Need II: phi-1.5 technical report". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2309.05463 https://arxiv.org/abs/2309.05463 cs.CL https://arxiv.org/archive/cs.CL . ↑ cite ref-24 Sardana, Nikhil; Frankle, Jonathan 2023-12-31 . "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2401.00448 https://arxiv.org/abs/2401.00448 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-25 Gadre, Samir Yitzhak; Smyrnis, Georgios; Shankar, Vaishaal; Gururangan, Suchin; Wortsman, Mitchell; Shao, Rulin; Mercat, Jean; Fang, Alex; Li, Jeffrey 2024-03-13 . "Language models scale reliably with over-training and on downstream tasks". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2403.08540 https://arxiv.org/abs/2403.08540 cs.CL https://arxiv.org/archive/cs.CL . ↑ cite ref-:1 26-0 Caballero, Ethan; Gupta, Kshitij; Rish, Irina; Krueger, David 2022 . "Broken Neural Scaling Laws". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2210.14891 https://arxiv.org/abs/2210.14891 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-27 Silver, David; Huang, Aja; Maddison, Chris J.; Guez, Arthur; Sifre, Laurent; van den Driessche, George; Schrittwieser, Julian; Antonoglou, Ioannis; Panneershelvam, Veda; Lanctot, Marc; Dieleman, Sander; Grewe, Dominik; Nham, John; Kalchbrenner, Nal; Sutskever, Ilya January 2016 . "Mastering the game of Go with deep neural networks and tree search" https://www.nature.com/articles/nature16961 . Nature . 529 7587 : 484–489. Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2016Natur.529..484S https://ui.adsabs.harvard.edu/abs/2016Natur.529..484S . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1038/nature16961 https://doi.org/10.1038%2Fnature16961 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 1476-4687 https://search.worldcat.org/issn/1476-4687 . PMID https://en.wikipedia.org/wiki/PMID identifier 26819042 https://pubmed.ncbi.nlm.nih.gov/26819042 . 1 cite ref-:5 28-0 2 cite ref-:5 28-1 Noam, Brown 2024-09-17 . Video . Retrieved 2024-09-24 – via YouTube. Lecture at Parables on the Power of Planning in AI: From Poker to Diplomacy: Noam Brown OpenAI Paul G. Allen School https://en.wikipedia.org/wiki/Paul G. Allen School of Computer Science & Engineering on Thursday, May 23, 2024, 3:30 pm ↑ cite ref-29 Li, Yujia; Choi, David; Chung, Junyoung; Kushman, Nate; Schrittwieser, Julian; Leblond, Rémi; Eccles, Tom; Keeling, James; Gimeno, Felix; Dal Lago, Agustin; Hubert, Thomas; Choy, Peter; de Masson d'Autume, Cyprien; Babuschkin, Igor; Chen, Xinyun 2022-12-09 . "Competition-level code generation with AlphaCode" https://www.science.org/doi/10.1126/science.abq1158 . Science . 378 6624 : 1092–1097. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2203.07814 https://arxiv.org/abs/2203.07814 . Bibcode https://en.wikipedia.org/wiki/Bibcode identifier : 2022Sci...378.1092L https://ui.adsabs.harvard.edu/abs/2022Sci...378.1092L . doi https://en.wikipedia.org/wiki/Doi identifier : 10.1126/science.abq1158 https://doi.org/10.1126%2Fscience.abq1158 . ISSN https://en.wikipedia.org/wiki/ISSN identifier 0036-8075 https://search.worldcat.org/issn/0036-8075 . PMID https://en.wikipedia.org/wiki/PMID identifier 36480631 https://pubmed.ncbi.nlm.nih.gov/36480631 . ↑ cite ref-30 Villalobos, Pablo 2023-07-28 . "Trading Off Compute in Training and Inference" https://epochai.org/blog/trading-off-compute-in-training-and-inference . Epoch AI . Retrieved 2024-09-24. ↑ cite ref-31 "Learning to Reason with LLMs" https://openai.com/index/learning-to-reason-with-llms/ . OpenAI. Retrieved 2024-09-16. ↑ cite ref-32 Snell, Charlie; Lee, Jaehoon; Xu, Kelvin; Kumar, Aviral 2024-08-06 . "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2408.03314 https://arxiv.org/abs/2408.03314 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-33 Zhai, Xiaohua; Kolesnikov, Alexander; Houlsby, Neil; Beyer, Lucas 2022 . "Scaling Vision Transformers" https://openaccess.thecvf.com/content/CVPR2022/html/Zhai Scaling Vision Transformers CVPR 2022 paper.html . CVPR : 12104–12113. 1 cite ref-:3 34-0 2 cite ref-:3 34-1 Ghorbani, Behrooz; Firat, Orhan; Freitag, Markus; Bapna, Ankur; Krikun, Maxim; Garcia, Xavier; Chelba, Ciprian; Cherry, Colin 2021-09-01 . "Scaling Laws for Neural Machine Translation". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2109.07740 https://arxiv.org/abs/2109.07740 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-35 Chen, Mia Xu; Firat, Orhan; Bapna, Ankur; Johnson, Melvin; Macherey, Wolfgang; Foster, George; Jones, Llion; Schuster, Mike; Shazeer, Noam; Parmar, Niki; Vaswani, Ashish; Uszkoreit, Jakob; Kaiser, Lukasz; Chen, Zhifeng; Wu, Yonghui July 2018 . "The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation" https://aclanthology.org/P18-1008 . Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers . Melbourne, Australia: Association for Computational Linguistics: 76–86. arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 1804.09849 https://arxiv.org/abs/1804.09849 . doi https://en.wikipedia.org/wiki/Doi identifier : 10.18653/v1/P18-1008 https://doi.org/10.18653%2Fv1%2FP18-1008 . ↑ cite ref-36 Gordon, Mitchell A; Duh, Kevin; Kaplan, Jared 2021 . "Data and Parameter Scaling Laws for Neural Machine Translation". Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing . Stroudsburg, PA, USA: Association for Computational Linguistics. pp. 5915–5922. doi https://en.wikipedia.org/wiki/Doi identifier : 10.18653/v1/2021.emnlp-main.478 https://doi.org/10.18653%2Fv1%2F2021.emnlp-main.478 . ↑ cite ref-37 Hernandez, Danny; Kaplan, Jared; Henighan, Tom; McCandlish, Sam 2021-02-01 . "Scaling Laws for Transfer". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2102.01293 https://arxiv.org/abs/2102.01293 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-38 Kumar, Tanishq; Ankner, Zachary; Spector, Benjamin F.; Bordelon, Blake; Muennighoff, Niklas; Paul, Mansheej; Pehlevan, Cengiz; Ré, Christopher; Raghunathan, Aditi 2024-11-30 . "Scaling Laws for Precision". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2411.04330 https://arxiv.org/abs/2411.04330 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-39 Sardana, Nikhil; Portes, Jacob; Doubov, Sasha; Frankle, Jonathan 2024-07-18 . "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2401.00448 https://arxiv.org/abs/2401.00448 cs.LG https://arxiv.org/archive/cs.LG . ↑ cite ref-40 Gadre, Samir Yitzhak; Smyrnis, Georgios; Shankar, Vaishaal; Gururangan, Suchin; Wortsman, Mitchell; Shao, Rulin; Mercat, Jean; Fang, Alex; Li, Jeffrey 2024-06-14 . "Language models scale reliably with over-training and on downstream tasks". arXiv https://en.wikipedia.org/wiki/ArXiv identifier : 2403.08540 https://arxiv.org/abs/2403.08540 cs.CL https://arxiv.org/archive/cs.CL .