{"slug": "neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters", "title": "Neural Scaling Laws: Why Loss Falls as a Power Law in Compute, Data and Parameters", "summary": "Research on neural scaling laws, originating with Kaplan and colleagues at OpenAI in 2020 and refined by the Chinchilla team at DeepMind in 2022, shows that transformer language model loss falls as a power law in parameters, data and compute, with fitted exponents of roughly 0.076, 0.095 and 0.050 and an irreducible floor near 1.69 nats per token. The Chinchilla work, which trained over 400 models from 70M to 16B parameters, overturned Kaplan's prescription by showing model size and training data should scale in equal proportion, and that a 70B-parameter Chinchilla model trained on 1.4 trillion tokens beat the 280B-parameter Gopher on nearly every benchmark at equal compute.", "body_md": "Train a language model ten times bigger and the loss does not fall by a fixed amount. It falls by a predictable fraction, and the same is true when you multiply the data or the compute. This regularity, discovered by Kaplan and co-workers at OpenAI in 2020 and refined by the Chinchilla team at DeepMind in 2022, is one of the few quantitative laws in deep learning. It tells you, before you spend the money, how much better a bigger run will be, and where that money is best spent.\n\nLet L be the test cross-entropy loss of a transformer language model, N the number of non-embedding parameters, D the number of training tokens, and C the total training compute in FLOPs. When each factor is varied on its own, with the others effectively unlimited, the loss follows a power law plus an irreducible floor:\n\nThe fitted exponents are roughly αN≈0.076 , αD≈0.095 and αC≈0.050 . The floor L∞≈1.69 nats per token is the entropy of natural text itself: no model, however large, can predict the next token better than the data allows. These numbers were fitted on models up to a few hundred million parameters and data up to tens of billions of tokens, yet they predicted the behaviour of models orders of magnitude larger.\n\nA power law in log-log space is a straight line, which is why the law is so useful. Plot loss against log compute and you get a line, not a curve that bends away. Doubling compute buys the same fractional improvement at 10^20 FLOPs as it did at 10^18. The practical consequence is blunt: returns diminish in absolute terms but never vanish, so there is always a reason to scale, and never a reason to expect a sudden jump from scale alone.\n\nThe single-factor laws are only half the story. In practice you have a fixed compute budget C , and you must split it between parameters and tokens. Since a forward-backward pass costs about 6N FLOPs per token, the budget constraint is approximately:\n\nKaplan's team fitted the joint law:\n\nand minimised it subject to the compute constraint. The result is that optimal model size and optimal data both grow as powers of compute:\n\nThat is the Kaplan prescription: as your budget grows, spend most of the extra compute on a bigger model and only a little on more data. For years this was taken as gospel, and it shaped a generation of models trained on relatively few tokens. GPT-3, with 175 billion parameters trained on 300 billion tokens, sits roughly on this prescription.\n\nThe Chinchilla paper (Hoffmann et al., 2022) re-ran the experiment properly. Instead of extrapolating from small models, the team trained more than 400 models from 70 million to 16 billion parameters, on 5 billion to 500 billion tokens, and fitted the frontier directly. Their conclusion overturned the Kaplan split:\n\nModel size and data should scale in equal proportion. The famous demonstration: Chinchilla, a 70 billion parameter model trained on 1.4 trillion tokens, beat Gopher, a 280 billion parameter model trained on 300 billion tokens, on nearly every benchmark, while using the same training compute. Gopher was four times larger and four times worse trained. The field had been systematically under-training its models.\n\nThe discrepancy came from methodology. Kaplan's fits extrapolated the frontier from models trained far from the optimum, and their learning rate schedule and token counts biased the exponents. Chinchilla's approach, fitting the envelope of the best loss achieved at each compute level across a dense grid, is the more trustworthy estimate. The lesson is not just the new exponents; it is that scaling laws are only as good as the experiments that fit them, and extrapolation from off-frontier runs is dangerous.\n\nWhy should loss follow a power law rather than, say, an exponential? The standard explanation is a distribution of skills. Language contains patterns of every difficulty: common syntax, rare facts, long-range dependencies. Each pattern is learned once the model has enough capacity and has seen it enough times. If the frequency of patterns follows a Zipf-like distribution, and each new pattern reduces loss by a roughly fixed amount, the aggregate loss falls as a power law in both parameters and data. The irreducible floor is the part of language that is genuinely unpredictable: the exact wording of a sentence that could have gone a dozen equally good ways.\n\nThis picture also explains the two regimes you see in practice. In the data-limited regime, adding parameters does nothing because the model has already memorised what the data can teach. In the parameter-limited regime, adding data does nothing because the model cannot represent what the extra data contains. The frontier is the boundary between the two, and the Chinchilla-optimal point is where the marginal return from one more parameter equals the marginal return from one more token.\n\nYou do not need a supercomputer to use this machinery. If you train the same architecture at three or four sizes, you can fit the exponent yourself. In log space, the power law is linear:\n\nA minimal fit in Python looks like this:\n\n``` python\nimport numpy as np\n\n# Model sizes and final test losses from your own runs\nN = np.array([1e8, 3e8, 1e9, 3e9])       # parameters\nL = np.array([2.85, 2.62, 2.41, 2.24])   # test loss in nats\nL_inf = 1.69                              # irreducible loss estimate\n\nx = np.log(N)\ny = np.log(L - L_inf)\nslope, intercept = np.polyfit(x, y, 1)\nalpha_N = -slope\nprint(f\"alpha_N = {alpha_N:.3f}\")\n\n# Predict the loss of a 10B model\nN_new = 1e10\nL_pred = L_inf + np.exp(intercept) * N_new ** (-alpha_N)\nprint(f\"Predicted loss at 10B params: {L_pred:.3f}\")\n```\n\nTwo warnings apply. First, the fit is only as good as your estimate of L∞ ; fixing it wrong bends the line. Fit it jointly with the exponent if you have enough points. Second, every point must be trained to convergence with a tuned learning rate and a proper schedule. An under-trained point sits above the line and corrupts the slope, which is exactly the error that misled the original Kaplan split.\n\nScaling laws predict pre-training loss, not downstream ability. Benchmark accuracy is a noisy, often saturating function of loss, and capabilities can appear abruptly even while loss improves smoothly. The \"emergent abilities\" debate is largely about this gap: a smooth loss curve can produce a sharp accuracy jump when a task crosses a threshold. Scaling laws say nothing about where those thresholds sit.\n\nThey also assume a fixed architecture, a fixed data distribution, and a fixed training recipe. A better architecture shifts the whole curve down; better data does the same. This is why algorithmic progress matters as much as compute: a two-fold improvement in data efficiency is worth as much as doubling the budget, and it compounds. And at inference time the calculus flips. A smaller model trained on far more tokens, like Chinchilla, is cheaper to serve forever, which is why the compute-optimal training point is rarely the cost-optimal deployment point.\n\nFinally, the laws break down at the extremes. Very small models deviate from the fit, and the largest runs have repeatedly shown small but real deviations from the predicted line. Treat a scaling law as a planning tool with error bars, not as a physical constant.\n\nLoss falls as a power law in parameters, data and compute, with an irreducible floor set by the entropy of language itself. The actionable form of the law is the Chinchilla split: scale parameters and tokens in equal proportion, and never let a huge model starve on a small dataset. Fit the exponent on your own runs before committing a large budget, keep every point on the frontier, and remember that the law predicts loss, not capability. Scale is a reliable lever, but it is a lever on a curve, not a shortcut around architecture and data quality.\n\n**Connect with me on LinkedIn:** [linkedin.com/in/umeshtharukaofficial](https://www.linkedin.com/in/umeshtharukaofficial/)", "url": "https://wpnews.pro/news/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters", "canonical_source": "https://dev.to/umeshtharukaofficial/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters-3md0", "published_at": "2026-11-16 02:18:30+00:00", "updated_at": "2026-11-16 02:22:05.502737+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["OpenAI", "DeepMind", "Kaplan", "Chinchilla", "Hoffmann", "Gopher", "GPT-3"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters", "markdown": "https://wpnews.pro/news/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters.md", "text": "https://wpnews.pro/news/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters.txt", "jsonld": "https://wpnews.pro/news/neural-scaling-laws-why-loss-falls-as-a-power-law-in-compute-data-and-parameters.jsonld"}}