{"slug": "scaling-is-subtler-than-it-seems", "title": "Scaling is subtler than it seems", "summary": "Scaling machine learning models is more complex than simply increasing size, requiring careful engineering to avoid pitfalls, according to a technical essay by Beren Millidge. The author explains that model capacity, training duration, and data quality trade off against compute, and that the industry's focus has shifted from parameters to data quality and reinforcement learning. The piece underscores that scaling laws, while foundational, demand nuanced understanding of signal-to-noise ratios and optimizability.", "body_md": "**Epistemic status**: *A lot of this is vibes and generally trying to express tacit knowledge from ML experience and training models. However there are a huge number of details here and the field of ML changes incredibly rapidly. This is an oversimplified, but hopefully still useful picture, and may well be wrong in some details since in a good number of cases we understand things at a practitioner’s level but not super well theoretically.*\n\nPeople nowadays often think of scaling laws as obvious and almost trivial – if you have a bigger model and use more compute to train it, then it will get better. This is obviously true at some level and is now the foundation of the entire AI industry 1. However, here I just wanted to write a quick note exploring how things are not necessarily as easy as this in practice. Scaling definitely does work, but it requires very careful engineering and avoiding numerous pitfalls to actually succeed. This explains why the field as a whole did not immediately discover scaling, since simply making models bigger is trivial to test at smaller scales, and where so much of the engineering and science behind scaling and training big models actually goes. Here we will mostly be thinking about things from a pretraining perspective. RL changes the game\n\n[yet again](https://www.beren.io/2026-07-26-How-Can-LLM-RL-Work-Despite-Information-Theoretic-Inefficiency/)\n\n.\n\n[2](#fn:2)Let us start from what (pretrained) models fundamentally *are*. They are [amortized approximators](https://www.beren.io/2022-09-25-Deconfusing-direct-vs-amortized-optimization/) of the data distribution. Training effectively transmutes and compresses the data distribution into the model which then learns a shadow of that distribution. The fidelity and detail that the model learns depends on its inherent capacity, the model’s numerical capacity for signal propagation of activations and gradients, and the SNR by which the optimizer can sculpt the data distribution into the weights, and the optimizability of the loss landscape coupled with the optimizer itself.\n\nFollowing on from my previous post on [scaling laws](https://www.beren.io/2025-03-01-The-Scaling-Laws-Are-In-Our-Stars-Not-Ourselves/), during training we can intuitively think about a model as sequentially learning deeper and deeper ‘features’ of its training dataset. First it learns basic correlations which happen all of the time. Then it learns more complex correlations that are rarer or have more moving parts. If you keep training, it keeps learning until either it runs out of capacity to learn, the signal to noise ratio of the gradients becomes too low so its ‘rate of learning’ and ‘rate of forgetting’ equilibrate, or we run out of new data to continue learning from. We can intuitively think of this as analogous to a telescope. The capacity of the model is like the aperture size. If we make it bigger we can capture more photons/correlations in the data and hence resolve finer features. The length of training or amount of unique data is like the exposure time. Even with a fixed aperture size, a longer exposure lets us gather more photons and average out the noise over time, also allowing us to resolve finer features. To get a good image we can either have a big aperture and a short exposure time, a small aperture and a long exposure time, or in the ideal case, both. The scaling laws such as Kaplan and Chinchilla basically tell us the way these two factors trade-off against one another since both are ultimately constrained by the same scarce resource: compute 3.\n\nInterestingly, historically since GPT3 the industry narrative has been resolutely committed to ‘scaling’. However, the target of ‘scaling’ has shifted amorphously between scaling parameters, scaling compute, scaling data (and crucially data ‘quality’) and finally scaling RL and reasoning training, chain of thought length, and test-time-compute. Only in the relationship between model parameters, a fixed pretraining data distribution (but of varying size), and validation-set loss on that corpus do we have clear ‘scaling laws’, and yet the broad idea of ‘scaling’ applies and is effective much more broadly. In some sense this is not surprising since it is hard to see how, if designed well, adding more compute, data, parameters etc could not be helpful. But these things often become conflated and people apply the scientific notion of proven scaling laws much more broadly than is technically warranted.\n\nMoreover, scaling does not succeed automatically. It is a surprisingly subtle phenomenon which requires many components to work well simultaneously. Let’s come back to our telescope metaphor. For the model to produce a good image of the data distribution and, crucially, for scaling the ‘aperture’ and ‘exposure time’ of the ‘telescope’ to produce continually better images we need there to be enough data in the photons actually arriving to produce an image. There must be an actual image out there to capture. There must be sufficiently low internal noise in the telescope to be able to see anything in the image to begin with. The mechanism for actually making the photons inscribe the image on some medium must work, etc. Analogously, for training and scaling a model, we need all of the following constraints to be satisfied:\n\n1.) An objective which, when minimized, results in ever better approximations to the data distribution. Ideally, this objective is unbounded in potential complexity vs e.g. some loss that is fairly easily solvable or some loss which diverges from modelling the data distribution to doing something else when minimized.\n\n2.) A data distribution with almost infinite, fractal-style, but ultimately learnable complexity. The complexity cannot be too low otherwise a model can perfectly learn the distribution and thus there are no more gains from scale. Alternatively, the complexity cannot be too high since there will not be meaningful signal to learn from. We need a data distribution which happens to have almost a continuous curriculum of harder and rarer features to keep learning as we increase scale.\n\n3.) A model architecture which can actually express the functions implied by the generative process of the data distribution to be able to model it, while remaining numerically stable, optimizable, and capable of propagating meaningful gradient signal into the majority of its parameters throughout training.\n\n4.) A training procedure and optimizer which is stable and can continue reducing loss up to the limits provided by the data and architecture. Many optimizers fail this, including naive SGD + momentum. We only really became capable of scaling when we reached the Adam era. Similarly, you need careful scheduling of learning rate, batch size, and other hyperparameters of training to maintain high SNR and keep learning progressing and to stop it destabilizing.\n\n5.) We need sufficiently good systems and infrastructure to actually allow training at scale. This includes large-scale data, parallelism, and actual training infrastructure, methods to select good hyperparameters, methods to track numerics and instabilities throughout training, and so on. Now that most of the research on points 1-4 has already been done, this is often the largest challenge in practice.\n\nWe can think that historically we entered the ‘scaling regime’ at the point at which we solved these constraints sufficiently well that they no longer bottleneck training. Once the bottlenecks are pushed back, then the regime of smooth scaling across orders of magnitude of compute and data was revealed.\n\nFirst, let’s begin with the objective. LLMs, our most ‘scaled’ object by far, are trained with a next-token-prediction (NTP) objective which is mathematically literally matching the empirical distribution of the data. The autoregressive process is literally the conditional factorization of the joint density over the data. The beauty of this approach is that the reconstruction objective implicitly requires compressing and preserving all of the predictively-useful information in the data, and there are no shortcuts or potential cheats (at least with respect to the objective; the model is free to learn all kinds of memorizing and surface correlations to predict the next token while you might want it to pursue a different strategy). This is different from other objectives such as classification and contrastive losses etc where the loss itself can be minimized without necessarily perfectly matching or ‘understanding’ the data distribution. More importantly, the reconstruction loss itself cannot be gamed. There is no way to minimize loss without being able to predict (in a perfectly calibrated way with cross-entropy!) the actual next token. Moreover, the bit-rate of the NTP loss is extremely high, achieving up to log vocab-size bits per token where the only main way to go higher is with logit distillation. The autoregressive objective in transformers during pretraining is thus essentially optimal in terms of being a direct and unbiased estimator of log-loss, information-density/bitrate, information-preservation and other desiderata which explains, I think, both its incredible success and stubborn endurance despite many other aspects of the field changing.\n\nAfter the objective, the data distribution itself is perhaps the most important thing, although this can easily be forgotten in the age of huge internet-scale datasets, which mostly, due to largely unknown processes, appear to have roughly the right structure 4. Earlier, much more constrained datasets have much weaker or generally weird structures. Moreover, vastly larger datasets reduce the problem of overfitting substantially and push the model towards generalization over memorization. This is because firstly we tend to do way less epochs. When ML was training on MNIST 100 epochs was typical, now an LLM almost always does\n\n[less than 4](https://arxiv.org/pdf/2305.16264)and even then only on highly selected ‘high quality’ subsets. Secondly, and more speculatively, the generality and structure of large-scale datasets rewards generalization much more. This is because many datapoints can be seen as different views of the same underlying object. The same concept might be written about hundreds of different times in slightly different ways. This combination of variety and recurrence likely makes generalization and ‘truly grokking’ the underlying substrate more favoured in the loss landscape than memorizing shallow features since it becomes more parameter- and loss-efficient to fit the shared global structure than having to separately fit every particular instance of the structure. This is unlike earlier datasets which typically had a much less diverse set of presentations of the data, even of the same class, which encouraged overfitting to more surface-level features of the domain.\n\nThe data provides some very important caveats to scaling laws. Firstly, *the model learns the dataset* not necessarily what you want. Often these suffer from some kind of [tails come apart effect](https://www.lesswrong.com/posts/dC7mP5nSwvpL65Qu5/why-the-tails-come-apart). What you want is generally not specified precisely by the data that you have, at least at massive scale 5. People started out pretraining LLMs on common-crawl because it is what they had. Generally, they did not want random common-crawl text predictors. They wanted models that are generally intelligent, can write code, solve math problems, and do AI research etc. Now when capability is very low, improving on common crawl improves on these tasks too, since common crawl includes some math, code, etc amidst all the other random internet crap. However, as the scale increases these objectives become less correlated. Becoming increasingly good at predicting internet text does\n\n[not necessarily mean](https://www.beren.io/2025-08-03-The-Limit-Of-Prediction-Is-Not-Omniscience/)getting dramatically better at solving novel math problems that are extremely rarely represented in the internet corpus (technically, these should still be positively correlated – scaling even on common crawl should help improve even high level math capabilities, but the slope of the improvement will become extremely low).\n\nWe can think about this at a high level in terms of the spectrum of the features. If we take common crawl, the ‘features’ that we may want might be pretty far down the spectrum – i.e. the model will learn all manner of weird internet forum styles, celebrity gossip, news, reddit posts etc before it learns any nontrivial math or AI research. That information is *there* but it requires a ridiculous amount of model capacity (size), and data to robustly reach these features. If, on the other hand, we carefully curate our dataset to primarily consist of exactly the kind of texts we want our model to be able to output, then the features that we desire are *much closer to the surface* and require much smaller models and less data to be able to reach and robustly represent them. As we [discussed previously](https://www.beren.io/2025-08-02-Most-Algorithmic-Progress-is-Data-Progress/), this means that a huge amount of ‘AI progress’ can be simulated by simply training on better data, making the features that are perceived as progress – i.e. math, coding, science skill and knowledge – closer to the surface. This is also why, in my opinion, we now have e.g. [tiny models](https://arxiv.org/abs/2605.05365) which are vastly ‘more capable’ than massive models from a few years ago. The data distribution is vastly closer to the distribution of what we consider core skills (and cynically, the evals) than it used to be for these models. There is also obviously other AI progress involved such as better architectures, [optimizers](https://kellerjordan.github.io/posts/muon/), and just training [for](https://arxiv.org/abs/2401.00448) [much](https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications) [longer](https://arxiv.org/abs/2604.01411), but a lot of this comes down to data.\n\nThis goes against a widespread and natural intuition that stuff like math and code and AI research is ‘hard’ in an objective sense while celebrity gossip, reddit posts, etc is ‘easy’ so that if we produce a dataset of just math the model will struggle and that it is better to create some kind of curriculum so that the model can learn the ‘easy’ stuff first and then the ‘hard’ stuff later, which is how humans learn. This seems completely false. Transformers seem (somehow) to be almost universally plastic at initialization. Any data distribution you throw at them they will learn, whether it is insanely challenging math or repetitive ad copy or malformed HTML pages 6. Interestingly, and this is perhaps an example of Moravec’s paradox, is that if you just look at loss, challenging (to humans) math and code problems often have less perplexity for good models than random internet text (!). This is because even though the content is intellectually challenging, these texts are often pretty predictable if you understand the intellectual structure behind them. They have low levels of intrinsic randomness. However, some random guy on the internet’s blog might be highly unpredictable. He might choose to use extremely idiosyncratic word choices for no apparent reason, go off on random tangents, use names of individual people he knows who are completely unpredictable from the rest of the corpus etc while, e.g. a math textbook is highly unlikely to do any of these things and is written in a generally formulaic academic style.\n\nThe core requirements of ‘good data’ for scaling then are as follows:\n\n1.) The data distribution aligns as closely as possible with the downstream capabilities you want the model to have.\n\n2.) The data distribution has a high degree of intrinsic complexity that is nevertheless learnable. If we are going to scale we need the data to contain fractal manifolds within it, where each separate feature can constitute an improvement on tasks we care about\n\n3.) The dataset we can obtain is sufficiently large for rare features to repeat often enough for us to resolve them, sufficiently diverse that features are represented in many different guises, thus encouraging generalizing representations, and yet the features are convergent enough that there is real underlying abstract structure that can be learnt vs a succession of things one after another presented with no rhyme, reason, or commonality.\n\nWhat we discover in practice is that for large enough datasets of naturalistic text, we usually get points 2.) and 3.) ‘for free’ 7. The name of the game primarily is point 1.:\n\n*making the data distribution match the downstream tasks we want the model to be good at*. As AI capabilities have advanced dramatically, this is becoming harder and harder to do since the things we now care about are not just speaking english and solving grade-school math problems but\n\n*proving novel theorems, inventing zero-day cyberattacks, constructing entire codebases, and pursuing novel AI research bets*. These are, unsurprisingly given their intrinsic difficulty for humans,\n\n*extremely rare*in naturalistic datasets. This thus requires a huge industry of data providers that pay humans to undertake these tasks and provide the outputs and thinking traces to AI labs or produce a synthetic data pipeline to bootstrap examples of otherwise incredibly rare capabilities and skills so that they can be trained into the next round of models\n\n.\n\n[8](#fn:8)Next, even if you have good data, you need a model architecture which you can compress that data distribution into. In theory, almost any architecture can do the job since a two layer MLP with a nonlinearity is sufficiently expressive to model any mathematical function. In practice, things are vastly more restrictive. The infinitely-wide MLP is bad because it does not have the capacity to build [hierarchical features](https://www.beren.io/2023-07-11-Loss-landscapes-and-understanding-deep-learning/) (which is why neural networks [need depth](https://www.lesswrong.com/posts/iNaLHBaqh3mL45aH8/magna-alta-doctrina)) which means hierarchical features must be represented in a ridiculously inefficient way, and secondly because, depending on the initialization, the central-limit theorem will average out its activations giving [near-trivial linear training dynamics](https://arxiv.org/abs/1806.07572) (although other initializations can recover the ‘feature-learning’ regime even for infinite width). For an architecture to be good it needs a few core properties:\n\nFirstly, and most critically, the architecture must be numerically stable and continue to be so with scale. This means that in depth and width we want the activations and gradients of the model to maintain roughly stable norms with scale alongside a maximum eigenvalue of 1 (to prevent explosion), and a generally healthy spectral decay of its eigenmodes. We also need good signal propagation for the model to be trainable – i.e. we want the gradients to be well-scaled and relatively uncorrelated with themselves (to prevent rank-collapse) across layers and neurons to maximize information flow and absorption during training. We need as many parameters as possible to be in the sensitive region of their numerical range – i.e. not stuck at e.g. the flat bottom or top of a sigmoid. We need neither exploding or vanishing gradients in depth. We need gradients to have an ideally full-rank backprojection back through the model rather than being downprojected through some unavoidable bottleneck. And we need nonlinearities that are sufficiently nonlinear to give nontrivial function approximation capabilities and be capable of forming efficient circuits but at the same time remain efficient to compute and numerically stable in both forward and backward pass across a wide range. If we fail at any of these our network will either explode and diverge, or eventually training progress will grind to a halt and we will not be able to scale 9.\n\nSecondly, the model architecture must be sufficiently expressive to be able to handle the data distribution and be efficiently trainable, while not encoding too strong inductive biases about the data distribution which can be false. This latter point is the core argument of [the bitter lesson](https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf). At small scales we do not have the capacity to properly model the entire distribution, so using a bias that encodes part of the structure ‘by hand’ can speed up training, increase sample efficiency, and generally improve performance. However, no real-world data distribution perfectly matches our hand-coded biases, so in the long run with sufficient compute and data it is better to encode the most minimal prior that sufficiently covers the hypothesis space. In line with this, although people have proposed all kinds of crazy architectures over the years, things have broadly standardized on a couple of extremely broad and non-specific primitives. MLP layers are essentially the most expressive possible block since they encode arbitrary X->X nonlinear transformers. Multiplicative gating and norms are broadly effective and can be tuned to implement a huge range of computational primitives (including doing nothing). Attention is essentially the most general second-order interaction possible since it faithfully computes the entire quadratic interaction space of every token interacting with every other token. Convolutions are a bit of a special case in that they indubitably encode quite a strong inductive bias, but this bias maps extremely well to the physical processes underlying the generation of the data in e.g. visual and audio modalities, which is where they are primarily used. In language, which has no innate convolutional smoothing, the use of convolutions is much rarer.\n\nIt is interesting to ponder transformers through this lens, since it becomes extremely obvious why they are so general. A transformer is just an interlocking series of attention and MLP blocks joined with a residual stream. The residual stream enables gradients and activations to be propagated stably through depth. The MLPs are two-layer which is the smallest number of MLP layers you can have while retaining a nonlinearity, and this is done so that almost all complex circuits are routed through the residual stream, again for signal propagation reasons. Attention is the simplest but fully general input-dependent quadratic sequence-to-sequence mapping possible. The query, key, and value weights are essentially linear hypernetworks that generate a sequence-dependent KV cache. Then, from the perspective of decode with a frozen KV cache, attention looks just like a two-layer MLP across time (with key cache as input weights and value cache as output weights) with a softmax activation function 10.\n\nMy thinking here is that the transformer is perhaps the first architecture in a general universality class of sufficiently expressive and parallelizably trainable general sequence mixers, as well as one of the simplest members of this class. Obviously there can be elaborations upon the transformer which improve certain components by adding additional biases or by giving up some general expressivity for computational benefit, but the transformer itself is a masterpiece of simplicity 11.\n\nSo, to sum up here. Scaling successfully requires five key components. We need an objective which is unbiased, has no degenerate minima, and encourages extracting a potentially unlimited amount of information from the dataset. Second, we need a dataset worthy of the objective. It must itself be sufficiently large and have fractal-like complexity rather than a finite amount of extractable information which is easily reached. The dataset must also be large enough that this information can be expressed in a sufficient number of slightly different forms and perspective that learning generalizing, rather than memorizing, features becomes energetically preferred. Thirdly, we need a model architecture with sufficient expressivity to be able to represent the intrinsic generating functions of the dataset in a reasonably efficient way. This architecture also needs to be numerically stable upon scaling both in width and depth, and enable activation and gradient propagation forward and backward while retaining a good numerical SNR and ideally sensible eigenvalue distributions in the weights. Fourthly, we need a generic optimizer which can move decently across the loss landscape and be tolerant of slight numerical glitches, while being tractable to compute on a system with a very large number of parameters. And finally we need the infrastructure and systems to be able to actually scale in practice.\n\nConversely, we then get quite a long list of ways to fail at scaling, which all must be avoided for success.\n\n1.) We could choose a bad objective. NTP is amazing here because it maximizes information compression and recall, it is trivial to compute in parallel, is stable and unhackable, and has extremely high bit-rate. Other objectives tend to fail some of these criteria. Classification objectives often require only a small subset of the information content to achieve high accuracies. Diffusion objectives also theoretically preserve all the information, since they also directly estimate the score, but the noise adds variance and it is often challenging to maintain high learning signal throughout the entire noise schedule 12. GAN objectives are typically theoretically convergent but highly unstable in practice due to the adversarial minimax-game formulation, and can quickly end up in a pathological fixed point whenever the generator or discriminator overpowers the other. Contrastive objectives need considerable care to make them unhackable and also often do not require full information preservation to solve.\n\n2.) The data could be too simple or too irreducibly complex. If there is either no structure in the data, or the structure can be easily learnt, or that there is some cap of the complexity where the structure starts out learnable but the rest is covered by ridiculous amounts of noise, then scaling cannot help you progress further.\n\n3.) The data could meaningfully diverge from what you want. If this is the case then scaling will help you better learn the data, but if the data isn’t what you want then it will not help on downstream tasks. This is basically just regular tails-come-apart reasoning.\n\n4.) The data contains subtle correlations which provide shortcuts for the model to cheat by learning or memorizing ephemeral surface correlations rather than the thing you want. If this is the case then scale may make the model more likely to do this, especially if the shortcut is very subtle.\n\n5.) The model architecture is insufficiently expressive to capture the underlying structure of the data. If this is the case then in Bayesian terms you have a mis-specification issue. The true hypothesis is not in your hypothesis set. If this is the case then depending on how inexpressive the architecture is, scaling will either help a bit but weakly as you get increasingly good but convoluted and arcane approximations of the capability you can’t properly express, or it just won’t help at all if there is a fundamentally missing capability.\n\n6.) The model architecture is fine theoretically but in practice the signal propagation dynamics are poor enough that you don’t get good scaling. This held up scaling for a long time and required architectural innovations like residual networks, layernorm, qk-norm, and novel expert balancing algorithms etc to fix. Notably, at larger scales the norms and dynamics problems tend to get worse – i.e. for random gaussian networks the norms scale as the square-root of the width. Similarly, scaling in depth even in residual architectures still has problems with only [very](https://arxiv.org/pdf/2606.16112) [recent](https://arxiv.org/abs/2603.15031) attempted solutions 13.\n\n7.) You can theoretically scale well architecturally and dynamics wise but you select bad hyperparameters so you scale very inefficiently and it is prohibitively expensive to do hyperparameter sweeps at scale. We needed very explicit understandings of signal propagation and norm control as they relate to training hyperparameters such as methods like [MuP](https://arxiv.org/abs/2203.03466) and [unit scaling](https://arxiv.org/abs/2407.17465) to solve this\n\n8.) Your architecture and hyperparameters are fine but your training dynamics are poor because of choice of optimizer, or other issues. This is generally less of an issue but is still possible if you choose poor optimizers. AdamW is similar to the transformer in that it is the first optimizer in the basic universality class able to train at almost indefinite scale. Muon seems to be an improvement over AdamW and if you want to strongly control optimizer norms you can use [Hyperball](https://arxiv.org/abs/2606.16899), which also appears effective. If you tried scaling with pure SGD or even SGD + momentum you would generally be in for a bad time.\n\n9.) Despite having all of this theoretically working, you have some subtle bug in your code that causes you to fail. There is an almost innumerable amount of these. It is surprisingly easy to have things subtly broken which often only slightly degrade the model at small scales but which blow up to be major issues at large scale. One very pernicious category, although one that is often obvious in retrospect, is subtle causal leaks which eventually destroy autoregression by providing shortcuts even if only a tiny number of bits leak. For subtle leaks these are often fine at small scale since the model does not have the capacity to discover or exploit the leak, but cause baffling collapses at large scale. A classic place to see subtle leaks is in normalization for instance in top-k across time as is done in naive [expert choice](https://arxiv.org/abs/2202.09368) implementations. Empirical ML is an engineering discipline requiring intense attention to subtle detail to avoid these kinds of silly failures.\n\n10.) Finally, you just need enough raw compute to be able to scale things at all. Without enough compute which is widely distributed, and the infrastructure for using it coherently, you can’t even get to see these problems in the first place.\n\nFrom a historical perspective, this is why I think it took so long and was so difficult to ‘discover’ scaling as a field. It isn’t as trivial as it sounds. Scaling is not an operation but a regime. If you try scaling in most cases you will fail. You need to have solved the constraints above sufficiently well to start to see consistent returns from scaling, and then you need to be able to stay in this regime across OOMs of scale as each of these constraints become more demanding in order to continue to see benefits from scaling. Moreover, the full ‘scaling package’ that we have today requires multiple independent pieces to snap into place in just the right way before it really takes off, and although most of these points seem obvious now, they were not obvious a-priori. This then also reframes the bitter lesson which we can think of as arguing essentially that first, due to Moore’s law, we should expect that scaling compute becomes increasingly powerful over time, and secondly that because of this we should expect that methods that remove the bottlenecks enabling us to reach or stay in this smooth scaling regime are the ones that become the most important, since they allow us to deploy the exponentially increasing amount of compute we have access to in a way that yields consistently positive returns.\n\nThe fractal but learnable complexity structure of web-scale naturalistic datasets provides the landscape on which\n\n-\nIt is very important to note, and surprisingly unremarked upon, that despite the industry continually touting scaling, the actual Kaplan/Chinchilla style scaling has played a relatively small part in the dramatic progress in AI. Nobody does Chinchilla-style LLM pretraining on web-corpora anymore.\n\n[↩](#fnref:1) -\nIt is interesting to note that even after the massive success of reasoning models using RL, there are still no (publicly) widely accepted empirical scaling laws in the same way that we have\n\n[Chinchilla](https://arxiv.org/abs/2203.15556)for pretraining. Perhaps[this](https://arxiv.org/html/2510.13786v1)is the closest serious attempt at this, although likely various labs have similar internal scaling studies. Generally, although RL definitely does get better with scale, there are even more subtleties than with pretraining, or at least we understand the pretraining subtleties pretty well while this understanding is at a much earlier stage with RL.[↩](#fnref:2) -\nAn important point here is that the scaling laws only tell you about validation loss on some particular dataset. Whether improving loss on this dataset actually improves the downstream capabilities you care about is another thing entirely.\n\n[↩](#fnref:3) -\nThis is because the power-law structure of naturalistic datasets likely roughly recapitulates the high level statistical structure of the real world, at least to a much higher degree of fidelity than prior datasets.\n\n[↩](#fnref:4) -\nNote that this does not fall out of the mathematical form of scaling laws. Scaling laws tend to be formulated in terms of training or validation perplexity\n\n*on the training dataset*. The relationship between the scaling laws (and hence train loss) and downstream evaluations is much more contested for precisely this reason – that it depends on the correlations between the train set and the downstream evaluation set itself.[↩](#fnref:5) -\nCurricula actually generally hurt performance because they introduce nonstationary data which causes plasticity and forgetting issues as the training objective is designed to be i.i.d. The reason models use curriculums and multiple training stages is because of scarcity of high quality data representing exactly what you want. This high quality data has a larger effect on the model if placed at the end where it forms a high proportion of the dataset rather than scattered uniformly throughout a much larger set during training. If, however, we just had a massive dataset of only ‘difficult’ data, it would be much better to use that than any curriculum.\n\n[↩](#fnref:6) -\nAlthough we have to be careful here with synthetic data which can often be incredibly repetitive and thus prone to overfitting if done poorly.\n\n[↩](#fnref:7) -\nI frame this as\n\n*distillation from humans*in my previous[post on distillation](https://www.beren.io/2026-06-12-Distillation-Aint-What-It-Used-To-Be/). One complexity here is that we no longer just do pretraining but also (and now primarily) RL. RL operates in a mostly similar but slightly different regime and here much of the challenge is constructing environments and verifiers rather than just sourcing data.[↩](#fnref:8) -\nEven existing transformers are not perfect here. They appear to\n\n[lose plasticity](https://arxiv.org/abs/2606.24752)over long periods of training and scaling parameter count delays but does not appear to prevent this transition. This implies that training somehow pushes these models out of their relatively nice initialization region and into some numerical traps of uncertain origin.[↩](#fnref:9) -\nOne important note here is that while attention itself encodes almost no inductive bias, the\n\n*position encoding scheme*often encodes a huge amount of bias. Early versions such as Alibi expressly encoded a geometric decrease in the strength of token-token correlations across time while the almost ubiquitous[RoPE](https://blog.eleuther.ai/rotary-embeddings/)encodes essentially an ensemble of such biases operating at different frequencies. The empirical success of these methods implies that this bias matches something reasonably close to the actual inherent structure of natural language where, indeed, token-token correlations decline roughly as a power-law in the distance between the tokens.[↩](#fnref:10) -\nAnd it has taken me and the field as a whole a surprisingly long time to realize this.\n\n[↩](#fnref:11) -\nTypically, the problem is basically impossible at the high noise end and trivial at the low noise end of the schedule so only the middle of the schedule provides real learning signal, and importantly the location of where signal can be extracted changes during training as well.\n\n[↩](#fnref:12) -\nImportant to note here is that despite all our talk of scaling we are often going\n\n*shallower*in depth. Good ol’[Llama3-405B](https://huggingface.co/meta-llama/Llama-3.1-405B/blob/main/config.json)has 126 layer while[Deepseek-v4](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/config.json)has 61 and even[Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3/blob/main/config.json)only has 93.[↩](#fnref:13)", "url": "https://wpnews.pro/news/scaling-is-subtler-than-it-seems", "canonical_source": "https://www.beren.io/2026-08-15-Scaling-is-Subtler-than-it-Seems/", "published_at": "2026-08-15 07:00:00+00:00", "updated_at": "2026-08-15 21:41:28.164456+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research"], "entities": ["Beren Millidge", "GPT-3", "Kaplan", "Chinchilla"], "alternates": {"html": "https://wpnews.pro/news/scaling-is-subtler-than-it-seems", "markdown": "https://wpnews.pro/news/scaling-is-subtler-than-it-seems.md", "text": "https://wpnews.pro/news/scaling-is-subtler-than-it-seems.txt", "jsonld": "https://wpnews.pro/news/scaling-is-subtler-than-it-seems.jsonld"}}