Whence the Fractals in our Stars?

Beren Millidge, an AI researcher, argues in a blog post that scaling laws in deep learning reflect the intrinsic power-law, fractal structure of natural datasets rather than properties of the architectures themselves, and that objectives like next-token prediction reveal this structure when they do not bottleneck learning. He raises the open question of why naturalistic data exhibits such fractal structure, citing Zipf's law, 1/f noise, and the Matthew effect as examples of ubiquitous power laws.

Author’s note : A follow-up and addendum to my previous The Scaling Laws are in Our Stars’ post. In my earlier post ‘The Scaling Laws are in Our Stars’ https://www.beren.io/2025-03-01-The-Scaling-Laws-Are-In-Our-Stars-Not-Ourselves/ , and then in my last few https://www.beren.io/2026-08-15-Scaling-is-Subtler-than-it-Seems/ posts https://www.beren.io/2026-08-23-Architecture-Research-as-Addressing-Constraints-to-Scaling/ on scaling, we have begun to pull together some kind of high level intuitive theory for why we get scaling laws and more generally why deep learning works and what it is doing. Specifically, the argument is essentially that, when designed well, deep learning models are just extremely general learners and compressors of structure in the datasets. The reason we observe scaling laws, then, is not so much a fact about the architectures or deep learning itself, per se, but rather that this kind of power-law structure of increasingly rare features is ubiquitous to natural datasets. The loss of the model with scale then principally reflects the marginal benefit the model obtains by learning an additional feature from the dataset, and thus its broad statistics are primarily shaped by the structure of the dataset itself. We can kind of think of this as a three-way multiplicative process. The ultimate source of the scaling laws are in the data, but they are only fully ‘revealed’ to us when the architecture, training dynamics, etc of the model, and the objective we use to train the model do not bottleneck learning and thus prevent this structure from being learnt. Scaling and scaling laws are thus the regime where everything else is sorted out so that the model just becomes essentially a compressed mirror of the dataset, thus the shape of the scaling laws reflects the intrinsic ‘shape’ of the data. The objective is also important here and often forgotten. The model learns essentially the structure of the data projected into the space relevant for learning the objective. If the model can trivially satisfice the objective, or the objective only requires looking at a small projection of the data, then the intrinsic structure of the data itself is largely irrelevant. This is why broad, almost information-theoretically maximal objectives like NTP are so vital for seeing this scaling behaviour. Under NTP, effectively all predictive information in the sequence is valuable to the objective, and thus is learned. This contrasts with other potential objectives such as e.g. parts-of-speech tagging where although the data may be maximally structured, the objective is fairly simple and does not require learning all of this structure. This then leads immediately to the question of why large-scale naturalistic data has this very peculiar power-law-like ‘fractal’ kind of structure, where there are features to be learned at every scale and that this keeps going on indefinitely as we keep scaling our models and dataset sizes across many OOMs. In our original post https://www.beren.io/2025-03-01-The-Scaling-Laws-Are-In-Our-Stars-Not-Ourselves/ we just asserted that empirically natural language and many other kinds of dataset have a power-law spectrum of their covariance matrix, and other studies that try to derive scaling laws often start from similarly constructed synthetic data https://arxiv.org/abs/2210.16859 . Of course this still does not answer why the data should have this structure. More broadly, a lot of power-law structures have been observed appearing widely https://arxiv.org/abs/1812.09662 in https://www.mdpi.com/1099-4300/18/4/108 the https://arxiv.org/abs/2512.21539 natural world https://pmc.ncbi.nlm.nih.gov/articles/PMC6480937/ . In language we have Zipf’s law https://en.wikipedia.org/wiki/Zipf%27s law . In signal processing and stochastic processes we have 1/f noise https://arxiv.org/pdf/physics/0204033 . In economics we have power laws in wealth through the Matthew effect https://en.wikipedia.org/wiki/Matthew effect and potentially other mechanisms. In graph theory there are all kinds of graphs which produce power-law statistics. Each of these processes have specific dynamics and generative models that people have come up with to explain them associated with them. Generally, when we see universality of this kind, it is interesting to search for a more abstract and general explanation. For instance, the ubiquity of Gaussians is explained by the CLT and the general maximum entropy argument for Gaussians for processes with constrained mean and variance – and similarly boltzmann distributions emerge from only constrained mean. There is a similar derivation for power-laws from maximum entropy constraints. Specifically, power-laws arise https://arxiv.org/abs/1212.5567 from the constrained variable having constant ‘importance’ over logarithmic scales https://arxiv.org/abs/0906.3507 . I.e., if changing rank from 1 to 10 has the same impact as changing from 10 to 100 then, once we aggregate these distributions under maximum entropy we will end up with a power law. Another way to think about this is to note that a power law is just a standard exponential max-ent tail in logarithmic coordinates. This is why many generative models with primarily multiplicative interactions end up showing power-law statistics although not all since there is a separate convergence basin for lognormals . -/need to make this clearer with the specific max ent details . An important, but not definitive note, is that we might broadly expect these kinds of power-law statistics from compositional hierarchical generative models for the simple fact that conditional probabilities multiply and hence surprise adds in log-space. An important note at this point. In the original post https://www.beren.io/2025-03-01-The-Scaling-Laws-Are-In-Our-Stars-Not-Ourselves/ , we argued that the necessary structure is implied by the spectrum of the data covariance being roughly power-law and Zipfian, which is to be expected of natural language. It turns out that this is necessary but not sufficient. Aitchison has a beautiful paper https://arxiv.org/pdf/1407.7135 describing the emergence of high-level Zipfian statistics from aggregation of a large number of component data distributions with distinct latent variables under fairly mild conditions. The idea here is simple. We have a bunch of distributions generated from separate latent variables $z$ and our final data distribution is created by concatenating/summing these distributions together which is essentially analogous to marginalizing out the latent variable. For every conditional distribution, the data are controlled by the latent variable, and thus the surprise of every conditional distribution of the data given $z$ is heavily influenced by $z$. If we sum together the surprises then we get a CLT-style concentration effect in the logarithmic surprise-space, so that the distribution of surprise for a given latent variable $z$ is somewhat peaked. The move, then, is to assume that the $z$s are diverse enough that the variance in surprise space between different latent variables is larger than the variance within the conditional probability distributions. If this is the case, then although each peak is highly concentrated at some point in surprise-space, the peaks themselves occupy different regions. If we then sum together all of the peaks, we might end up with an almost uniform distribution in surprise space over some region. This uniform distribution over logarithmic surprise is essentially the Zipfian/power-law spectrum that we want since it implies there is structure at every scale and that each scale is roughly equally important. It is interesting to note that although this can produce our desired power-law statistics at the scale of the data, it does not necessarily produce the kind of scaling law learning that we want. This is because surprise at the token level is not necessarily the same as the predictive information contained within the distribution. We can imagine our Aitchison-style aggregation as containing a simple latent variable $z$ such as e.g. the bernoulli probability of a biased coin flip and having that swept over a log scale of values – i.e., p head = 1, 0.1, 0.01 etc. Aggregating the resultant datasets would create a naive power-law-looking spectrum under the Aitchison model but the amount of predictive information here is incredibly low, so a general learner could extract the meaningful information very quickly, and then stop being able to progress further since there is no more information to extract. We thus need a stronger condition – effectively that the predictive information of the latents themselves has this kind of scale-free power-law structure. This is essentially the argument of Tishby https://arxiv.org/pdf/physics/0007070 that characteristic power-law scaling emerges in complex systems with varying correlation lengths including very long-range correlations, and moreover that it is characteristic of nonparametric or infinite-dimensional models vs simpler parametric models with exhaustible predictive latent structure. Now to satisfy this condition there are basically two mechanisms. First, we can have the latent variable structure of some particular dataset naturally possess this kind of scale-free structure. Secondly, we can synthetically aggregate a bunch of not fully scale-free latents together and end up having the entire dataset possess this kind of scale-free power-law structure. This is essentially an extension of the Aitchison result in terms of predictive information, but with some qualifications that the individual latents in the aggregation themselves must vary sufficiently over scales as well as that each new latent in the aggregation contributes new predictive information. In practice, I think we have both mechanisms operating somewhat simultaneously. Many sources of naturalistic data such as language, sufficiently broad image statistics etc naturally end up somewhat scale-free structure-wise with both very important short and long-range correlations. Due to the diversity of natural data, then these scale-free individual latent structures tend to come from pseudo-orthogonal or non-redundant domains. For instance, a chemistry textbook, a series of imagenet images of dogs, a clip of a podcast audio segment, and some teenager’s erotica fanfiction all each both individually possess this kind of semi-fractal latent structure although not to infinite complexity while they each contribute effectively orthogonal and non-redundant and novel predictive mechanisms to the dataset as a whole. The basic idea here is that sufficiently naturalistic data almost intrinsically satisfies this new-predictive-information constraint just because the real world is so vast that if you take random arbitrary slices of the real world, it is highly likely to have non-overlapping predictive information in it at multiple scales. Conversely, this is why purely synthetic data is often subtle and dangerous, since even if a huge amount of apparent complexity is produced at the token level, the actual latent generators of this data may be much simpler than they naively seem, and thus produce less fodder for scaling than you might think. An important point about this aggregation mechanism is that it is permissive . You only need some of the data to have this scale free structure or be distributed widely to get scaling laws. Having some overly simplistic data or some irreducible noise data may either saturate or add to the irreducible loss, and thus affect the exponents and how useful your data distribution is in practice, however it preserves the general power-law structures underneath. This explains why scaling laws first showed up on almost maximally uncurated data. It doesn’t matter so much what the data contains or where it comes from, or even if a good fraction of it is either trivial or garbage so long as it is sufficiently diverse and samples enough of ‘reality’ to produce this structure in the tails. Of course, to actually get good models, which have learnt what you want them to learn, rather than just scaling laws, the data composition or ‘quality’ is of paramount importance. Okay so we have got the aggregation mechanism and pretty mild assumptions, but we still have not yet properly figured out why we should expect any , or at least some decent fraction of the sub-datasets to have precisely this scale-free structure in its latent generators. It turns out, when we really dig into this, that this basically comes from the fact that, firstly, reality is much more complex than the learner , and secondly, that reality is sparse, hierarchical, and compositional . Let’s take the second case first, since it is more intuitive and easier to build up into a picture. It is common and generally intuitive, that many of the ‘latent causes’ of reality are organized into a tree-like hierarchical structure. A book might have an overall theme or arc, then individual characters, then a plot, then specific paragraphs, then sentences, then words etc. The natural biological world has kingdoms then phyla then species etc. Organisms are made up of cells that are made up of various structures such as chromosomes, mitochondria etc, which are made up of molecules, which are made up of atoms, which are made up of protons, electrons, neutrons etc. Scenes are made of objects, which are made of subcomponents with edges etc. Generally, we see that reality possesses a huge amount of hierarchical descriptions where distinctions give rise to even more fine-grained distinctions, seemingly without limit. To visualize and start thinking mathematically about this, we can think of this as a tree with some branching factor. The key fact about this tree, from the learning perspective, is the ratio of predictive information between features at different levels. To get a stable power law we need the ratio of predictive information between levels of the tree to be near-identity – that is, the parent possesses roughly as much predictive information as all the children. At every level there are exponentially more children than parents, and hence each child, individually, must have exponentially decreasing frequency, or predictive information, but the sum of all children must equal the information of the parents. If this doesn’t happen, then either parents have more predictive information than children, which means that almost all the information accrues near the root of the tree, or children have more predictive information than parents meaning that almost all the information is at the utmost leaves. To have a smoothly learnable set of features at all scales simultaneously, we need the child-parent predictive-information-ratio to hover near unity 1. Zipf’s law provides a very intuitive statement of this. One way to think about Zipf’s law is to think about randomly selecting a word from some natural language corpus. Zipf’s law states that the probability of selecting a word in some logarithmic frequency band is equal no matter the frequency band. This is kind of a crazy statement when you think about it, but very precise and important. The probability of sampling a ‘common’ word such as ‘a’, ‘the’ etc is the same as the probability of selecting a rare word such as ‘mellifluous’ . The reason for this, of course, is that for every level of frequency, there are exponentially many more ‘rare’ words than common words – i.e. there are only a few super common words like ‘the’ but a huge diversity of ‘rare’ words like ‘mellifluous’. Specifically, for each rare word you are exponentially less likely to select it than a common word, but at the same time there are exponentially more rare words in total than common words. The two exponentials cancel and you end up with a beautifully constant relationship between rank and frequency. Many naturalistic phenomena are like this. Each fine-grained distinction is, individually, exponentially rarer, but together they are exponentially more common, and the two exponentials cancel. This is very natural to think about in a tree structure where clearly the number of children grows exponentially in depth, but if each parent must effectively split its predictive information between children, then each child must attain exponentially predictive information, and thus retain the ratio. This thus appears to be an extremely general property of hierarchically structured phenomena. 2 fn:2 Moreover, this is not specific to hierarchy, but rather a general property of probabilistic composition, of which hierarchy is just one example. Conjunctive probabilities multiply, and thus if we have N latent variables the chance of some outcome involving specific settings of all N of them decreases as the power function x^N. However, at the same time, the number of possible conjunctions increases combinatorically at the same rate, creating this effect again. In fact, it is even more general than this. It turns out we can derive a similar scaling property of any sufficiently complex infinite-dimensional function with assumptions of finite roughness, or alternatively, some degree of locality. The notion here is to inspect the function at varying levels of resolution. As we increase our ‘resolution’, obviously the amount of points we must inspect grows exponentially in the dimension of the function. However, at the same time, due to our assumption of finite roughness, the amplitude of the potential frequencies between points at our particular resolution must decline. Under mild assumptions, these two factors cancel to produce a scale-free spectrum of features again. Another more signal-processing-y way to think about this is as a wavelet decomposition of some signal. The first modes capture coarse structure, but have large amplitudes and are important for reducing a lot of ‘loss’ about the function. At higher resolutions there are many many more modes, but each mode conveys less residual information about the structure as a whole. Importantly, if our function is actually infinite-dimensional, we never ‘run out’ of information and perfectly capture the function. Rather, we can keep increasing resolution and modelling more and more attenuated modes indefinitely, asymptoting towards infinite resolution and zero amplitude, but never actually reaching it 3. This brings us back to our first property, that the data generating process must be more complex than the learner https://arxiv.org/abs/2512.23419 . This makes our definition of an ‘infinite dimensional function’ effectively true. Obviously the real world is not literally infinite-dimensional. But at the same time, it is vastly more complex than the learner. This means that its complexity can appear infinite without actually being infinite. This is similar to how in statistical mechanics, we happily model systems with extremely large, but ultimately finite, dimensionality as infinite dimensional and nevertheless achieve incredible modelling precision. Moreover, it must be said that due to this hierarchy, the real world also provides many different processes operating at many different timescales with independent generative mechanisms, thus satisfying the conditions for the aggregation of these processes to satisfy the desired scale-free properties. Perhaps a better way to think of this is that a rich, but parametric, process appears non-parametric if sufficiently larger than the capacity of the learner. Pulling this all together we get a very nice synthesis. The kind of scale-free power-law fractal-like structure we need for scaling laws is ubiquitous in natural data because of two core facts. Firstly, that reality is much more complex than the learner. This means that reality functions as a natural kind of ‘infinite-dimensional object’ but which nevertheless has certain smoothness, locality, and regularity conditions, and which can be inspected at different resolutions while remaining ‘generative’ at all levels. This kind of mathematical structure under max-ent constraints produces exactly the kind of scale-free structure we are looking for to learn from. Secondarily, even if the individual underlying latent variables are not scale free, simply aggregating enough variables which span a sufficiently large range of scales and are novel non-redundant mechanisms which can contribute additional predictive information to the ensemble is also sufficient. Once we have this scale-free structure in our data, the challenge is simply to produce a learner capable of learning it. The power-law structure then basically becomes the upper bound on the learner. When all is well and we have removed the core bottlenecks to scale, we see this scaling structure. When we are bottlenecked by something else such as by dataset or by model architecture or optimizer or numerics etc, we see scaling fail. In the longer term, then, we should expect scaling to end either when we somehow become unable to sustain the capacity and SNR of the model to keep learning the underlying structure, or when we have somehow exhausted the underlying structure that there is to learn. In certain cases, we can directly see this structure then being transferred into the weights, although this depends heavily on the parametrization of the neural network. But, for instance, this occurs in a very literal way in deep linear models https://arxiv.org/abs/1312.6120 . From here, it is simple to construct Chinchilla-style scaling laws. We simply assume that linear increases in parameters mean we can learn linearly more modes, and that a mode can be learnt in a roughly constant amount of samples to a given level of fidelity. With these assumptions, we then get the classic logarithmic scaling of the scaling laws in log-loss since for each increment of surprise there are exponentially many more modes to learn. Given a fixed amount of compute which is linear in both parameters and data, we can then compute the optimal data-compute ratio which depends on things like the SNR of the mode learning process in terms of samples and the ‘conversion rate’ between parameters and learnt modes. Stepping back further, there is also an interesting anthropic argument here, that this kind of power-law ‘infinitely’ resolvable, but still compressible feature hierarchy is the only regime in which general scalable learning algorithms make sense. If our predictive information ratio in depth is substantially below 1, then almost all predictive information accrues to the first few depths. Essentially, a few coarse features control almost all of the lower-level data distribution. The world, despite apparent complexity, is actually very simple. This is basically our failure mode of the naive Aitchison-style aggregation. A super obvious case of such a distribution is a bunch of bernoulli coinflips. While at the surface level this data may seem complex – e.g. HHTHTTTHTHTTTHTHHTHTH – in practice, all the predictive information in this distribution is contained in a single scalar which is the bias of the coin. In such a world, there is little point using complex and scalable learning algorithms. Once you have some simple learner that can uncover the first few scalars, you are done since the world is simple. Conversely, if our predictive information vs depth ratio is greater than one, then the amount of predictive information at a level of the hierarchy explodes exponentially in depth. Almost all ‘information’ is pushed into the finest lower-level features far beyond the ‘resolution’ that our learner can practically reach. This means, in effect, that the world appears to our learning algorithm as irreducible noise and no learner can make progress until it ends up with more computational power than ‘the world’. Only close to the critical ratio of 1, and slightly on the compressive side, is the system stable enough to give a meaningful learning signal across scales for a general scalable learner to exploit. Only this kind of world is ever likely to produce general intelligences such as ourselves and deep learning models. This also comes back to our discussion of ‘data progress’ vs ‘algorithmic progress’. The key fact here is the obvious one that we can control the data distribution . The data isn’t handed to us immutably from on high. If we have some specific set of capabilities that we care about, we can design the data distribution to upweight the features roughly corresponding to the capabilities that we care about, and if we can increase their feature rank this has a multiplicative improvement in ‘capability’ vs data and compute size. I.e., very schematically, suppose we really care about the model learning advanced math. In a normal common-crawl dataset, the ‘advanced math’ features start occurring at about rank 10^6, so they are certainly present if you learn enough but still fairly rare, since after all most common-crawl webpages are not about advanced math. Suppose then that we filter the dataset and generate synthetic math data such that these same features now appear at about rank 10^5 or even 10^4 – this gives us a 10-100x improvement to the model according to the standard scaling-laws analysis – i.e. to achieve the same math benchmark capabilities we require 10-100x less resources, split in some proportion between parameters and data. Whether this is ‘algorithmic progress’ or ‘data progress’ then becomes a question of semantics. The model improves principally because the data became dense in the features we care about, so the model no longer learns so much ‘irrelevant’ stuff. At the same time, the ability to filter large naturalistic corpora to upweight the data we care about, and generate synthetic data of sufficient quality etc, is itself algorithmic in some sense. What this really tells us is that ‘algorithmic progress’ is much too broad a label, and that we should, at minimum, separate our algorithms that principally improve dataset creation and algorithms that improve the core architecture, optimizer etc. On the data algorithms side we can think of this almost like refining. Reality has given us this immensely detailed world with a huge amount of totally irrelevant information, of which we only want a few small but very valuable nuggets. What we want to do, then, is to figure out how to effectively extract and collect all of these nuggets together to produce a much ‘higher quality’ dataset for our training, and we need to do this in a way which is not too expensive to make it tractable. More broadly, this is really a question of where the training signal comes from. In the early days of scaling, basically all of the signal was already immanent in common-crawl and other online data and the challenge was simply to filter and extract it. Then, as we search for increasingly rare features that are barely or almost never represented in existing datasets such as solving novel proofs, long horizon SWE and agentic coding, converting the world to paperclips etc, we have to be clever about how to continue to produce training signal that we can distill into a dataset. These training signals can come either from humans directly, which is what data companies now sell; from other models which have learnt from these signals before you, which is distillation; or from models open-endedly exploring tasks and occasionally succeeding. In practice, a combination of these is used conjointly and ends up being highly effective. Distillation and human data can ‘warm-start’ your model from nothing to close to the frontier, at which point you need to switch over to model-generated rollouts to push out the frontier a little further than before for this round of models, and then all progress in this round can be distilled into the next set of models so they start at a higher point. This explains why we have progressively moved from pretraining on common crawl to midtraining on synthetic and human-generated reasoning and other highly targeted data distributions, to more open-ended RL. - There is an interesting mathematical wrinkle here. A ratio of exactly one gives the exact Zipf law but is actually non-summable and hence the series diverges. Strictly speaking, you need a ratio extremely close to, but slightly less than 1 to ensure that you have finite predictive information even with a theoretically infinitely deep tree. ↩ fnref:1 - Very importantly, this is not the probability of sampling a specific rare word, which is of course rare. Rather, it is the probability of sampling any rare word vs any common word. ↩ fnref:2 - This is similar to the ‘resolution-view’ of Bahri et al https://www.pnas.org/doi/10.1073/pnas.2311878121 ↩ fnref:3