Architecture Research as Addressing Constraints to Scaling

Architecture research aims to remove bottlenecks to scaling rather than merely tweaking loss, according to a post by Beren on beren.io. The transformer's attention mechanism and residual stream enabled scaling in sequence length and depth, respectively, which was previously impossible with RNNs and LSTMs due to vanishing gradients and sequential processing. This perspective reframes the goal of architecture innovation as extending the scaling regime.

Author’s note : Short note. Really a quick extension to scaling is subtler than it seems. This is probably obvious to many but still worth writing, I think/hope. Sometimes when I talk to people about architecture research, they have a somewhat naive impression that it is mostly about empirically searching to find tweaks that lower loss/perplexity a small amount. E.g. we try adding a layernorm here or there or changing this activation function or adding a convolution layer or whatever, and just trying to improve loss. This is often coupled with the idea that LLM architecture research is basically an entirely empirical ‘throwing darts at the wall’ kind of research. Certainly there is some of that 1. However, here I want to propose a different perspective on architectures. Namely, that the point of architecture research is not principally to improve pretraining loss under some fixed iso-param and iso-flop conditions but rather to remove, or at least push back, the bottlenecks to further scaling . In our previous post, scaling is subtler than it seems https://www.beren.io/2026-08-15-Scaling-is-Subtler-than-it-Seems/ , the core argument is that scaling is not so much an idea, or a primitive operation, but rather a regime . We get clean and predictable scaling laws when we have removed all bottlenecks that would prevent further scaling . Once all bottlenecks are removed, then the generality of neural networks provides us a direct and predictable way to convert compute into amortized compressions of increasingly rare features of the training dataset. The hard part, of course – and the reason why figuring out ‘scaling’ took so long historically – is that removing enough bottlenecks to enter this ‘scaling regime’ is hard. You can have messed up hyperparameters, or not enough or too simplistic data. You can just have a poor optimizer, terrible numerical dynamics, or just buggy training code. And finally, of course, you can have a bad architecture which is bottlenecked by something fundamental before it can enter this scaling regime at all. The most classic case of this is, of course, the invention of attention and then the transformer. Before attention, we had various recurrent models such as RNNs and LSTMs. These do not scale well, both in depth but also, crucially, in time . For these sequence models they have to be run sequentially forwards in time to compute the outputs per timestep then backwards in time to compute the gradients. This is obviously very bad and slow from a GPU-compute perspective, but even if you had the GPUs to burn, there is a more fundamental problem. This backward gradient process suffers from fundamental vanishing gradients across time . This effectively imposes a fairly strict bottleneck on the length of the sequences these architectures could model, preventing a large degree of the features in natural language datasets from being effectively expressable by the model. So attention removes two bottlenecks at the same time; first the fixed-size recurrent state and secondly the issues with signal propagation forwards and backwards through time. Then the transformer essentially just couples self-attention to a residual stream and some MLPs. The residual stream is crucial here since it allows effective signal propagation through, and thus scaling in, depth. So attention enables scaling in sequence length. Residuals enables scaling in depth. This obviously is a very potent combination to the extent that, before the transformer, it was extremely difficult to meaningfully scale at all, which is why, I think, scaling laws and the idea of scaling only became widely understood a few years after the invention of the transformer. I think a lot of other core architectural innovations are similar in that they remove or ameliorate key bottlenecks to the transformer and thus enable extending the scaling regime much further than before. This, to me, is the core aim of architecture research rather than producing small improvements on loss although often by removing a bottleneck we do see improvements in loss even at the small scale 2 . From this perspective, we can understand a lot of core architectural components with the bottleneck that they remove. For instance: - The is fundamental to all modern architectures and solves a very deep and vital bottleneck: how to prevent activation and gradient vanishing or exploding. If we just naively stack layers like we did until 2015, then your function mapping is essentially the composition of the weight matrices/jacobians. If the top singular values of each element is greater than 1 then the composition tends towards exploding exponentially, and if less than 1 your activations and gradients vanish with depth and you effectively only end up learning a shallow model even if theoretically deep. Residual streams, since they compose layers additively rather than multiplicatively stave off this problem until a much larger depth, which pushes back the bottleneck far enough to allow for a good number of OOMs of scaling before becoming a major problem again. residual stream https://arxiv.org/abs/1512.03385 - : are attention variants that GQA https://arxiv.org/abs/2305.13245 , MLA https://arxiv.org/abs/2405.04434 , etc https://arxiv.org/abs/2510.04476 reduce the KV cache size . This is an extremely important bottleneck because for every token that is generated, self-attention requires the query of the new token to be compared with the keys and values of all prior tokens in the KV cache. This is a massive amount of memory bandwidth that must be utilized for every single decoded token. This becomes an extremely harsh constraint since modern GPUs have vastly greater compute available than the memory bandwidth needed to saturate the compute. Without reducing the KV cache size, we would be effectively saturated on memory bandwidth even for moderately long contexts without innovations such as GQA which, crucially, reduce the number of KV heads compared to query heads and thus reduce by a large constant factor the amount of memory bandwidth needed for decode compared to full multi-head attention. Historically, GQA and related variants let us go from context lengths of thousands to hundreds of thousands without needing to fundamentally redesign our entire hardware architectures. - : These try to remove the computational bottleneck of attention which is its quadratic scaling with sequence length. Even once you have reduced memory bandwidth costs by a constant factor in decode, this only buys you a few orders of magnitude until you are more fundamentally bottlenecked by the compute costs of prefilling massive sequences and the increasing cost of decode. You ultimately cannot resolve a quadratic scaling issue with constant time improvements, although empirically these do buy you a surprising amount. SSMs try to replace the attention operation entirely with essentially a RNN-like operation over a fixed-size state. The mathematical trick here is writing an RNN using only associative operations so that the sequential recurrence can be efficiently parallelized using scans during prefill otherwise we are stuck with the same issues that bottlenecked RNNs originally. The core downside here is obvious though, the fixed-size state. Attention is effectively a non-parametric memory which linearly expands its state size as the sequence length increases. This is why it is quadratic in complexity to begin with. You cannot fully model a linearly growing state with a fixed state unless all the additional information in the sequence is redundant which, clearly, it is not. Thus SSMs are powerful for short sequences yet cannot ultimately match attention indefinitely. And this is where they have ended up empirically. They are very useful as a complement and addition to regular attention but cannot serve as a complete replacement for attention for long-context reasoning, and in practice we simply interleave them with attention along the residual stream which again provides a constant factor improvement over attention but still ultimately retains the quadratic bottleneck. Nevertheless, it also pushes out the usefully achievable sequence lengths a decent amount. SSMs https://arxiv.org/abs/2312.00752 and sparse attention https://arxiv.org/abs/2512.02556 More recently there has been success with sparse attention https://arxiv.org/abs/2512.02556 variants https://arxiv.org/abs/2606.13392 . The idea here is to basically apply sparsity to the kv cache directly. You still keep the full kv cache, but for each token to decode you do not necessarily read the full kv-cache as is done with standard attention. Instead, you select a sparse subset and then read only those tokens. In theory this allows the best of both worlds. You can maintain the non-parametric linearly-increasing state size of full attention, but you remove the core bottleneck which is simply reading from that state during decode, since you can only read from a subset of fixed length. The challenge, of course, is how to select what sparse subset to read from in a way that is both sufficiently expressive to perform well but also not so expensive that you have basically replicated the original cost of attention. The current best methods such as DSA https://arxiv.org/abs/2512.02556 effectively use a much smaller attention block, called an indexer, which often has only one or a few heads, to select the tokens to be read from. Then the full attention is applied to only these tokens. This is effective in that the indexer does not necessarily have to be as expressive as the full attention, and hence can be much cheaper. However, currently to maintain sufficient expressivity to not tank performance, the indexer itself needs to be some variant of attention which selects a different sparse mask for each decoded token. This thus does not fully remove the quadratic scaling of attention, but rather only moves it into the indexer, which gives a constant factor speedup. Nevertheless, these sparse attentions appear highly effective and will allow us to push out the sequence length frontier another OOM or two to over 10M tokens without truly bottlenecking decode. - : This one is subtle and underappreciated but it solves a fundamental bottleneck: attention logit explosion and resultant softmax saturation. When implemented naively, the optimization and numerical dynamics of self-attention are slightly pathological. To achieve a highly concentrated softmax distribution, it requires the norms of the q and k matrices to grow so that there are larger disparities in scores between useful and useless tokens. However, the exponential in the softmax can then take even relatively small numerical differences and magnify them dramatically, eventually causing numerical overflow or underflow when using lower-precision floats. This eventually causes instability especially at scale since the pre-softmax scores are dot products of more variables which means the variance in norm grows with scale. This can eventually cause loss spikes and other instabilities from within attention during training of larger models. The fix for this is very simple and is just to normalize the qk projections while optionally letting the softmax temperature be learnt dynamically. This does not really have much impact on the pretraining loss at small scales but is vital for stable training at scale. QK norm https://arxiv.org/abs/2010.04245 - Mixture of Experts MoE : This is a very obvious case. MoEs decouple the inference cost active parameters from the total parameters of the model. This enables the model to store increasingly large amounts of information in its parameters while not bearing the cost of invoking every parameter on every forward pass. This has been vital in pushing scaling far beyond what would be possible with pure dense models. The largest dense models today are in the hundreds of billions of parameters, the largest sparse models are approximately 10T. MoEs have thus allowed an extension of the viable scaling regime for two orders of magnitude in total parameter count but crucially not active parameters . More broadly, the idea of architectural sparsity in general allows the decoupling of storage from compute – i.e. we can store a massive system but only need to directly compute over some small fraction of it. This is ideal because in hardware storage itself is very cheap relative to both compute flops and memory bandwidth. This is itself not an arbitrary thing but mostly a property of physics. Information can mostly just be stored passively but actively reading or writing from it and computing things requires energy. Thus, through sparsity, we adapt our architectures to the fundamental constraints of our physical computing substrate. 3 fn:3 - Hyperconnections : Hyperconnections address a subtle but increasingly important bottleneck: the residual stream itself . As models and sequence lengths continue to scale, all information in the model is still routed through the residual stream which is just a single N dimensional vector per token. It makes sense, therefore, that the capacity of this vector will eventually become a bottleneck on the amount of information the model can maintain between layers. The residual stream is effectively the ‘working memory’ of the model during the forward pass, since each block of layers reads from and writes to the residual stream. Moreover, the final token selection at the end of the model is simply a linear projection of the residual stream to the vocabulary size to generate the logits for each token. This immediately leads to the question of, if the residual stream itself is becoming a bottleneck, can we expand the capacity of the residual stream in a way that isn’t extremely costly in terms of compute and activation memory? One answer to this, which appears highly effective in practice, is hyperconnections, especially the mHC https://arxiv.org/abs/2512.24880 method, and later extensions https://arxiv.org/abs/2607.14530 . The idea here is to maintain several parallel residual streams at once, thus expanding the original capacity of the residual stream multiple times. However, before reaching a layer, these residual streams are combined with a learnt weighting so the larger residual stream isn’t just generically expanding the width of the model and thus needing to expand all of the input parameters of each block. Similarly, when the block writes its output, it writes it to the combined vector of residual streams and then we use a learned splitting mechanism to split this vector back into updates for each of the individual component residual streams. - : Finally there is an even more subtle issue with the residual stream that it does not truly eliminate the vanishing/exploding gradient issues we identified earlier. It merely dampens the effect considerably, allowing multiple OOMs of scaling to be traversed before these issues begin to bite again. Here, the gradient vanishing issue is manifested as an Attention residual https://arxiv.org/abs/2603.15031 and norm-agnostic residual streams https://arxiv.org/abs/2606.16112 exponentially increasing norm https://www.lesswrong.com/posts/8mizBCm3dyc432nK8/residual-stream-norms-grow-exponentially-over-the-forward within the residual stream. This occurs because the residual stream is essentially the additive sum of each layer’s contributions. If the layer updates are correlated with one another over depth, which in practice they almost always are, then adding multiple correlated vectors together results in a systematic increase in norm. If the residual stream norm is larger, then this means that for each layer’s output to have the same impact on the residual stream, then that layer’s output norm must also grow exponentially. However in practice it is not simple for the output norm of a layer to grow as fast as the residual stream norm. This is because often the layers inputs are normalized meaning that the weights themselves have to grow in norm to compensate, but then the norm growth in the weights is directly penalized with weight decay. In practice what this means is that later layers cannot match the norm of the residual stream and thus have weaker impacts on the overall residual stream representations than earlier layers, leading to effectively an asymptoting effectiveness of depth with concurrent gradient vanishing on the backwards. This then imposes a soft bottleneck on the continued scaling of model-depth. To solve this some recent methods have been proposed. Attention residuals replace the additive residual stream with an attention mechanism operating over the activation outputs of each layer. Since the softmax in attention effectively only selects a few layer outputs at each layer, this prevents the norm increase that occurs in regular residual streams and thus eliminates this potential bottleneck. The downside to this is that we have to perform attention over L layer activations which, in the naive case, requires reading L activation vectors from HBM for every new layer, which is more expensive than the regular residual stream, although it is cheap compared to standard attention decoding since the number of layers is usually drastically less than the sequence length. Norm-Agnostic NAG proposes a different approach where we keep the residual stream as normal, but instead directly control the geometry of the additions to the residual stream such that the growth of the stream is precisely controlled. This allows maintaining each layer’s ability to impact the residual stream an equal amount, if desired, at the cost of slightly reducing expressivity and adding a complex norm-control apparatus to the model. A bunch more architectural innovations can also be thought of in such a way. The interesting question, then, becomes trying to understand what the current bottlenecks to continuing to scale is and then designing new architectural primitives that can either ameliorate or push back these bottlenecks, and if this cannot be done cleanly, at least understand the pareto frontier of options. For instance, the hard quadratic bottleneck of attention seems fundamental and we have not yet figured out a way to just completely remove this with no downsides. Instead, we seem to be sketching out a pareto frontier between state-size and expressivity where progress looks like finding a new or better point on this frontier than before. Of course, when we say ‘scaling’ we are really talking about scaling a whole bunch of different factors simultaneously such as depth, width, sequence length, flops, memory bandwidth etc simultaneously. The interesting part is then figuring out which part of the system is likely to become the most important bottleneck as we continue to scale and then figuring out how to push that bottleneck out further until another one becomes more pressing. Another interesting but more abstract way to think about this is that our computational substrate has essentially three key resources: compute, memory, and memory bandwidth. The cost of these is different and depends on the hardware but generally our hierarchy of expense is memory bandwidth compute memory 4. As we scale, we tend to stress different bottlenecks to network function at the same time – i.e. first we might be limited by basic stuff like signal propagation and gradient vanishing. Then, if we solve that, we get limited by sequence length and memory bandwidth. Then, if we push these problems out we get stuck by subtler issues of residual stream capacity, and so on. The challenge of architecture design then is essentially either finding clever ways to circumvent or improve the latest bottleneck, or else finding superior conversion ratios of the underlying hardware constraints which are favourable – i.e. if we can swap memory for memory bandwidth and compute, as sparsity does, or if we can trade memory bandwidth for compute, like diffusion, then these are good trades to make. - Famously, the invention of swiglu is due to divine benevolence https://arxiv.org/abs/2002.05202 . ↩ fnref:1 - My intuition here is that this is because bottlenecks are not really hard walls you are fine until you run into. Rather, they are continuous inefficiencies which start out small and noticeable, but manageable, but then compound with scale into increasingly large hits such that at large enough scale you essentially cannot progress at all. ↩ fnref:2 - Note the brain does this https://www.beren.io/2023-04-09-GPUs-vs-brains-hardware-and-architecture/ at an even more fine-grained level, where individual neuronal firing and thus synaptic potentiation is extremely sparse across the entire brain. The brain has many many trillions of synapses, but at any given moment only a tiny fraction of them are firing, and this is necessary energetically, since the metabolic cost of keeping the entire brain active continually would be insurmountable. ↩ fnref:3 - Note that this is specific to current Nvidia-like GPUs. SRAM chips like Groq and Cerebras have essentially an inverted cost hierarchy of memory compute memory bandwidth and thus architectures designed for these face very different trade-offs. ↩ fnref:4