# Scaling Subquadratic Attention to Trillions of Parameters

> Source: <https://www.zyphra.com/our-work/scaling-zha>
> Published: 2026-11-17 12:00:00+00:00

# Scaling Subquadratic Attention to Trillions of Parameters

We recently introduced Zyphra Hierarchical Attention (ZHA; see [paper](https://arxiv.org/abs/2610.42112) and [blog](https://zyphra.com/our-work/zyphra-hierarchical-attention)), a subquadratic attention method that uses a tree of compressed KV-cache summaries to efficiently retrieve relevant information from arbitrarily long contexts. We demonstrated that ZHA could be applied to existing pretrained language models through an unusually inexpensive distillation procedure, preserving performance on challenging reasoning, retrieval, and agentic benchmarks. In contrast to other sparse attention methods like DSA, MSA, or QSA that produce a constant-factor speedup, our ZHA adds tree depth (with demonstrated performance retention up to 6 levels) as sequence length increases, which in combination with our top-p selection mechanism enables truly subquadratic computational scaling of attention. However, our initial experiments were limited to models in the 30B parameter range.

Here, we scale ZHA to some of the largest and most capable open-weight models available, including Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, and MiMo-V2.6-Pro. We find that ZHA conversion continues to work remarkably well at scale, generally preserving model capabilities with little or no degradation, including on challenging long-context evaluations (e.g., we observe less than 2% degradation on 1M RULER for Qwen).

However, the computational scaling of ZHA depends critically on the underlying structure of the model's attention distributions. We find consistently clean sublinear power-law scaling in Qwen, while other architectures exhibit more complicated behavior, including approximately uniform attention distributions in the first few layers and anomalous long-context scaling. These results suggest that the principal obstacle to truly subquadratic attention at frontier scale may not be the sparse attention algorithm itself, but the ability to train sufficiently well-structured attention distributions in the underlying model backbone.

## Introduction

Two weeks ago, we introduced Zyphra Hierarchical Attention (ZHA), a new approach to sparse attention that can overcome the quadratic scaling bottleneck of standard transformers.

The central idea behind ZHA is simple: Instead of searching the entire KV cache to determine which tokens are relevant to a particular query, we construct a hierarchical tree of compressed summaries of the cache. When performing attention, we traverse this tree from the top down, first identifying relevant large regions of the context, then progressively narrowing our search until we reach the original keys and values.

Because we can discard irrelevant regions without examining every token they contain, the work required to search the context itself can grow sublinearly with sequence length. This is what distinguishes ZHA from sparse attention methods such as [DSA](https://arxiv.org/abs/2512.02556) or [MSA](https://arxiv.org/abs/2606.13392), whose indexers still need to scan the full context.

An especially useful property of ZHA is that we do not need to train a new language model from scratch. Instead, we can convert an existing pretrained model by freezing its backbone and training a very small set of reference queries to reproduce the attention distributions of the original model. In our original paper, we demonstrated that this procedure can preserve performance on difficult mathematical reasoning, long-context retrieval, and agentic coding benchmarks, even when attending to only a small fraction of the KV cache.

However, these experiments were restricted to relatively modest models, Qwen3.6-35B-A3B and Nemotron-3.5-Lightning-30B-A3B. A natural question is whether the same approach continues to work for much larger, frontier-scale models. In particular, does the conversion process become more difficult as model size increases? Does maintaining performance require progressively more distillation? And, perhaps most importantly, do the attention distributions of these larger models actually support the subquadratic scaling that ZHA is designed to exploit?

To answer these questions, we converted several frontier open-weight models with substantially different architectures and training histories.

## The Importance of Top-p Scaling

Before discussing the individual models, it is useful to briefly revisit an important idea from our original paper: the relationship between attention sparsity and the statistical structure of long sequences.

ZHA uses a *top-p* selection mechanism to determine which regions of the context to explore. For a given attention distribution, the top-p set is the smallest set of tokens whose cumulative attention probability reaches some threshold p.

For example, if a small number of tokens account for 99% of the attention probability, then the top-0.99 set will be small. If attention is distributed nearly uniformly across the context, recovering 99% of its probability requires retaining approximately 99% of the tokens. Therefore, unlike methods that select a fixed number of tokens, top-p allows the amount of retrieval to adapt to the attention distribution of each query.

This distinction becomes especially important when considering very long contexts. Suppose a sequence has length , and the number of tokens required to capture a fixed fraction of its attention probability grows approximately as , where . In this regime, the amount of information actively retrieved by attention continues to increase as the context grows, but more slowly than the context itself. This is precisely the kind of scaling we want: the model can access an increasing amount of information without requiring a fixed or increasing *fraction* of the entire sequence.

By contrast, if attention becomes approximately uniform, the top-p set grows linearly with . To preserve a given amount of attention probability, we must retrieve a constant fraction of the entire context, eliminating much of the opportunity for subquadratic scaling.

In our previous blog, we described this as the **narrow path to truly ultra-long contexts**. On one side, we cannot compress the context into a fixed-size representation without eventually losing useful information. On the other, we cannot afford to retrieve an approximately linear fraction of an arbitrarily long context for every query.

Our hypothesis is that the amount of predictive information contained in a natural sequence should increase sublinearly with its length. The top-p scaling curves of a model provide an empirical indication of how successfully its attention distributions capture this structure. They are not a direct information-theoretic measurement, but they give us a useful way to study the relationship between long-context modeling and computational efficiency.

An important implication is that **subquadratic attention is not purely an architectural property**. Even with a theoretically subquadratic search algorithm, its realized efficiency depends on the structure of the attention distributions being searched.

As we will see, different frontier models exhibit surprisingly different behavior in this respect.

## Scaling ZHA to Frontier Models

We converted four substantially different models: Qwen3.8-2.4T-A95B, Nemotron-3.5-Super, GPT-OSS-120B, and MiMo-V2.6-Pro.

Together, these models let us study ZHA across different parameter scales, positional encoding schemes, and attention architectures, including RoPE and partial-RoPE models, hybrid NoPE architectures, and models mixing local and global attention.

Across these experiments, the conversion procedure itself works extremely well. Most evaluations remain very close to their original dense-attention baselines, including benchmarks such as SWE-bench, LongBench v2, and RULER that require retrieving and using information across extended contexts.

The differences become much more pronounced when we look at the structure of the underlying attention distributions and the computational scaling they permit.

### Qwen3.8-2.4T-A95B: Clean Scaling at Trillion-Parameter Scale

We begin with Qwen3.8-2.4T-A95B, the largest model in our experiments, with 2.4 trillion total parameters and approximately 95 billion active parameters. This model provides a particularly strong test of whether ZHA conversion can scale beyond the relatively small models considered in our original work.

The results are remarkably good.

On challenging long-context retrieval, mathematics, and agentic coding evaluations, the differences between the full attention and ZHA-converted model are essentially at the noise floor. Even at the longest 1M context lengths, RULER scores only drop by under two points, which is within the noise distribution.

This is a particularly encouraging result. We can replace the attention mechanism of a model with trillions of parameters, while leaving the pretrained backbone completely frozen, and retain almost all of its capabilities even on evaluations that require substantial long-range retrieval.

The more interesting result, however, comes from examining its attention distributions.

Across a wide range of p values, the size of the relevant attention set increases approximately as a sublinear power law with context length. The curves remain smooth across several orders of magnitude, without obvious discontinuities or changes in scaling behavior. We observe similar behavior throughout the Qwen model family, including across different model sizes and attention layers.

This is exactly the kind of structure that makes ZHA effective.

When the top-p set scales sublinearly, progressively refining our search through the hierarchy allows us to retrieve the relevant information without examining a linear fraction of the original sequence. Correspondingly, our modeled attention costs show increasingly favorable scaling relative to dense attention as the context grows.

More broadly, we suspect that this unusually clean power-law behavior may be a key signature of a well-trained long-context attention mechanism. A well-trained model should be able to distinguish relevant information from the growing amount of irrelevant context, rather than allowing attention probability to diffuse indiscriminately across more tokens. Smooth sublinear top-p scaling is consistent with this behavior.

We do not yet know whether such scaling is universal or precisely which aspects of Qwen's training are responsible for it. Nevertheless, it is an encouraging example of the kind of attention structure that makes efficient ultra-long-context modeling possible.

### Nemotron and MiMo: Early-Layer Exceptions to Subquadratic Scaling

While Qwen exhibits remarkably clean top-p scaling throughout its attention layers, not every model behaves quite so nicely.

Two particularly interesting examples are Nemotron-3.5-Super and MiMo-V2.6-Pro. Both models convert well to ZHA, preserving essentially all their original capabilities, and both exhibit favorable subquadratic attention scaling in most of their layers.

However, they also share a curious pattern: the earliest attention layers have substantially less favorable top-p scaling than the rest of the model. In Nemotron, this occurs in the first attention layer. In MiMo, it occurs in the first two. In both cases, these layers fall off the narrow path to subquadratic scaling due to near-linear scaling of top-p.

### Nemotron-3.5-Super-120B-12A

Nemotron-3.5-Super is a frontier hybrid Mamba-Transformer model that uses attention without explicit positional encoding (NoPE).

NoPE architectures are interesting because they demonstrate that models can recover useful positional information without explicitly applying RoPE or another positional encoding to their queries and keys.

In our earlier work on [positional information in hybrid NoPE models](https://zyphra.com/our-work/learning-position-in-hybrid-nope-models), we showed how local mixing mechanisms can introduce implicit positional information into the residual stream, which can then be recovered by subsequent global attention layers.

This makes Nemotron a useful test of ZHA's ability to generalize beyond the RoPE-based models on which most of our initial experiments were performed.

Once again, the conversion results are very strong. Overall, the quality of the converted model remains extremely close to the original.

However, its attention distributions reveal a curious anomaly.

Most of Nemotron's attention layers show the kind of graceful subquadratic scaling that we would hope for. However, the first full-attention layer behaves quite differently.

Its attention logits are relatively uniform over the sequence, with substantially less of the concentration observed in the later layers. Consequently, its top-p set grows nearly linearly with sequence length, and the potential computational savings from ZHA are significantly reduced.

One possibility is that the first global attention layer has not learned as effective a positional or relevance structure as subsequent layers. In a hybrid NoPE model, where useful positional information must emerge implicitly from preceding computation, this may be particularly sensitive to the arrangement of attention and local mixing layers.

Another possibility is that this first attention layer is simply not making especially useful content-dependent retrieval decisions, and therefore has relatively weakly differentiated attention scores.

We do not yet know which explanation is correct, but the fact that all subsequent layers exhibit substantially more favorable scaling suggests that this is a relatively localized phenomenon rather than a fundamental limitation of the model's attention architecture.

### MiMo-V2.6-Pro

We find a surprisingly similar pattern in MiMo-V2.6-Pro.

MiMo uses partial RoPE (as does Qwen above), applying rotary positional embeddings to only part of the query and key representations rather than to their full dimensionality. This makes it another interesting case for studying how positional encoding choices interact with the structure of learned attention distributions.

As with Nemotron, the ZHA conversion matches the original model's performance. There are minor fluctuations in evaluation scores but by and large these are within noise and are at most a point or two. Some evaluations such as LongBench-v2 even improve.

The more interesting result comes from the attention scaling. Similarly to Nemotron, MiMo exhibits nearly linear top-p scaling in its earliest attention layers. In this case, however, the effect is present in the first two attention layers rather than just the first.

The remaining layers exhibit much more favorable sublinear top-p scaling, showing that the model is capable of learning well-structured, concentrated attention distributions across the majority of its depth. Nevertheless, these first two layers have a substantial effect on the overall computational savings available from ZHA.

In aggregate across all 10 global attention layers, our modeled latency predicts a crossover point of approximately 89K tokens, beyond which ZHA becomes more efficient than dense attention. However, the total speedup at 10 million tokens is only 3.9×, which though substantial is considerably smaller than the gains available when all layers exhibit favorable top-p scaling.

This is not because the model's attention distributions are uniformly diffuse throughout the network. Quite the opposite: most of its attention layers have relatively good top-p structure. Rather, a small number of poorly scaling layers can disproportionately affect the total attention cost, limiting the overall speedup even when the majority of the model behaves well.

### What Is Happening in the First Few Layers?

The similarity between Nemotron and MiMo is particularly interesting because their positional encoding schemes are not identical. Nemotron uses hybrid-NoPE, while MiMo uses partial RoPE. Yet both exhibit nearly linear top-p scaling concentrated in the earliest attention layers, followed by substantially better scaling deeper in the network.

We do not yet understand the precise cause of this phenomenon.

One possibility is that the earliest attention layers are performing a somewhat different function from the later ones. Rather than selectively retrieving a small set of relevant tokens, these layers may be involved in broader information mixing or in establishing positional and contextual representations that subsequent layers can use for more selective retrieval.

Another possibility is that the positional representations available to early attention layers are simply less well developed. This would be consistent with our previous observations about implicit positional information in NoPE architectures. However, it is not clear that the same mechanism explains MiMo, given that Qwen exhibits consistently favorable scaling while also using partial RoPE.

It is also possible that these layers are simply less well trained or less effectively utilized than later attention layers, resulting in attention distributions that are less selective than they could be.

Distinguishing between these possibilities would require further investigation.

What is particularly encouraging, however, is that the problem is highly localized. In both models, the later attention layers exhibit the favorable scaling properties that we want from a subquadratic attention mechanism.

This suggests that relatively targeted interventions, such as finetuning the earliest attention layers to improve their top-p structure, could potentially recover much of the computational efficiency that is currently lost, without requiring any substantial modification to the rest of the model.

### GPT-OSS-120B: The Complications of Long-Context Extension

Finally, we converted GPT-OSS-120B, an important open-weight model with broad relevance to practical and enterprise deployments.

GPT-OSS is an interesting contrast to Nemotron and MiMo. While those models show relatively localized problems in their first few attention layers, GPT-OSS exhibits a different anomaly associated with the behavior of its attention distributions at long context lengths.

Our conversion results are generally good, although performance degradation is more noticeable compared to Qwen, Nemotron, or MiMo.

While most evaluations stay constant, SWE-bench performance declines somewhat more noticeably, from 59.6 to 55.8, and the largest degradation occurs on RULER at 128K tokens, where performance falls from 63.3 to 46.0.

While this is a meaningful loss, it is also worth noting that the dense baseline performs relatively poorly on this long-context retrieval evaluation compared with the other models we study.

Examining the attention distributions gives us a potential clue as to why.

Unlike Qwen, GPT-OSS does not exhibit uniformly clean power-law top-p scaling across its full context range. Instead, we observe an unusual flattening of the curves toward the end of the context window, suggesting a change in the structure of the attention distribution at long sequence lengths.

One possible explanation is the model's context-extension procedure.

GPT-OSS uses YaRN to extend its RoPE-based attention to a 131K-token context window. It is plausible that some combination of positional extrapolation, context-extension training, or insufficient exposure to long contexts produces the anomalous attention behavior we observe.

However, we cannot establish this from the available evidence. Although the architecture and use of YaRN are publicly documented, the full details required to identify the cause are not available.

In part because of these complications and the resulting abnormal attention sparsity at long contexts, ZHA in fact provides especially significant modeled computational savings.

Our latency model predicts a crossover point of approximately 40.1K tokens, with ZHA becoming progressively more efficient as context length increases. At 10 million tokens, the predicted attention speedup is approximately 58.2× relative to dense attention.

These results highlight the difference between two separate questions: whether a model's attention can be approximated efficiently, and whether the resulting sparse model preserves its ability to use very long contexts effectively.

For GPT-OSS, we obtain substantial computational benefits, but the unusual long-context attention structure appears to make faithful conversion more difficult, particularly on retrieval-intensive evaluations. Whether due to the underlying dense attention patterns, learning deficits in our router, or both, GPT-OSS consequently falls off the narrow path in the other direction compared to Nemotron and MiMo: retrieval does not scale sufficiently with sequence length.

This is somewhat different from the behavior observed in Nemotron and MiMo, where the main issue is a relatively small number of poorly scaling early layers, while the overall model retains its capabilities extremely well.

Together, these examples demonstrate that the structure of a model's attention distributions can affect both the efficiency and the fidelity of ZHA conversion, but not always in the same way.

## ZHA Conversion Remains Extremely Cheap at Scale

One of the most encouraging results from these experiments is how little additional training is required to convert even the largest models.

Our distillation procedure operates locally on individual attention layers. Rather than updating the model's backbone to reproduce its original outputs through a large-scale end-to-end training process, we independently train the reference queries used to construct and search the hierarchical summaries. The objective is simply to match the attention probability mass assigned to different chunks of the context, which is a much smaller learning problem than conventional knowledge distillation.

Crucially, we do not train the backbone at all. Its original weights remain frozen, meaning that we do not need to maintain optimizer states or gradients for trillions of pretrained parameters. The additional trainable state consists only of the reference queries, which are negligible in size compared with the model itself.

In practice, the distillation procedure is so inexpensive that **running the full evaluation suite took substantially longer than the actual conversion training**, even for our largest models.

This is a rather remarkable property for an architectural modification that fundamentally changes how attention accesses long contexts.

It suggests that ZHA conversion need not be a major new training stage. Instead, it can potentially be treated as a lightweight adaptation procedure applied to an already capable pretrained model, without requiring a costly retraining process or extensive changes to its underlying capabilities.

Perhaps surprisingly, we have not found model size itself to be a major obstacle. Indeed, the largest model in our experiments, Qwen3.8-2.4T-A95B, exhibits the cleanest attention scaling and strongest performance preservation with ZHA conversion.

The harder problem lies elsewhere.

## The Real Bottleneck May Be the Backbone

Returning to the narrow-path argument from our original paper, these experiments suggest that the most important obstacle to scaling efficient attention may not be the attention algorithm itself.

ZHA provides a mechanism for finding relevant information without examining every token in a sequence. It can do so while preserving most of the behavior of extremely large pretrained models, and the distillation procedure remains inexpensive even at trillion-parameter scale.

But none of this guarantees that the underlying model has learned attention distributions that are well suited to sublinear retrieval. For this, the model must already have learned to allocate its attention probability in a meaningful way as context length grows. Our results suggest that this property is highly sensitive to how the backbone is trained.

Qwen exhibits remarkably clean power-law scaling across long contexts. Nemotron and MiMo mostly exhibit similarly favorable behavior, but with nearly linear scaling in their earliest attention layers. GPT-OSS shows anomalous top-p behavior toward the end of its context window.

Why do these differences emerge? We suspect that several aspects of training may be important.

First, **context extension** may play a particularly significant role. A model that performs well at shorter sequence lengths does not necessarily learn the correct statistical structure for much longer contexts. Extending positional encodings or performing limited long-context finetuning might produce superficially capable long-context behavior without recovering the same attention structure as a model trained more thoroughly at those lengths.

Second, **undertraining at long contexts** may be especially relevant for frontier-scale models. Training extremely large models on very long sequences is expensive, and the number of tokens or optimization steps available at the longest context lengths may be limited. As a result, some models may have insufficient opportunity to learn stable, well-calibrated long-range attention distributions.

Third, **the choice and training of positional representations** may affect the structure of attention, particularly in the earliest layers. The similarities between Nemotron and MiMo suggest that it would be interesting to understand whether NoPE and partial-RoPE architectures systematically exhibit different top-p behavior from full-RoPE models, or whether this is simply a consequence of other training decisions. However, the different top-p behavior observed in Qwen3.8 2.4T, which also uses partial RoPE, suggests that positional encoding alone does not explain these patterns.

Finally, **training instabilities and attention-logit calibration, especially at frontier-scale models** may be important. Larger models are known to be less stable and to suffer more strongly from exploding attention logit norm or other attention-based pathologies if not correctly handled via architecture or hyperparameter selection. The pathologies we are observing for these larger models may thus be consequences of scale itself.

We do not yet know which of these explanations dominates for any particular model.

It is also possible that some differences reflect genuine architectural choices rather than failures of training. For example, a layer whose purpose is broad information mixing may naturally have a more diffuse attention distribution than one specialized for selective retrieval.

Nevertheless, the variation we observe suggests that the top-p scaling structure of pretrained models deserves considerably more attention than it has received so far.

If we want to build models that use contexts of tens or hundreds of millions of tokens, it may not be enough to simply design increasingly efficient attention mechanisms. We may also need to explicitly encourage the underlying models to learn attention distributions whose relevant support scales sublinearly with context length.

## Looking Forward

Overall, these experiments give us considerable confidence that ZHA can scale to frontier language models.

We successfully converted models ranging from tens of billions to trillions of parameters, generally preserving the original model's capabilities across a wide range of evaluations. Even challenging long-context reasoning and agentic coding tasks remain remarkably robust to the change in attention mechanism. The conversion itself remains exceptionally inexpensive, requiring only a small amount of local distillation rather than any significant retraining of the underlying model. Perhaps most importantly, we have not encountered an intrinsic difficulty in scaling the ZHA conversion algorithm with model size.

The main remaining challenge appears to be the structure of the attention distributions learned by the backbone. This suggests two especially promising directions for future work.

The first is improving these distributions directly. Rather than treating the pretrained backbone as entirely fixed, we could finetune its attention components to encourage more favorable top-p scaling, particularly in layers where the original attention exhibits near linear scaling or behaves anomalously at long contexts.

For Nemotron and MiMo, this could mean targeting only the earliest attention layers, which appear to account for a disproportionate amount of the remaining computational cost. For models like GPT-OSS, improving the underlying long-context attention structure may require more careful context-extension training.

The second direction is continued optimization of ZHA itself, including more efficient tree construction, routing, and sparse attention kernels, as well as dynamically selecting tree geometries and routing thresholds as context length increases.

Ultimately, our goal is to develop an attention mechanism that can continue scaling proportionally to the amount of useful information in a sequence, without either compressing everything into a fixed-size state or paying the quadratic cost of examining every token.

The results here suggest that ZHA provides a viable foundation for this at even the largest model scales.
