Since GPT, nearly every Transformer repeats the same attention mechanism at every layer. Forty-eight
identical blocks, differing only in learned weights.
Nobody tested that. It is a convention, not a conclusion.
A paper out of VIDRAFT AI Research (arXiv:2609.20269, CC BY 4.0)
tests it, and the interesting part is not the headline. The headline is placement is free, composition is not. The interesting part is what happens when you read the ablation table.
If you build a stack with several different sequence mixers and it beats a uniform stack, you
cannot say why. Was it which mechanisms you used, or where you put them? Those two are
tangled in every heterogeneous architecture paper, because any concrete stack is one particular
arrangement of one particular set.
The paper's move is to remove the second variable by construction rather than by search.
A Latin square is an N×N grid where each symbol appears exactly once in every row and every
column. Sudoku is a Latin square with extra constraints. Put seven mechanisms on a 7×7 square and
read it out across 49 layers, and every mechanism is guaranteed to be spread evenly through depth.
layer 0..6 A B C D E F G
layer 7..13 B C D E F G A
layer 14..20 C D E F G A B
...
No mechanism can cluster. Not "did not cluster in this run" — cannot, structurally. That is the
whole point: you get balance without running an architecture search to find it.
The flagship is Aether-7B-5Attn — 6.59B parameters MoE, ~2.98B active, 49 layers, seven mixers
on a 7×7 square. Trained on 16× B200 (2-node FSDP), 162,000 steps, 144.2B tokens, ~11,700
B200-hours in the final stage.
You cannot ablate a 6.59B flagship eight ways with eight seeds each. So the paper builds a
parameter-matched proxy: four mechanisms on a 4×4 square over sixteen layers, 700.9M parameters, eight seeds per arm.
Four arms, identical parameter counts, differing only in arrangement:
| Arm | Mean CE | SD | Δ vs latin | 2·pooled SD | Verdict |
|---|---|---|---|---|---|
latin |
5.28639 | 0.00867 | — | — | reference |
periodic |
5.29484 | 0.01611 | +0.16% | 0.02588 | null (within noise) |
block |
5.31754 | 0.01520 | +0.59% | 0.02475 | real (2.5× pooled SD) |
homo_F |
— | — | +1.68% | — | real |
Read it as a gradient of how tightly each mechanism is confined in depth:
latin vs periodic) → indistinguishable. Shuffling a balanced
schedule buys you nothing.block) → you pay 0.59%. homo_F) → you pay 1.68%.
So the first practical result: stop burning GPU hours searching for the right layer order. If
your schedule is balanced and distributed, the permutation is noise. That is a real cost saving,
and it is the kind of null result that rarely gets published.
The paper then removes one mechanism at a time from the four-mechanism stack, re-matching parameters
and cycling the remaining three through depth:
| Removed | Mean CE | Δ vs latin | Verdict |
|---|---|---|---|
| sliding window | 5.28495 | −0.03% | null |
| differential | 5.28450 | −0.04% | null |
| full attention | 5.29615 | +0.18% | null |
| Mamba-2 / SSM | 5.39940 | +2.14% | real |
| (baseline) | 5.28639 | — | 8 seeds |
Three of the four mechanisms can be deleted with no measurable effect. Two of them are slightly negative — the model is nominally better without them, well inside noise.
One is load-bearing.
Look at which one. Sliding window, differential, and full attention are all members of the attention family — variations on the same operator. Mamba-2 is a
The honest reading of this table is narrower and more useful than "heterogeneity helps":
What pays is having at least one mechanism from a different family. Seven flavors of attention is still, functionally, a homogeneous stack.
That is a concrete design rule you can apply tomorrow, and it explains why the homogeneous arm
(1.68%) and the no-SSM arm (2.14%) land in the same neighborhood — from the loss's point of view,
removing the SSM is collapsing toward homogeneity.
The obvious objection: 700.9M with 1,500 steps is a pilot, not a shipped model.
The paper re-runs the three decisive arms — latin, homo_F, no_M — at 1.514B parameters, a
2.16× increase, everything else held constant, three seeds each. Both penalties get bigger:
| 700.9M | 1.514B | |
|---|---|---|
| homogeneous stack | +1.68% | +2.63% |
| remove SSM family | +2.14% | +3.20% |
Penalties that grow with scale are the good kind of evidence. A finding that shrinks as you scale is
usually a small-model artifact; one that widens is more likely structural.
Note what was not re-run: the placement axis (periodic, block) stays at 700.9M, and neither
axis was tested at N=7 or at flagship size. The paper says so in its limitations rather than leaving
you to notice.
1. A pre-registered decision rule.
The threshold is |Δ| > 2 · pooled_SD, fixed on 2026-07-22 — before the final seeds ran. Section
6.4 explains why that matters: an early two-seed run showed latin and periodic cleanly
separated, which would have been a placement effect, which is the more marketable story. Four seeds
dissolved it. Eight confirmed the dissolution, at Δ 0.008 — a third of the threshold.
The rule that killed the authors' preferred result is the same rule that certifies block and
homo_F as real. As the paper puts it: a rule that only ever confirms what you hoped for is not a
rule.
If you run architecture experiments, the two habits here are cheap and they work: no claim below four seeds, and
2. A positive control in the safety audit.
Mixing operators along the time axis is exactly where causality violations hide — a stack that leaks
future information will show beautiful loss curves for the wrong reason. All 49 layers pass a
negative control, and, on the same loaded checkpoint, a positive control: 16 of 16 deliberately injected faults localized exactly, across NSA, hybrid, and linear-attention layers.
This is the right shape for any correctness gate. A clean pass proves nothing unless you have also
shown the detector fires when something is actually broken. Most audits skip the second half.
Weights, training-data recipes, training code, logs, and architecture source. There is also
Aether-6B-11Attn-base — eleven mechanisms (attention, Mamba-2, Hyena, GDN, MLA among them) on an
11×11 square over 121 layers — shipped as-is, a mid-training artifact whose only job is to show
the construction is not specific to N=7. Whether the composition findings hold at N=11 is explicitly
left open.
Paper: arXiv:2609.20269 — Placement Is Free, Composition Is Not: The Latin Square as a Provably-Balanced Construction for Heterogeneous Sequence-Mixer Stacks