Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention

A new study analyzing seven pretrained models reveals that the positional encoding scheme in transformers determines the spectral properties of attention heads, with RoPE inducing rotational spectra and absolute/ALiBi schemes producing non-rotational spectra. The spectral profile acts as a fingerprint that emerges after circuit formation, not as a hard constraint, as models can reroute around banned spectral channels at a cost. The findings show that positional schemes set a default spectral algebra for attention heads, shaping but not strictly limiting their function.

arXiv:2607.06621v1 Announce Type: new Abstract: The pre-softmax score of an attention head is a bilinear form $score i,j = x i^T M x j$ in a learned operator $M = W q^T W k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK learned-absolute and ALiBi ; the model-level separation is perfect at every top-k examined exact permutation $p=0.029$ , and zeroing the per-frequency RoPE phase $Im M t $ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix Ginibre null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay four pre-registered contrasts, $q BH <= 0.016$ . The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.