{"slug": "fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of", "title": "Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention", "summary": "A new study analyzing seven pretrained models reveals that the positional encoding scheme in transformers determines the spectral properties of attention heads, with RoPE inducing rotational spectra and absolute/ALiBi schemes producing non-rotational spectra. The spectral profile acts as a fingerprint that emerges after circuit formation, not as a hard constraint, as models can reroute around banned spectral channels at a cost. The findings show that positional schemes set a default spectral algebra for attention heads, shaping but not strictly limiting their function.", "body_md": "arXiv:2607.06621v1 Announce Type: new\nAbstract: The pre-softmax score of an attention head is a bilinear form $score(i,j) = x_i^T M x_j$ in a learned operator $M = W_q^T W_k$. Because M is generally non-symmetric, hence non-normal, it has a complex eigenspectrum and non-orthogonal eigenvectors, the regime where non-Hermitian and random-matrix tools apply. We ask what this spectrum encodes, at three levels for previous-token and induction circuits. Statically, across seven pretrained models spanning three positional schemes, the strongest previous-token heads are spectrally rotational under RoPE and non-rotational, or content-like, where position enters outside QK (learned-absolute and ALiBi); the model-level separation is perfect at every top-k examined (exact permutation $p=0.029$), and zeroing the per-frequency RoPE phase $Im(M_t)$ eliminates induction on a pre-identified previous-token head in all three RoPE models. Dynamically, over public Pythia checkpoints every head originates at the random-matrix (Ginibre) null; the rotational signature emerges with the behavior, not before it, and the population-median suppression that yields the final profile follows circuit formation, so the profile is a consolidated fingerprint, not a precursor. Causally, and at toy scale, no spectral channel is necessary: constrained two-layer training reroutes around every ban with capability intact, albeit at a significant formation delay (four pre-registered contrasts, $q_BH <= 0.016$). The cost structure exposes each scheme's default: imposing symmetry slows learned-absolute models by a factor of 2.9, whereas a RoPE head with a fully symmetric static M still routes directionally via the phase channel, impossible under absolute positions. Within the settings examined, the positional scheme sets the default spectral algebra of an attention head's solution: a fingerprint sculpted after function, not a hard constraint upon it.", "url": "https://wpnews.pro/news/fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of", "canonical_source": "https://arxiv.org/abs/2607.06621", "published_at": "2026-07-09 04:00:00+00:00", "updated_at": "2026-07-09 04:25:07.354275+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "artificial-intelligence", "large-language-models"], "entities": ["RoPE", "ALiBi", "Pythia", "Ginibre"], "alternates": {"html": "https://wpnews.pro/news/fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of", "markdown": "https://wpnews.pro/news/fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of.md", "text": "https://wpnews.pro/news/fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of.txt", "jsonld": "https://wpnews.pro/news/fingerprint-not-blueprint-how-positional-schemes-set-the-default-spectral-of.jsonld"}}