{"slug": "off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so", "title": "Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So", "summary": "A new arXiv preprint (2608.10251v1) shows that a 12-layer transformer computes concepts in a subspace held near-orthogonal to its read-out axis, with attention 75 to 96 degrees off the read-out at every depth, and that moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation. The authors demonstrate that this off-axis geometry is functional, insulating composition from vocabulary, and that pressing every layer onto the read-out (as early-exit training does) cuts the concept-phase workspace from about 25 effective dimensions to 14 without affecting perplexity, LAMBADA, or BLiMP benchmarks. They also show the geometry can be imposed by inserting a fixed rotation at the phase boundary, which converges on all nine seeds at baseline quality, whereas prescribing it through the loss collapses in six of eight seeds.", "body_md": "arXiv:2608.10251v1 Announce Type: new\nAbstract: A transformer's answer lives on one axis: the direction its unembedding reads. Its intermediate states largely do not, and that off-axis position is usually treated as an obstacle to interpretation. We show it is functional. A 12-layer model computes in two phases. Through the first, every sublayer writes into a subspace held near-orthogonal to the read-out, attention 75 to 96 degrees off it at every depth. Moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation, and the damage is entirely in cross-token mixing: the subspace insulates composition from the vocabulary. Beneath it the frame itself turns rigidly with depth. In the second phase the answer arrives on-axis, late, and by addition rather than by turning accumulated content onto the read-out.\nPressing every layer onto the read-out instead, as training for early exit does, matches the baseline on perplexity, LAMBADA and BLiMP while cutting the concept-phase workspace from about twenty-five effective dimensions to fourteen, a change none of those benchmarks register. The geometry can also be imposed, though not by asking for it. Prescribing it through the loss is a lottery: six of eight seeds collapse, because a model told to null its read-out projection obeys most cheaply by discarding dimensions. Inserting one fixed rotation at the phase boundary lands it instead, at baseline quality. A sparse rotation the surrounding weights can absorb converges on all nine seeds, against five of nine for ordinary training. Which rotation is immaterial: twenty-five runs across thirteen distinct ones reach the same quality, and two baselines from different seeds hold their concepts in near-orthogonal frames while agreeing on their read-outs. That freedom is usable: a basis drawn at random and prescribed before training is adopted across the concept phase, with quality unchanged.", "url": "https://wpnews.pro/news/off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so", "canonical_source": "https://arxiv.org/abs/2608.10251", "published_at": "2026-08-12 04:00:00+00:00", "updated_at": "2026-08-12 04:10:04.926374+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "neural-networks", "ai-research"], "entities": ["arXiv"], "alternates": {"html": "https://wpnews.pro/news/off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so", "markdown": "https://wpnews.pro/news/off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so.md", "text": "https://wpnews.pro/news/off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so.txt", "jsonld": "https://wpnews.pro/news/off-axis-on-purpose-where-a-transformer-computes-concepts-and-why-it-does-so.jsonld"}}