{"slug": "canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and", "title": "Canonical-basis realignment for Transformer LLMs: every hidden axis becomes independently measurable and controllable", "summary": "A new technique called canonical-basis realignment rotates a Transformer's internal coordinate system into a lossless canonical basis, enabling every hidden axis to be independently measured and controlled without altering outputs or perplexity scores. Researchers applying it to models like Qwen and Pythia observed internal mechanisms including a bipolar oscillator, rhythmic layer-wise respiration, and a homeostatic defense that erases localized perturbations within a couple of layers, and found that a half-billion-parameter model's effective correlation rank may be as low as eleven independent patterns.", "body_md": "The code essentially gives you a way to rotate a Transformer's internal coordinate system into a canonical basis that aligns with its own weight matrices in a lossless way. By absorbing the normalization gains directly into the adjacent weights and using orthogonal matrices built from the singular vectors of the model, you can transform architectures like Qwen or Pythia without altering their outputs or perplexity scores.\n\nApplying this transform reveals the actual hidden geometric structures operating inside the network. Once the model is rotated into this new perspective, you can see its internal mechanisms that were previously opaque. The authors found things like a bipolar oscillator where specific axes form inhibitory pairs that fire against each other in perfect opposition. They also observed a kind of rhythmic respiration across layers where the model alternates between absorbing knowledge and filtering it. On top of that, it exposed a homeostatic defense mechanism that aggressively erases any localized perturbations within just a couple of layers.\n\nPractically speaking, researchers now have a powerful lens for mapping out how models actually do reasoning. For example, it turns out that the effective rank of the correlation matrix in a half billion parameter model might be as low as eleven independent patterns. Reframing how we look at the internal activations of language models provides a standardized way to study their underlying architecture.", "url": "https://wpnews.pro/news/canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and", "canonical_source": "https://github.com/todotge/canonical-basis", "published_at": "2026-08-29 20:16:54+00:00", "updated_at": "2026-08-29 20:48:39.218440+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research"], "entities": ["Qwen", "Pythia"], "alternates": {"html": "https://wpnews.pro/news/canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and", "markdown": "https://wpnews.pro/news/canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and.md", "text": "https://wpnews.pro/news/canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and.txt", "jsonld": "https://wpnews.pro/news/canonical-basis-realignment-for-transformer-llms-every-hidden-axis-becomes-and.jsonld"}}