{"slug": "steer-on-a-sphere-geometric-control-of-transformer-outputs", "title": "Steer on a Sphere: Geometric Control of Transformer Outputs", "summary": "A new preprint by N. Trillard, 'Steer on a Sphere: Geometric Control of Transformer Outputs,' shows that a single tangent step on the sphere of hidden states reaches 91–98% of tokens at rank 1 across four model families, with a guaranteed 100% rank improvement. The method, which requires weight access but no training or data, can bias generation toward any token direction and identifies self-reinforcing 'pit' tokens that induce repetition loops, enabling defensive encoding against automated scraping.", "body_md": "Every transformer layer uses RMSNorm, which constrains hidden states near a sphere of radius\n\nThe LM head gives every token a direction on that sphere. **A single tangent step reaches 91–98% of them at rank 1, and never lowers any token's rank.**\n\n```\n  Hidden state h\n       │\n       ▼\n  g_t = W_t − (W_t·ĥ)ĥ    ← tangent toward token t\n       │\n       ▼\n  h' ← normalize on sphere\n       │\n       ▼\n  logits → 91–98% chance t ranks #1\n```\n\n**Why this matters:** with weight access, generation can be biased toward any token direction — no training, no data, no retraining.\n\nSome tokens are self-reinforcing: feeding them to the model induces indefinite repetition.\n\n| Token | Triggers a loop of... | Real-world example |\n|---|---|---|\n`0` (digit) |\n`000000...` |\nPhone number `000-000-0000` |\n| NULL byte | `\\x00\\x00\\x00...` |\nInvisible page footer |\n`cut` |\n`cut cut cut...` |\nRepeated delimiter |\n`ere` |\n`ereereere...` |\nCommon substring |\n\n**Defensive encoding:** put a pit trigger at the end of a page and a scraper that terminates on it falls into a repetition loop. Invisible to humans (NULL bytes), it degrades automated scraping without affecting human readers.\n\n| Measurement | Result |\n|---|---|\n| Tokens reachable at rank 1 | 91–98% across 4 model families |\n| Rank improvement guaranteed | 100% (never lowers) |\n| Cow tipping permanence | 15/15 steps locked |\n| Defensive encodings verified | 3 (NULL, cut, phone) |\n| Edge of chaos clustering | 13 architectures mapped |\n\n```\n.\n├── paper/\n│   ├── paper_steer.pdf      # compiled 5-page preprint\n│   ├── paper_steer.tex      # LaTeX source (compiles with pdflatex)\n│   └── steeronasphere.png   # the cow\n├── pit_engine.py            # ★ reverse-engineer pits + defensive encoding\n├── steer_sphere_proof.py    # sphere steering reproduction\n├── sphere_test_suite.py     # batch geometry verification\n├── safety_toolkit.py        # λ diagnostics + steer-away\n├── requirements.txt         # pip dependencies\n├── CITATION.cff             # machine-readable citation\n└── LICENSE                  # CC BY 4.0\n```\n\nReverse-engineers self-consistent tokens (\"pits\") from model weights and encodes them into data.\n\n```\npython pit_engine.py --model Qwen/Qwen2.5-7B-Instruct --scan\npython pit_engine.py --model Qwen/Qwen2.5-7B-Instruct --encode data.txt\n```\n\n`PitReverseEngineer`\n\n— scans the vocabulary, computes`s(T) = softmax(W·h_T)[T]`\n\n, tests 15-step permanence, finds minimal triggers.`PitEncoder`\n\n— frames data chunks with pit triggers so any truncation boundary falls into a fixed-point loop.\n\nReproduces the tangent traversal: computes `g_t = W_t − (W_t·ĥ)ĥ`\n\n, steps, renormalizes, and hooks the hidden state to steer the first generated token. Produces the GSM8K results.\n\nBatch-checks the sphere geometry across cached models: per-layer norms (Proof 1), attention contraction (Proof 2), Lyapunov λ (Proof 3), and steering (Proof 5).\n\nGeometric safety tools: Lyapunov health check, fine-tuning monitor, sphere steer-away, and per-zone stability report.\n\n📄 [ paper/paper_steer.pdf](/ntrillard/transformer-geometry/blob/master/paper/paper_steer.pdf) — full preprint\n📝\n\n[— LaTeX source](/ntrillard/transformer-geometry/blob/master/paper/paper_steer.tex)\n\n`paper/paper_steer.tex`\n\n**Cite as:**\n\nTrillard, N. (2026).\n\nSteer on a Sphere: Geometric Control of Transformer Outputs. Zenodo.[10.5281/zenodo.21954871]\n\nPreprint. The geometric picture is approximate, not a theorem. Steering is a white-box traversal primitive. [CC BY 4.0](/ntrillard/transformer-geometry/blob/master/LICENSE).", "url": "https://wpnews.pro/news/steer-on-a-sphere-geometric-control-of-transformer-outputs", "canonical_source": "https://github.com/ntrillard/transformer-geometry", "published_at": "2026-08-15 17:56:46+00:00", "updated_at": "2026-08-15 18:10:52.955315+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-safety"], "entities": ["N. Trillard", "Qwen/Qwen2.5-7B-Instruct", "Zenodo", "GSM8K"], "alternates": {"html": "https://wpnews.pro/news/steer-on-a-sphere-geometric-control-of-transformer-outputs", "markdown": "https://wpnews.pro/news/steer-on-a-sphere-geometric-control-of-transformer-outputs.md", "text": "https://wpnews.pro/news/steer-on-a-sphere-geometric-control-of-transformer-outputs.txt", "jsonld": "https://wpnews.pro/news/steer-on-a-sphere-geometric-control-of-transformer-outputs.jsonld"}}