Steer on a Sphere: Geometric Control of Transformer Outputs

A new preprint by N. Trillard, 'Steer on a Sphere: Geometric Control of Transformer Outputs,' shows that a single tangent step on the sphere of hidden states reaches 91–98% of tokens at rank 1 across four model families, with a guaranteed 100% rank improvement. The method, which requires weight access but no training or data, can bias generation toward any token direction and identifies self-reinforcing 'pit' tokens that induce repetition loops, enabling defensive encoding against automated scraping.

Every transformer layer uses RMSNorm, which constrains hidden states near a sphere of radius The LM head gives every token a direction on that sphere. A single tangent step reaches 91–98% of them at rank 1, and never lowers any token's rank. Hidden state h │ ▼ g t = W t − W t·ĥ ĥ ← tangent toward token t │ ▼ h' ← normalize on sphere │ ▼ logits → 91–98% chance t ranks 1 Why this matters: with weight access, generation can be biased toward any token direction — no training, no data, no retraining. Some tokens are self-reinforcing: feeding them to the model induces indefinite repetition. | Token | Triggers a loop of... | Real-world example | |---|---|---| 0 digit | 000000... | Phone number 000-000-0000 | | NULL byte | \x00\x00\x00... | Invisible page footer | cut | cut cut cut... | Repeated delimiter | ere | ereereere... | Common substring | Defensive encoding: put a pit trigger at the end of a page and a scraper that terminates on it falls into a repetition loop. Invisible to humans NULL bytes , it degrades automated scraping without affecting human readers. | Measurement | Result | |---|---| | Tokens reachable at rank 1 | 91–98% across 4 model families | | Rank improvement guaranteed | 100% never lowers | | Cow tipping permanence | 15/15 steps locked | | Defensive encodings verified | 3 NULL, cut, phone | | Edge of chaos clustering | 13 architectures mapped | . ├── paper/ │ ├── paper steer.pdf compiled 5-page preprint │ ├── paper steer.tex LaTeX source compiles with pdflatex │ └── steeronasphere.png the cow ├── pit engine.py ★ reverse-engineer pits + defensive encoding ├── steer sphere proof.py sphere steering reproduction ├── sphere test suite.py batch geometry verification ├── safety toolkit.py λ diagnostics + steer-away ├── requirements.txt pip dependencies ├── CITATION.cff machine-readable citation └── LICENSE CC BY 4.0 Reverse-engineers self-consistent tokens "pits" from model weights and encodes them into data. python pit engine.py --model Qwen/Qwen2.5-7B-Instruct --scan python pit engine.py --model Qwen/Qwen2.5-7B-Instruct --encode data.txt PitReverseEngineer — scans the vocabulary, computes s T = softmax W·h T T , tests 15-step permanence, finds minimal triggers. PitEncoder — frames data chunks with pit triggers so any truncation boundary falls into a fixed-point loop. Reproduces the tangent traversal: computes g t = W t − W t·ĥ ĥ , steps, renormalizes, and hooks the hidden state to steer the first generated token. Produces the GSM8K results. Batch-checks the sphere geometry across cached models: per-layer norms Proof 1 , attention contraction Proof 2 , Lyapunov λ Proof 3 , and steering Proof 5 . Geometric safety tools: Lyapunov health check, fine-tuning monitor, sphere steer-away, and per-zone stability report. 📄 paper/paper steer.pdf /ntrillard/transformer-geometry/blob/master/paper/paper steer.pdf — full preprint 📝 — LaTeX source /ntrillard/transformer-geometry/blob/master/paper/paper steer.tex paper/paper steer.tex Cite as: Trillard, N. 2026 . Steer on a Sphere: Geometric Control of Transformer Outputs. Zenodo. 10.5281/zenodo.21954871 Preprint. The geometric picture is approximate, not a theorem. Steering is a white-box traversal primitive. CC BY 4.0 /ntrillard/transformer-geometry/blob/master/LICENSE .