{"slug": "distributed-ranked-structure-in-contrastive-logit-steering", "title": "Distributed Ranked Structure in Contrastive Logit Steering", "summary": "A causal investigation of vocabulary-logit steering on Qwen2-1.5B found that the largest coordinates are not necessarily the ones driving the effect, with transport appearing around K≈150 and persisting through K≈300, and K=200 as the maximum-alignment point (cos→dLref = 1.000). The study, by ntrillard, showed that the pure row(W) projection (λ=1) yielded no transport in 30 seeds, while all λ<1 conditions retained a nonzero out-of-row component and showed some transport, with absolute rates low (3–6/30). The effect is lexical forcing, not semantic transport, as LEX (boosted top-200) achieved 27/30 transport vs 0/30 unsteered, while SEM (unboosted semantic neighbors) showed 0/30 in both conditions.", "body_md": "Hi,\n\nI ran a few tests on the steering vectors and found an interesting result: the largest coordinates aren’t necessarily the ones driving the effect.\n\n## Sparse ranked logit steering on Qwen2-1.5B: which coordinates actually matter?\n\nResults from a causal investigation of vocabulary-logit steering on Qwen2-1.5B. All code, scripts, and per-seed logs are in the repository; the numbers below are directly reproducible from it.\n\nRepository:[github.com/ntrillard/logit-steering]— writeup in`writeup-orthogonal-complement.md`\n\n(Parts I–IX)\n\nRelated (sphere/geometry lineage, negative controls):[github.com/ntrillard/transformer-geometry]\n\nMethod details:[writeup-orthogonal-complement.md]## Setup\n\n- Model: Qwen2-1.5B (bf16). Contrast of vocabulary-logit readouts on target-topic vs neutral sentences, per-token z-scored, averaged, re-z-scored.\n- Steering vector\n`dL = zscore(mean_tgt − mean_neu) · top200`\n\n(top-200 positive coordinates with ranked magnitudes), norm-matched; applied as a logit offset (`α=2.0`\n\n, after step 20, nucleus sampling`p=0.9`\n\n).Transport= generated text contains a held-out target word (stem-matched, case-normalized) and is non-degenerate (no token run ≥6, type/token > 0.6).`medMinR`\n\n= median over seeds of the minimum held-out token rank; rank 0 = top token. 30 seeds per condition unless noted.## Results\n\n1. K window (30-seed confirmation):\n\nK transport medMinR cos→dLref 150 2/30 1 +0.890 200 3/30 0 +1.000 250 3/30 1 +0.913\n\nTransport appears around K≈150 and persists through K≈300; dilution beyond. K=200 is the maximum-alignment point (cos→dLref = 1.000), not a unique behavioral optimum.\n\n2. K × λ causal surface (λ = blend from residual toward the row(W) projection):\n\nλ transport medMinR R_row cos_ref 0.00 4/30 2 0.000 +0.981 0.25 4/30 1 0.004 +0.992 0.50 3/30 0 0.038 +1.000 0.75 6/30 0 0.263 +0.942 1.00 0/30 23 1.000 +0.195\n\nEvery λ<1 condition retains a nonzero out-of-row component and shows some transport; the pure row(W) projection (λ=1) shows none in 30 seeds. Absolute rates are low (3–6/30) and do not increase monotonically with the out-of-row component.\n\n3. Causal factorial (SEEDS=6):`rand200`\n\n/`magmatch200`\n\n/`shuffle200`\n\n/`equal200`\n\n/`rowW_proj`\n\n→ 0/6 (no transport).`raw_t200`\n\n/`perz_t200`\n\n(correct coordinates × ranked magnitudes × out-of-row) → 2/6. Normalization (raw/centered/z/perz) did not materially change results once top-k coordinates + ranked magnitudes were fixed.\n\n4. Lexical vs semantic (SEEDS=30):\n\nprobe UNSTEERED STEERED LEX (boosted top-200) 0/30, rank 45 27/30, rank 0 SEM (unboosted semantic neighbors) 0/30, rank 188 0/30, rank 172 UNR (unboosted unrelated) 18/30, rank 0 16/30, rank 1\n\nEffect is lexical forcing, not semantic transport.\n\n5. Static vs adaptive (SEEDS=30):STATIC 3/30; DYN_PREFIX (recompute per prefix) 1/30 (cos vs static +0.955); DYN_SELF 0/30 (cos +0.013). Recomputation barely changes the vector and does not help.\n\n6. Ranking causality / ablations (Test C + Test D):\n\ncondition SEEDS=3 SEEDS=30 (Test D) NONE (baseline) 0/3 0/30, rank 160 top1/5/10/20/50 0/3 0/30 (rank ~141–271) full200 2/3 5/30, rank 4 full200 − largest coord 2/3 4/30 full200 − random coord 2/3 5/30 rand50 / top50_shuf 0/3 0/30\n\n- Test D: 5/30 transport vs 0/30 baseline, median held-out rank 160 → 4, 8/30 seeds reach rank 0 (baseline 0/30).\n- No single load-bearing coordinate: deleting the single largest coordinate survives.\n- The observation depends on both coordinate identity and the coordinate↔ranked-magnitude association.\n\n7. Generalization (Test A, baseline-corrected, SEEDS=3):\n\nconcept prompt NONE base best steered FANTASY town 0/3 2/3 (rank 0) FANTASY beach 0/3 ~0–1/3 SPACE both 0/3 0/3 (no observed transport) PIRATE both 1/3 ~1/3 (baseline-contaminated)\n\nTransport is concept- and prompt-dependent; some contrasts show no observed transport. No cheap vector metric cleanly predicts success in this 3-concept set.\n\n## Notes\n\n- Statistical status: exact Fisher tests on per-seed counts. LEX 27/30 vs 0/30 is decisive (p<0.001). The λ=0.75 cell is nominal two-sided p=0.024 but not significant after Holm correction across the 4 tested λ cells. Headline window-vs-baseline contrasts (5/30 vs 0/30, two-sided p=0.052; pooled 8/90 vs 0/30, p=0.199) are suggestive, not significant at n=30. Pilot findings on one model family.\n- Prior art: sparse steering is not new (CAA, arXiv:2308.10248; SAS, arXiv:2503.00177; SAE-SSV, 2025.emnlp-main.112; CAS-BiPO, 2026.findings-eacl.57). ActAdd Appendix H reports a partial-vector window observation (70% of dims > 100% for one prompt). “What Drives Representation Steering?” (arXiv:2604.08524) runs bottom-k (retain-largest-coordinates) and random-dropout baselines with refusal ASR metrics. “Steerable but Not Decodable” (arXiv:2604.02608) shows function-vector steering can work when token projection is incoherent.\n\nReproduce:all commands in the repository README; scripts`mechanism_matrix.py`\n\n,`neighbor_probe.py`\n\n,`dynamic_contrast.py`\n\n,`generalize.py`\n\n,`rank_causality.py`\n\nwith env-configured SEEDS/K/LAMBDA/COND.", "url": "https://wpnews.pro/news/distributed-ranked-structure-in-contrastive-logit-steering", "canonical_source": "https://discuss.huggingface.co/t/distributed-ranked-structure-in-contrastive-logit-steering/179703#post_1", "published_at": "2026-09-01 16:33:21+00:00", "updated_at": "2026-09-01 16:56:09.923529+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research"], "entities": ["Qwen2-1.5B", "ntrillard", "github.com/ntrillard/logit-steering", "github.com/ntrillard/transformer-geometry"], "alternates": {"html": "https://wpnews.pro/news/distributed-ranked-structure-in-contrastive-logit-steering", "markdown": "https://wpnews.pro/news/distributed-ranked-structure-in-contrastive-logit-steering.md", "text": "https://wpnews.pro/news/distributed-ranked-structure-in-contrastive-logit-steering.txt", "jsonld": "https://wpnews.pro/news/distributed-ranked-structure-in-contrastive-logit-steering.jsonld"}}