{"slug": "depth-aware-activation-steering-via-attention-residuals", "title": "Depth-Aware Activation Steering via Attention Residuals", "summary": "A new arXiv paper (2611.21655v1) reports that distributing a persona vector across layers using Attention Residuals (AttnRes) depth weights produces stronger internal alignment and behavioral shifts than single-layer activation steering on two small open models, Qwen3-0.6B and Qwen3.5-0.8B, with deeper-layer concept cosine and rank gains at p<0.0001. The authors report the effect is sign-specific — reverse vectors invert all six metrics and random directions degrade alignment (p<0.0001) — that injection rewires attention strictly downstream (upstream JS = 0.00000), and that training the depth weights on a steering objective for 200 steps with a frozen backbone raises final alignment 10x on held-out prompts. They also flag an intensity window where effects are monotone over 0.3–0.7 before generation collapses (617 to 306 to 25 chars), and that keyword-rule metrics failed LLM-judge validation (kappa <= 0.09), forcing two rule-based conclusions to be retracted; code and data are released.", "body_md": "arXiv:2611.21655v1 Announce Type: new \nAbstract: Activation steering is a simple, training-free way to modulate LLM behavior, yet existing methods typically inject vectors at a single layer and lack mechanisms for depth-aware propagation or internal verification. We combine two recent ideas - Attention Residuals (AttnRes) providing learnable cross-layer pathways, and Jacobian-lens (J-space) concept readout - into a lightweight depth-aware steering pipeline. On two small open models (Qwen3-0.6B and Qwen3.5-0.8B), distributing a persona vector according to AttnRes-style depth weights yields stronger internal alignment (deeper-layer concept cosine and rank, p<0.0001) and stronger behavioral shifts than single-layer injection. The effect is sign-specific: reverse vectors invert all six metrics, random directions degrade alignment (p<0.0001), and a short LoRA fine-tune erases the imprint. Injection rewires attention strictly downstream of the injection point (upstream JS = 0.00000). Three practical findings follow. First, a layer-wise geometric decomposition of cosine readouts: at the injection layer 123% of the increase is arithmetic, while 5-11 layers downstream 100% is dynamical. Second, an intensity window: effects are monotone over 0.3-0.7, then generation collapses (617 to 306 to 25 chars). Third, metric discipline: keyword-rule metrics fail LLM-judge validation (kappa <= 0.09), and two rule-based conclusions were retracted. Training the depth weights on a steering objective (200 steps, frozen backbone) raises final alignment 10x on held-out prompts. Cross-persona vectors retain 96-99% of the effect: specificity holds against random, not semantic, directions. We contribute a practical, low-overhead depth-aware steering method with a careful evaluation protocol. Code and data are released.", "url": "https://wpnews.pro/news/depth-aware-activation-steering-via-attention-residuals", "canonical_source": "https://arxiv.org/abs/2611.21655", "published_at": "2026-11-18 05:00:00+00:00", "updated_at": "2026-11-18 05:14:49.645562+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-safety", "machine-learning"], "entities": ["Attention Residuals", "AttnRes", "Qwen3-0.6B", "Qwen3.5-0.8B", "arXiv", "LoRA"], "also_reported_by": [], "takeaway": "Distributing a persona steering vector across layers by AttnRes depth weights beats single-layer injection on internal alignment (p<0.0001), but keep steering intensity within 0.3-0.7, since generation collapses from 617 to 306 to 25 characters beyond that window.", "recommendation": {"advice": "Validate steering metrics against an LLM judge before trusting them: keyword-rule metrics scored kappa <= 0.09 and forced two retractions, so gate conclusions on judge agreement.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/depth-aware-activation-steering-via-attention-residuals", "markdown": "https://wpnews.pro/news/depth-aware-activation-steering-via-attention-residuals.md", "text": "https://wpnews.pro/news/depth-aware-activation-steering-via-attention-residuals.txt", "jsonld": "https://wpnews.pro/news/depth-aware-activation-steering-via-attention-residuals.jsonld"}}