Depth-Dependent Safety Geometry in Vision-Language Models: Layer-Wise Localization and Bidirectional Refusal Control

A lightweight inference-time framework can localize and manipulate safety-sensitive representations in frozen vision-language models without retraining, according to arXiv paper 2610.32182v1. On Qwen3-VL-4B, activation steering along a learned safety direction in early vision layers produced bidirectional refusal control with unsafe rejection spanning 6% to 96%, while deeper layers stayed separable along the safety direction but were largely insensitive to steering; InternVL2.5-4B showed qualitatively similar depth-dependent geometry. The authors say the results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.

arXiv:2610.32182v1 Announce Type: cross Abstract: Vision-Language Models VLMs are widely deployed, yet the internal mechanisms behind their safety behavior remain poorly understood. We present a lightweight inference-time framework for localizing and manipulating safety-sensitive representations in frozen VLMs, without retraining or modifying parameters. On Qwen3-VL-4B, layer-wise activation divergence analysis reveals non-uniform safety geometry across depth. Causal interventions show that early vision layers support strong bidirectional refusal control through activation steering along a learned safety direction, with unsafe rejection spanning 6%-96%, whereas later layers respond more to activation magnitude scaling and exhibit a substantial safety-selectivity trade-off. Deeper layers remain well separated along the safety direction yet are largely insensitive to steering, distinguishing representational separability from causal controllability. InternVL2.5-4B shows qualitatively similar depth-dependent activation geometry despite differences in layer location and magnitude. Our results expose a white-box, representation-level refusal-bypass attack surface in VLM vision encoders and motivate inference-time stabilization of safety-sensitive representations.