Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection

Researchers have developed a principled method for multilingual language steering in large language models using sparse autoencoders (SAEs), addressing the unreliability of existing English-only SAE approaches. By training SAEs on multilingual data and introducing a layer-selection rule based on the intersection of multilingual alignment and language separability, the team achieved more reliable language control across models like LLaMA-3.1-8B and Gemma-2-9B. The approach stabilizes the trade-off between language identification accuracy and generation quality, offering a predictive framework for multilingual SAE steering without exhaustive layer searches.

arXiv:2605.23036v1 Announce Type: new Abstract: Sparse autoencoders SAEs enable feature-level mechanistic interpretability and activation steering in large language models LLMs , but SAE-based language control remains unreliable in multilingual settings: most SAEs are trained on English-only data, and steering layers are chosen heuristically. We address these limitations by advancing a principled, mechanistic account of multilingual language steering with SAEs. First, we show that training SAEs on multilingual data consistently strengthens cross-lingual representations and yields more reliable, quality-preserving language control across layers and model families. Second, we introduce an \emph{a priori} steering layer-selection rule based on the intersection of multilingual alignment and language separability, which predicts effective intervention depths without exhaustive layerwise search. We evaluate our approach on LLaMA-3.1-8B and Gemma-2-9B across machine translation and cross-lingual summarization CrossSumm , using SpBLEU, ROUGE-L, COMET, and LaSE. Our results show that multilingual SAEs combined with intersection-selected layers stabilize the trade-off between language identification accuracy and generation quality, providing a principled, predictive, representation-level account of multilingual SAE steering.