{"slug": "sharedsae-one-feature-dictionary-across-language-models", "title": "SharedSAE: One Feature Dictionary Across Language Models", "summary": "Researchers introduced SharedSAE, a method that trains a single sparse autoencoder dictionary shared across multiple language models, replacing dedicated per-model SAEs. On four 1B-scale base language models from distinct families, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance, achieves cross-model latent correlations 1.8 times higher than post-hoc aligned separate SAEs, and enables efficient adaptation of new models to the frozen dictionary. The approach preserves activation magnitudes and supports single-model inference via model dropout.", "body_md": "arXiv:2609.04344v1 Announce Type: new \nAbstract: Sparse autoencoders (SAEs) are widely used to interpret language model activations, but SAE training and latent labelling are typically repeated for every model. Here, we show that a single shared SAE can replace a collection of dedicated per-model SAEs. Our method, SharedSAE, combines a shared dictionary with model-specific encoder-decoder pairs. Unlike the closest prior method, which discards activation magnitudes and requires all models at inference, SharedSAE instead normalizes only selection scores, preserving magnitudes, and uses model dropout for single-model inference. We train SharedSAE on four 1B-scale base language models spanning distinct families and tokenizers. Despite sharing its latents across models, SharedSAE retains 96.6% of dedicated SAEs' mean explained variance; its latent activations exhibit cross-model correlations 1.8 times as high as separate SAEs aligned post-hoc, and its latent descriptions transfer across models. After the dictionary is frozen, new models can be efficiently adapted to it, achieving near-dedicated-SAE reconstruction quality while reusing the shared latent descriptions.", "url": "https://wpnews.pro/news/sharedsae-one-feature-dictionary-across-language-models", "canonical_source": "https://arxiv.org/abs/2609.04344", "published_at": "2026-09-07 04:00:00+00:00", "updated_at": "2026-09-07 04:27:05.005477+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-research"], "entities": ["SharedSAE"], "alternates": {"html": "https://wpnews.pro/news/sharedsae-one-feature-dictionary-across-language-models", "markdown": "https://wpnews.pro/news/sharedsae-one-feature-dictionary-across-language-models.md", "text": "https://wpnews.pro/news/sharedsae-one-feature-dictionary-across-language-models.txt", "jsonld": "https://wpnews.pro/news/sharedsae-one-feature-dictionary-across-language-models.jsonld"}}