Dissecting Sensitivity to Training Language in Self-Supervised Speech Learning Using Neural Audio Codec Tokens

A systematic analysis of language sensitivity in codec-based self-supervised learning (SSL) models shows that downstream performance is insensitive to the neural audio codec (NAC) training language but strongly dependent on the SSL pre-training language, according to a new arXiv preprint (2607.26350v1). The findings suggest a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.

arXiv:2607.26350v1 Announce Type: cross Abstract: Neural audio codecs NACs have become popular for obtaining speech representations as discrete tokens. Beyond compression, discrete tokens can be used to train self-supervised learning SSL models. Such models, referred to as codec-based SSL models, reduce data storage and computational cost, enabling scalable SSL pre-training. However, their language sensitivity remains unclear. When the language changes, codec-based SSL models may require retraining, which undermines their efficiency. In this paper, we present a systematic analysis of language sensitivity by varying either the NAC training language or the SSL pre-training language while keeping the other fixed. Experimental results show that downstream performance is insensitive to the NAC training language but strongly dependent on the SSL pre-training language. These findings suggest that a single NAC can be reused across languages, while aligning the SSL pre-training language with the target language is crucial.