Using the Mimi codec for metalinguistic representations

Researchers Artem Saloev, Erin Pacquetet, and Nicolas Ballier found that the ABX experiment with Mimi's semantic token codebook fails to capture the mapping of semantic tokens to phone realisations, but by realigning Mimi's representations to the TIMIT corpus transcriptions, they showed that the 2048 token IDs map to quadphone, triphone, biphone, phone, and subphone realisations. The study, presented at the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics (BriGap-3) in Paris, France, in July 2026, examines theoretical consequences for speech tokenisation at allophone and subphonemic levels.

Abstract Codec-based audio language models are developing, but little explainability research has been dedicated to the representation of this type of speech tokenisation. In this paper, we focus on the dictionary of 2048 tokens used in Mimi’s semantic token codebook, the neural codec of the Moshi language model Défossez et al., 2024 . We show that the ABX experiment carried out with Mimi fails to capture the mapping of the semantic tokens to phone realisations. By realigning Mimi’s representations to the TIMIT corpus transcriptions Garofolo et al., 1993 , we show that the 2048 tokens IDs of the semantic codebook map to quadphone, triphone, biphone, phone and subphone realisations. We used the TIMIT transcriptions as evidence of the validity of the allophone-based representations of these 80ms semantic token representation and examine some of the theoretical consequences for the tokenisation of speech at allophone and subphonemic level.- Anthology ID: - 2026.brigap-1.7 - Volume: Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics BriGap-3 /volumes/2026.brigap-1/ - Month: - July - Year: - 2026 - Address: - Paris, France - Editors: Timothée Bernard /people/timothee-bernard/ , Emmanuele Chersoni /people/emmanuele-chersoni/ , Giulia Rambelli /people/giulia-rambelli/unverified/ - Venues: BriGap /venues/brigap/ | WS /venues/ws/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 63–73 - Language: - URL: https://aclanthology.org/2026.brigap-1.7/ https://aclanthology.org/2026.brigap-1.7/ - DOI: - Cite ACL : - Artem Saloev, Erin Pacquetet, and Nicolas Ballier. 2026. Using the Mimi codec for metalinguistic representations https://aclanthology.org/2026.brigap-1.7/ . In Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics BriGap-3 , pages 63–73, Paris, France. Association for Computational Linguistics. - Cite Informal : Using the Mimi codec for metalinguistic representations https://aclanthology.org/2026.brigap-1.7/ Saloev et al., BriGap 2026 - PDF: https://aclanthology.org/2026.brigap-1.7.pdf https://aclanthology.org/2026.brigap-1.7.pdf