Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals

Yihan Liu, Tongjing Zhang and Pierre Nugues report in a paper presented at the Second BabyLM Workshop in Budapest that a compact multilingual pre-training recipe under the 100M-word budget, built on a Calibrated Regex-Guided Byte-Level BPE tokenizer with a 16K vocabulary, Attention Residuals and the Muon optimizer, improved fine-tuning language-level averages by 8.55, 9.55 and 10.09 accuracy points for English, Dutch and Chinese respectively. Among the three tested English/Dutch/Chinese mixing ratios, the Chinese-dominant configuration (20% English, 20% Dutch, 60% Chinese) gave the strongest aggregate fine-tuning results, while zero-shot results remained slightly below the official GPT-2 trilingual baseline on average. The authors treat the differing sensitivity patterns of the two West Germanic languages and Chinese as an observation within the tested language triple rather than a general cross-lingual law.

Abstract Multilingual pre-training under severe data con straints can suffer from limited representational capacity and uneven tokenization across lan guages. Addressing the BabyLM 2026 Multi lingual Track, we study a compact pre-training recipe under the 100M-word budget. Our main contribution is a Calibrated Regex-Guided Byte-Level BPE Regex-BBPE tokenizer that combines script-aware pre-tokenization with byte-premium-adjusted tokenizer training in a 16K vocabulary. We pair this tokenizer with the previously proposed Attention Residuals mechanism and the Muon optimizer, and eval uate three English/Dutch/Chinese mixing ra tios. Among the tested mixtures, the Chinese dominant configuration 20% English, 20% Dutch, 60% Chinese gives the strongest ag gregate fine-tuning results. Zero-shot results are mixed and remain slightly below the offi cial GPT-2 trilingual baseline on average, while fine-tuning improves the language-level aver ages by 8.55, 9.55, and 10.09 accuracy points for English, Dutch, and Chinese, respectively. The mixing-ratio experiments also show differ ent sensitivity patterns for the two West Ger manic languages and Chinese; we treat this as an observation within the tested language triple rather than a general cross-lingual law. - Anthology ID: - 2026.babylm-main.32 - Volume: - Proceedings of the Second BabyLM Workshop https://aclanthology.org/volumes/2026.babylm-main/ - Month: - October - Year: - 2026 - Address: - Budapest, Hungary - Editors: - Leshem Choshen https://aclanthology.org/people/leshem-choshen/ , Ryan Cotterell https://aclanthology.org/people/ryan-cotterell/ , Mustafa Omer Gul https://aclanthology.org/people/mustafa-omer-gul/ , Jaap Jumelet https://aclanthology.org/people/jaap-jumelet/unverified/ , Tal Linzen https://aclanthology.org/people/tal-linzen/ , Aaron Mueller https://aclanthology.org/people/aaron-mueller/ , Suchir Salhan https://aclanthology.org/people/suchir-salhan/ , Raj Sanjay Shah https://aclanthology.org/people/raj-sanjay-shah/ , Alex Warstadt https://aclanthology.org/people/alex-warstadt/ , Ethan Gotlieb Wilcox https://aclanthology.org/people/ethan-wilcox/ - Venues: - BabyLM https://aclanthology.org/venues/babylm/ | WS https://aclanthology.org/venues/ws/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 403–412 - Language: - URL: - https://aclanthology.org/2026.babylm-main.32/ https://aclanthology.org/2026.babylm-main.32/ - DOI: - Cite ACL : - Yihan Liu, Tongjing Zhang, and Pierre Nugues. 2026. Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals https://aclanthology.org/2026.babylm-main.32/ . In Proceedings of the Second BabyLM Workshop , pages 403–412, Budapest, Hungary. Association for Computational Linguistics. - Cite Informal : - Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals https://aclanthology.org/2026.babylm-main.32/ Liu et al., BabyLM 2026 - PDF: - https://aclanthology.org/2026.babylm-main.32.pdf https://aclanthology.org/2026.babylm-main.32.pdf