Abstract
Multilingual pre-training under severe data con straints can suffer from limited representational capacity and uneven tokenization across lan guages. Addressing the BabyLM 2026 Multi lingual Track, we study a compact pre-training recipe under the 100M-word budget. Our main contribution is a Calibrated Regex-Guided Byte-Level BPE (Regex-BBPE) tokenizer that combines script-aware pre-tokenization with byte-premium-adjusted tokenizer training in a 16K vocabulary. We pair this tokenizer with the previously proposed Attention Residuals mechanism and the Muon optimizer, and eval uate three English/Dutch/Chinese mixing ra tios. Among the tested mixtures, the Chinese dominant configuration (20% English, 20% Dutch, 60% Chinese) gives the strongest ag gregate fine-tuning results. Zero-shot results are mixed and remain slightly below the offi cial GPT-2 trilingual baseline on average, while fine-tuning improves the language-level aver ages by 8.55, 9.55, and 10.09 accuracy points for English, Dutch, and Chinese, respectively. The mixing-ratio experiments also show differ ent sensitivity patterns for the two West Ger manic languages and Chinese; we treat this as an observation within the tested language triple rather than a general cross-lingual law.
- Anthology ID:
- 2026.babylm-main.32
- Volume:
- [Proceedings of the Second BabyLM Workshop](https://aclanthology.org/volumes/2026.babylm-main/)
- Month:
- October
- Year:
- 2026
- Address:
- Budapest, Hungary
- Editors:
- Leshem Choshen ,Ryan Cotterell ,Mustafa Omer Gul ,Jaap Jumelet ,Tal Linzen ,Aaron Mueller ,Suchir Salhan ,Raj Sanjay Shah ,Alex Warstadt ,Ethan Gotlieb Wilcox
- Venues:
- [BabyLM](https://aclanthology.org/venues/babylm/) |[WS](https://aclanthology.org/venues/ws/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 403–412
- Language:
- URL:
- [https://aclanthology.org/2026.babylm-main.32/](https://aclanthology.org/2026.babylm-main.32/)
- DOI:
- Cite (ACL):
- Yihan Liu, Tongjing Zhang, and Pierre Nugues. 2026. Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals . InProceedings of the Second BabyLM Workshop , pages 403–412, Budapest, Hungary. Association for Computational Linguistics.
- Cite (Informal):
- [Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals](https://aclanthology.org/2026.babylm-main.32/) (Liu et al., BabyLM 2026)
- PDF:
- [https://aclanthology.org/2026.babylm-main.32.pdf](https://aclanthology.org/2026.babylm-main.32.pdf)