# Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals

> Source: <https://aclanthology.org/2026.babylm-main.32/>
> Published: 2026-10-24 00:00:00+00:00

##### Abstract

Multilingual pre-training under severe data con straints can suffer from limited representational capacity and uneven tokenization across lan guages. Addressing the BabyLM 2026 Multi lingual Track, we study a compact pre-training recipe under the 100M-word budget. Our main contribution is a Calibrated Regex-Guided Byte-Level BPE (Regex-BBPE) tokenizer that combines script-aware pre-tokenization with byte-premium-adjusted tokenizer training in a 16K vocabulary. We pair this tokenizer with the previously proposed Attention Residuals mechanism and the Muon optimizer, and eval uate three English/Dutch/Chinese mixing ra tios. Among the tested mixtures, the Chinese dominant configuration (20% English, 20% Dutch, 60% Chinese) gives the strongest ag gregate fine-tuning results. Zero-shot results are mixed and remain slightly below the offi cial GPT-2 trilingual baseline on average, while fine-tuning improves the language-level aver ages by 8.55, 9.55, and 10.09 accuracy points for English, Dutch, and Chinese, respectively. The mixing-ratio experiments also show differ ent sensitivity patterns for the two West Ger manic languages and Chinese; we treat this as an observation within the tested language triple rather than a general cross-lingual law.
- Anthology ID:
- 2026.babylm-main.32
- Volume:
- [Proceedings of the Second BabyLM Workshop](https://aclanthology.org/volumes/2026.babylm-main/)
- Month:
- October
- Year:
- 2026
- Address:
- Budapest, Hungary
- Editors:
- [Leshem Choshen](https://aclanthology.org/people/leshem-choshen/) ,[Ryan Cotterell](https://aclanthology.org/people/ryan-cotterell/) ,[Mustafa Omer Gul](https://aclanthology.org/people/mustafa-omer-gul/) ,[Jaap Jumelet](https://aclanthology.org/people/jaap-jumelet/unverified/) ,[Tal Linzen](https://aclanthology.org/people/tal-linzen/) ,[Aaron Mueller](https://aclanthology.org/people/aaron-mueller/) ,[Suchir Salhan](https://aclanthology.org/people/suchir-salhan/) ,[Raj Sanjay Shah](https://aclanthology.org/people/raj-sanjay-shah/) ,[Alex Warstadt](https://aclanthology.org/people/alex-warstadt/) ,[Ethan Gotlieb Wilcox](https://aclanthology.org/people/ethan-wilcox/)
- Venues:
- [BabyLM](https://aclanthology.org/venues/babylm/) |[WS](https://aclanthology.org/venues/ws/)
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 403–412
- Language:
- URL:
- [https://aclanthology.org/2026.babylm-main.32/](https://aclanthology.org/2026.babylm-main.32/)
- DOI:
- Cite (ACL):
- Yihan Liu, Tongjing Zhang, and Pierre Nugues. 2026. [Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals](https://aclanthology.org/2026.babylm-main.32/) . In*Proceedings of the Second BabyLM Workshop* , pages 403–412, Budapest, Hungary. Association for Computational Linguistics.
- Cite (Informal):
- [Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals](https://aclanthology.org/2026.babylm-main.32/) (Liu et al., BabyLM 2026)
- PDF:
- [https://aclanthology.org/2026.babylm-main.32.pdf](https://aclanthology.org/2026.babylm-main.32.pdf)