{"slug": "compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex", "title": "Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals", "summary": "Yihan Liu, Tongjing Zhang and Pierre Nugues report in a paper presented at the Second BabyLM Workshop in Budapest that a compact multilingual pre-training recipe under the 100M-word budget, built on a Calibrated Regex-Guided Byte-Level BPE tokenizer with a 16K vocabulary, Attention Residuals and the Muon optimizer, improved fine-tuning language-level averages by 8.55, 9.55 and 10.09 accuracy points for English, Dutch and Chinese respectively. Among the three tested English/Dutch/Chinese mixing ratios, the Chinese-dominant configuration (20% English, 20% Dutch, 60% Chinese) gave the strongest aggregate fine-tuning results, while zero-shot results remained slightly below the official GPT-2 trilingual baseline on average. The authors treat the differing sensitivity patterns of the two West Germanic languages and Chinese as an observation within the tested language triple rather than a general cross-lingual law.", "body_md": "##### Abstract\n\nMultilingual pre-training under severe data con straints can suffer from limited representational capacity and uneven tokenization across lan guages. Addressing the BabyLM 2026 Multi lingual Track, we study a compact pre-training recipe under the 100M-word budget. Our main contribution is a Calibrated Regex-Guided Byte-Level BPE (Regex-BBPE) tokenizer that combines script-aware pre-tokenization with byte-premium-adjusted tokenizer training in a 16K vocabulary. We pair this tokenizer with the previously proposed Attention Residuals mechanism and the Muon optimizer, and eval uate three English/Dutch/Chinese mixing ra tios. Among the tested mixtures, the Chinese dominant configuration (20% English, 20% Dutch, 60% Chinese) gives the strongest ag gregate fine-tuning results. Zero-shot results are mixed and remain slightly below the offi cial GPT-2 trilingual baseline on average, while fine-tuning improves the language-level aver ages by 8.55, 9.55, and 10.09 accuracy points for English, Dutch, and Chinese, respectively. The mixing-ratio experiments also show differ ent sensitivity patterns for the two West Ger manic languages and Chinese; we treat this as an observation within the tested language triple rather than a general cross-lingual law.\n- Anthology ID:\n- 2026.babylm-main.32\n- Volume:\n- [Proceedings of the Second BabyLM Workshop](https://aclanthology.org/volumes/2026.babylm-main/)\n- Month:\n- October\n- Year:\n- 2026\n- Address:\n- Budapest, Hungary\n- Editors:\n- [Leshem Choshen](https://aclanthology.org/people/leshem-choshen/) ,[Ryan Cotterell](https://aclanthology.org/people/ryan-cotterell/) ,[Mustafa Omer Gul](https://aclanthology.org/people/mustafa-omer-gul/) ,[Jaap Jumelet](https://aclanthology.org/people/jaap-jumelet/unverified/) ,[Tal Linzen](https://aclanthology.org/people/tal-linzen/) ,[Aaron Mueller](https://aclanthology.org/people/aaron-mueller/) ,[Suchir Salhan](https://aclanthology.org/people/suchir-salhan/) ,[Raj Sanjay Shah](https://aclanthology.org/people/raj-sanjay-shah/) ,[Alex Warstadt](https://aclanthology.org/people/alex-warstadt/) ,[Ethan Gotlieb Wilcox](https://aclanthology.org/people/ethan-wilcox/)\n- Venues:\n- [BabyLM](https://aclanthology.org/venues/babylm/) |[WS](https://aclanthology.org/venues/ws/)\n- SIG:\n- Publisher:\n- Association for Computational Linguistics\n- Note:\n- Pages:\n- 403–412\n- Language:\n- URL:\n- [https://aclanthology.org/2026.babylm-main.32/](https://aclanthology.org/2026.babylm-main.32/)\n- DOI:\n- Cite (ACL):\n- Yihan Liu, Tongjing Zhang, and Pierre Nugues. 2026. [Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals](https://aclanthology.org/2026.babylm-main.32/) . In*Proceedings of the Second BabyLM Workshop* , pages 403–412, Budapest, Hungary. Association for Computational Linguistics.\n- Cite (Informal):\n- [Compact Multilingual Pre-training under the BabyLM 100M-Word Budget with Regex-Guided Tokenization and Attention Residuals](https://aclanthology.org/2026.babylm-main.32/) (Liu et al., BabyLM 2026)\n- PDF:\n- [https://aclanthology.org/2026.babylm-main.32.pdf](https://aclanthology.org/2026.babylm-main.32.pdf)", "url": "https://wpnews.pro/news/compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex", "canonical_source": "https://aclanthology.org/2026.babylm-main.32/", "published_at": "2026-10-24 00:00:00+00:00", "updated_at": "2026-10-25 21:41:32.749670+00:00", "lang": "en", "topics": ["natural-language-processing", "large-language-models", "ai-research", "machine-learning"], "entities": ["Yihan Liu", "Tongjing Zhang", "Pierre Nugues", "BabyLM Workshop", "Association for Computational Linguistics", "GPT-2", "Muon optimizer", "Attention Residuals"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex", "markdown": "https://wpnews.pro/news/compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex.md", "text": "https://wpnews.pro/news/compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex.txt", "jsonld": "https://wpnews.pro/news/compact-multilingual-pre-training-under-the-babylm-100m-word-budget-with-regex.jsonld"}}