A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE

Researchers introduced PARAMΔ, a method that upcycles a dense LLM into a Mixture-of-Experts (MoE) architecture to expand language capabilities without costly alignment. By grafting a MoE-expanded parameter delta onto a CPT-enhanced base model, the approach preserves original abilities while improving performance on new languages. Experiments showed superiority over baselines with similar FLOPs or parameters, demonstrating applicability across different models and post-training deltas.

A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE https://aclanthology.org/2026.acl-long.1238.pdf Hao Zhou /people/hao-zhou/unverified/ , Tianhao Li /people/tianhao-li/ , Zhijun Wang /people/zhijun-wang/unverified/ , Shuaijie She /people/shuaijie-she/ , Linjuan Wu /people/linjuan-wu/unverified/ , Hao-Ran Wei /people/hao-ran-wei/unverified/ , Baosong Yang /people/baosong-yang/unverified/ , Jiajun Chen /people/jiajun-chen/unverified/ , Shujian Huang /people/shujian-huang/ Abstract Expanding Large Language Models LLMs to new languages is a costly endeavor, demanding extensive Continued Pre-Training CPT and data-intensive alignment. While recent data-free merging techniques attempt to bypass alignment by fusing a multilingual CPT-enhanced model with its instruct counterpart, they are plagued by a critical trade-off: mitigating parameter conflicts to preserve original abilities inevitably dilutes new language acquisition, and vice-versa. To resolve this conflict, we introduce , which upcycles a dense model into a Mixture-of-Experts MoE architecture, allocating different experts to different languages. Alignment ability is then transferred by grafting a MoE-expanded parameter delta 𝛥instruct to the CPT-enhanced base model, bypassing the complex alignment phase. Experiments demonstrate ’s superiority even against baselines with similar FLOPs or number of parameters; it improves performance on expanded languages while effectively preserving original capabilities. We further show our approach is highly applicable across different models and Post-training deltas. - Anthology ID: - 2026.acl-long.1238 - Volume: Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers /volumes/2026.acl-long/ - Month: - July - Year: - 2026 - Address: - San Diego, California, United States - Editors: Maria Liakata /people/maria-liakata/ , Viviane P. Moreira /people/viviane-p-moreira/unverified/ , Jiajun Zhang /people/jiajun-zhang/unverified/ , David Jurgens /people/david-jurgens/ - Venue: ACL /venues/acl/ - SIG: - Publisher: - Association for Computational Linguistics - Note: - Pages: - 26888–26904 - Language: - URL: https://aclanthology.org/2026.acl-long.1238/ https://aclanthology.org/2026.acl-long.1238/ - DOI: - Cite ACL : - Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, and Shujian Huang. 2026. A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE https://aclanthology.org/2026.acl-long.1238/ . In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers , pages 26888–26904, San Diego, California, United States. Association for Computational Linguistics. - Cite Informal : A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE https://aclanthology.org/2026.acl-long.1238/ Zhou et al., ACL 2026 - PDF: https://aclanthology.org/2026.acl-long.1238.pdf https://aclanthology.org/2026.acl-long.1238.pdf