A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE

[A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE](https://aclanthology.org/2026.acl-long.1238.pdf)

[Hao Zhou](/people/hao-zhou/unverified/),
[Tianhao Li](/people/tianhao-li/),
[Zhijun Wang](/people/zhijun-wang/unverified/),
[Shuaijie She](/people/shuaijie-she/),
[Linjuan Wu](/people/linjuan-wu/unverified/),
[Hao-Ran Wei](/people/hao-ran-wei/unverified/),
[Baosong Yang](/people/baosong-yang/unverified/),
[Jiajun Chen](/people/jiajun-chen/unverified/),
[Shujian Huang](/people/shujian-huang/)

Abstract

Expanding Large Language Models(LLMs) to new languages is a costly endeavor, demanding extensive Continued Pre-Training(CPT) and data-intensive alignment. While recent data-free merging techniques attempt to bypass alignment by fusing a multilingual CPT-enhanced model with its instruct counterpart, they are plagued by a critical trade-off: mitigating parameter conflicts to preserve original abilities inevitably dilutes new language acquisition, and vice-versa. To resolve this conflict, we introduce , which upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages. Alignment ability is then transferred by grafting a MoE-expanded parameter delta(𝛥instruct) to the CPT-enhanced base model, bypassing the complex alignment phase. Experiments demonstrate ’s superiority even against baselines with similar FLOPs or number of parameters; it improves performance on expanded languages while effectively preserving original capabilities. We further show our approach is highly applicable across different models and Post-training deltas.

- Anthology ID:
- 2026.acl-long.1238
- Volume:
[Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](/volumes/2026.acl-long/)- Month:

• July
• Year:
• 2026
• Address:
• San Diego, California, United States

- Editors:
[Maria Liakata](/people/maria-liakata/),[Viviane P. Moreira](/people/viviane-p-moreira/unverified/),[Jiajun Zhang](/people/jiajun-zhang/unverified/),[David Jurgens](/people/david-jurgens/)- Venue:
[ACL](/venues/acl/)- SIG:
- Publisher:

• Association for Computational Linguistics

- Note:
- Pages:

• 26888–26904

- Language:
- URL:
[https://aclanthology.org/2026.acl-long.1238/](https://aclanthology.org/2026.acl-long.1238/)- DOI:
- Cite (ACL):

• Hao Zhou, Tianhao Li, Zhijun Wang, Shuaijie She, Linjuan Wu, Hao-Ran Wei, Baosong Yang, Jiajun Chen, and Shujian Huang. 2026. A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26888–26904, San Diego, California, United States. Association for Computational Linguistics. - Cite (Informal):

[A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM𝛥 Integration into Upcycled MoE](https://aclanthology.org/2026.acl-long.1238/)(Zhou et al., ACL 2026)- PDF:
[https://aclanthology.org/2026.acl-long.1238.pdf](https://aclanthology.org/2026.acl-long.1238.pdf)