Abstract
Localization quality assurance (LQA) is a critical component of game development, where manual review of large volumes of translated text is time-consuming and costly. Recent advances in large language models (LLMs) suggest strong potential for automated LQA, yet their effectiveness across different models, target languages, and game domains remains insufficiently understood. We present a comprehensive benchmark evaluating eight LLMs, including both closed-source and open-weight models, on English-to-six-language gaming LQA tasks across two game genres. Our dataset comprises 96 evaluation settings with a total of 48,000 translation samples. The results show that Claude Sonnet 4 achieves the best overall performance (F1 = 0.766), followed by Qwen-2.5-72B (F1 = 0.711) and Gemini 2.0 Flash (F1 = 0.691). We observe that (1) the target language does not significantly affect model performance (p = 0.285), (2) models achieve their most consistent performance on French, while Japanese is the most challenging target language, and (3) game genre (RPG vs. strategy) has minimal impact on accuracy. While closed-source models achieve the highest overall performance, open-weight alternatives such as Qwen-2.5-72B provide competitive quality at substantially lower cost. These findings provide practical guidance for deploying LLM-based LQA systems in production game localization workflows.- Anthology ID:
- 2026.amta-research.12
- Volume:
[Proceedings of the 17th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)](/volumes/2026.amta-research/)- Month:
- August
- Year:
- 2026
- Address:
- Québec City, Canada
- Editors:
[Eleftheria Briakou](/people/eleftheria-briakou/unverified/),[Jeremy Gwinnup](/people/jeremy-gwinnup/),[Shivali Goel](/people/shivali-goel/unverified/)- Venue:
[AMTA](/venues/amta/)- SIG:
- Publisher:
- Association for Machine Translation in the Americas
- Note:
- Pages:
- 186–201
- Language:
- URL:
[https://aclanthology.org/2026.amta-research.12/](https://aclanthology.org/2026.amta-research.12/)- DOI:
- Cite (ACL):
- Mao Tian and Na Wu. 2026. Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis. InProceedings of the 17th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 186–201, Québec City, Canada. Association for Machine Translation in the Americas. - Cite (Informal):
[Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis](https://aclanthology.org/2026.amta-research.12/)(Tian & Wu, AMTA 2026)- PDF:
[https://aclanthology.org/2026.amta-research.12.pdf](https://aclanthology.org/2026.amta-research.12.pdf)