Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali

A standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets found that strong English summarization performance does not transfer to Bengali, according to the arXiv paper 2610.19003v1. GPT-5.6 Luna led several English lexical metrics but lost that advantage in Bengali, where LLaMA 3.3 70B achieved the highest ROUGE and BLEU scores and Gemini 2.5 Flash obtained the highest observed Bengali factuality score. The study also found that different evaluation methods produced different model rankings, highlighting the limits of relying on a single metric for multilingual summarization assessment.

arXiv:2610.19003v1 Announce Type: new Abstract: Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.