An Empirical Study on Reasoning and Generalization in Large Language Models

A study by Lu Xiang, Yang Zhao, Yaping Zhang, Zixuan Ren, Xingquan Zhang, Zhiyuan Chen and Chengqing Zong, published in Computational Linguistics Volume 52, Issue 3 (pages 841–896, September 2026), found that reasoning-focused LLMs such as DeepSeek-R1 and GPT-o1 significantly outperform non-reasoning models on logic-intensive tasks like mathematics and physics, while non-reasoning models remain competitive on factual or observational problems such as biology and geography. The study compared the models across mathematical and general reasoning benchmarks and generalization over 30 sub-disciplines in six scientific domains, and validated a multi-level LLM-as-a-Judge evaluation framework against human experts. DeepSeek-R1 produced more elaborate reasoning processes that correlated with better performance on harder tasks despite redundancy, while GPT-o1 was more concise and clearer on simpler problems but struggled with complex scenarios; distilled reasoning models consistently outperformed non-reasoning counterparts across all domains, indicating reasoning capabilities can be partially preserved through distillation.

Abstract Recent reasoning-focused large language models LLMs e.g., GPT-o1 and DeepSeek-R1 combine chain-of-thought prompting with reinforcement learning to generate explicit, multi-step derivations. However, their practical advantages over standard instruction-tuned models remain underexplored. In this article, we present a comprehensive empirical study to address these gaps. We compare reasoning-focused and non-reasoning LLMs across several critical dimensions: 1 their performance on mathematical and general reasoning benchmarks; 2 their generalization capabilities across 30 sub-disciplines spanning six scientific domains; 3 the qualitative and quantitative differences in their underlying solution processes; and 4 the impact of model compression via distillation on core reasoning capabilities. Crucially, to reliably conduct our in-depth process analysis, we address the critical issue of LLM-as-a-Judge reliability by proposing and validating a multi-level evaluation framework against human experts. Our experimental results demonstrate that reasoning-focused models significantly outperform non-reasoning counterparts on logic-intensive tasks such as mathematics and physics, while the latter remain competitive on factual or observational problems e.g., biology, geography . Specifically, our validated process analysis reveals distinct styles: DeepSeek-R1 generates more elaborate reasoning processes, correlating with greater effectiveness on harder tasks, despite potential redundancy. In contrast, GPT-o1 exhibits greater conciseness and frequently presents solutions with superior clarity and efficiency, particularly on simpler problems, while struggling with complex scenarios. Additionally, reasoning capabilities can be partially preserved through distillation, with distilled reasoning models consistently outperforming their non-reasoning counterparts across all domains. These findings provide new insights into when and how explicit reasoning contributes to LLM performance, offering practical guidance for model development and deployment. - Anthology ID: - 2026.cl-3.1 - Volume: - Computational Linguistics, Volume 52, Issue 3 - September 2026 https://aclanthology.org/volumes/2026.cl-3/ - Month: - September - Year: - 2026 - Address: - Cambridge, MA - Venue: - CL https://aclanthology.org/venues/cl/ - SIG: - Publisher: - MIT Press - Note: - Pages: - 841–896 - Language: - URL: - https://aclanthology.org/2026.cl-3.1/ https://aclanthology.org/2026.cl-3.1/ - DOI: - 10.1162/coli.a.619 https://doi.org/10.1162/coli.a.619 - Cite ACL : - Lu Xiang, Yang Zhao, Yaping Zhang, Zixuan Ren, Xingquan Zhang, Zhiyuan Chen, and Chengqing Zong. 2026. An Empirical Study on Reasoning and Generalization in Large Language Models https://aclanthology.org/2026.cl-3.1/ . Computational Linguistics , 52 3 :841–896. - Cite Informal : - An Empirical Study on Reasoning and Generalization in Large Language Models https://aclanthology.org/2026.cl-3.1/ Xiang et al., CL 2026 - PDF: - https://aclanthology.org/2026.cl-3.1.pdf https://aclanthology.org/2026.cl-3.1.pdf