Competence-Based Evaluation of Logical Reasoning in Large Language Models

A new research paper proposes competence-based evaluation to measure logical reasoning in large language models, targeting shortcomings that include inconsistencies under linguistic variation. The authors argue that improving LLM reasoning requires accurately measuring the underlying competence rather than relying on surface task performance. The work is framed as a measurement contribution for assessing how well models reason.

Recent advances in large language models LLMs have brought impressive performance across diverse tasks; however, shortcomings remain in logical reasoning, including inconsistencies under linguistic variation. To improve the reasoning capacity of LLMs, we must be able to accurately measure their re