Investigating LLM's Problem Solving Capability -- a Study on Statics Questions

Researchers at arXiv found that large language models (LLMs) perform well on text-only statics problems but their accuracy drops when diagrams are added and multi-step reasoning is required. The performance decline is attributed to difficulties in multi-step reasoning and consistent application of visual information, not image recognition limitations.

arXiv:2606.26103v1 Announce Type: new Abstract: Large Language Models LLMs have rapidly influenced many aspects of society, particularly education, due to their demonstrated ability to complete assignments and examinations across a wide range of subjects. Although prior studies have examined the educational impact of LLMs, much of the existing work relies on public or open problem datasets and lacks topic-specific analysis. In engineering education, especially within mechanical engineering, systematic investigations of LLM performance on specific problem types remain limited. Instead of using traditional methods that directly ask textbook questions to an LLM tool, our study adopts a model distillation process to evaluate LLM capabilities in solving statics problems. By distilling ChatGPT, we extracted 25 text-only statics questions and further constructed two additional datasets by adding diagrams and modifying their numerical values. Experimental results show that while LLMs perform well on text-only statics problems, their accuracy decreases when diagrams are introduced and the problems require multi-step reasoning. Further analysis suggests that this performance drop is not primarily caused by limitations in image recognition, but rather by difficulties in multi-step reasoning and in consistently applying extracted visual information across successive solution stages.