Large Language Models and their Awareness of Mechanics and Spatial Geometry

A new benchmark called MecEng, introduced in an arXiv paper (2608.14615v1), evaluates large language models on mechanical engineering tasks, finding that the best open-weight model achieves an 86.0% success rate on rigid-body tasks compared to 91.4% for the strongest proprietary model. The benchmark includes 84 tasks across three difficulty levels and tests 32 open-weight and two proprietary LLMs, revealing that flexible multibody tasks remain considerably harder.

arXiv:2608.14615v1 Announce Type: new Abstract: Large Language Models LLMs perform well on established code-generation and mathematical-reasoning benchmarks, but their capabilities in mechanics and spatial geometry, here denoted as mechanical engineering awareness, has not been quantified systematically. We present MecEng, a fully automated benchmark that evaluates LLMs on the creation of multibody simulation models from parameterized textual descriptions. The benchmark comprises 84 generic tasks on three difficulty levels, ranging from rigid-body systems with joints and contact to flexible multibody systems that require exact 3D geometry generation, tetrahedral finite-element meshing, and Hurty-Craig-Bampton model order reduction of machine parts. A dedicated pipeline with LLMs generates simulation-ready geometry from text using Netgen, and builds multibody system models for the code Exudyn, which are then verified against expert ground truth on several levels: system-graph isomorphism including graph node annotations, numerical solutions, and part-specific measures such as mass, geometry, and eigenfrequencies. In total, 32 open-weight and two proprietary LLMs are evaluated. On rigid-body tasks, the best open-weight model obtains an overall success rate of 86.0%, compared to 91.4% for the strongest proprietary model, while flexible multibody tasks remain considerably harder. Additional studies quantify the influence of sampling temperature, reasoning, prompt design, model size, and LLM-release date. The results indicate rapidly improving, but still error-prone, mechanical engineering awareness of current LLMs.