arXiv:2608.28481v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have demonstrated strong capabilities in natural language understanding and mathematical reasoning. However, their ability to translate informal mathematical problems into formal representations remains underexplored. This limitation is particularly important for neuro-symbolic geometry systems such as AlphaGeometry, whose theorem-proving engine requires inputs in a specialized domain-specific language (DSL). Although AlphaGeometry achieves near-IMO gold-medalist performance, manually converting natural-language problems into its formal syntax remains a significant usability bottleneck. To address this challenge, we introduce the Natural Language to AlphaGeometry Benchmark (NL2AGBench), which evaluates LLMs in translating English geometry problems into AlphaGeometry-compatible formal representations. NL2AGBench uses execution-based verification within AlphaGeometry to assess translation quality rather than relying solely on textual similarity. We evaluate ten state-of-the-art open- and closed-source LLMs across multiple parameter scales and analyze executable translation accuracy, syntactic correctness, and error characteristics. Our experiments reveal a substantial performance gap between closed- and open-source models: leading closed-source models achieve executable translation rates above 80%, while even the largest open-source models struggle to consistently preserve geometric constraints and produce valid formalizations. We introduce an error taxonomy distinguishing syntax and logic errors and investigate mitigation strategies, including few-shot prompting, fine-tuning, and human-guided hinting, which yield measurable improvements across multiple model families.
NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry
A new arXiv paper (2608.28481v2) introduces NL2AGBench, a benchmark that evaluates large language models on translating English geometry problems into AlphaGeometry-compatible formal representations, using execution-based verification inside AlphaGeometry rather than textual similarity. Across ten state-of-the-art open- and closed-source LLMs at multiple parameter scales, leading closed-source models achieved executable translation rates above 80%, while even the largest open-source models struggled to consistently preserve geometric constraints and produce valid formalizations. The authors also present an error taxonomy separating syntax from logic errors and report that few-shot prompting, fine-tuning and human-guided hinting yield measurable improvements across multiple model families.
Verify auto-formalization by executing translations in the target DSL rather than scoring textual similarity, and add few-shot prompting or fine-tuning when open-source models drop geometric constraints.
more in playbook: Evals and testing for LLM appsRun your AI side-project on zahid.host
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.