{"slug": "nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry", "title": "NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry", "summary": "A new arXiv paper (2608.28481v2) introduces NL2AGBench, a benchmark that evaluates large language models on translating English geometry problems into AlphaGeometry-compatible formal representations, using execution-based verification inside AlphaGeometry rather than textual similarity. Across ten state-of-the-art open- and closed-source LLMs at multiple parameter scales, leading closed-source models achieved executable translation rates above 80%, while even the largest open-source models struggled to consistently preserve geometric constraints and produce valid formalizations. The authors also present an error taxonomy separating syntax from logic errors and report that few-shot prompting, fine-tuning and human-guided hinting yield measurable improvements across multiple model families.", "body_md": "arXiv:2608.28481v2 Announce Type: replace \nAbstract: Recent advances in large language models (LLMs) have demonstrated strong capabilities in natural language understanding and mathematical reasoning. However, their ability to translate informal mathematical problems into formal representations remains underexplored. This limitation is particularly important for neuro-symbolic geometry systems such as AlphaGeometry, whose theorem-proving engine requires inputs in a specialized domain-specific language (DSL). Although AlphaGeometry achieves near-IMO gold-medalist performance, manually converting natural-language problems into its formal syntax remains a significant usability bottleneck. To address this challenge, we introduce the Natural Language to AlphaGeometry Benchmark (NL2AGBench), which evaluates LLMs in translating English geometry problems into AlphaGeometry-compatible formal representations. NL2AGBench uses execution-based verification within AlphaGeometry to assess translation quality rather than relying solely on textual similarity. We evaluate ten state-of-the-art open- and closed-source LLMs across multiple parameter scales and analyze executable translation accuracy, syntactic correctness, and error characteristics. Our experiments reveal a substantial performance gap between closed- and open-source models: leading closed-source models achieve executable translation rates above 80%, while even the largest open-source models struggle to consistently preserve geometric constraints and produce valid formalizations. We introduce an error taxonomy distinguishing syntax and logic errors and investigate mitigation strategies, including few-shot prompting, fine-tuning, and human-guided hinting, which yield measurable improvements across multiple model families.", "url": "https://wpnews.pro/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry", "canonical_source": "https://www.machinebrief.com/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageom-k14p", "published_at": "2026-11-20 05:00:00+00:00", "updated_at": "2026-11-20 05:44:38.724619+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "machine-learning", "artificial-intelligence"], "entities": ["NL2AGBench", "AlphaGeometry", "arXiv"], "also_reported_by": [], "takeaway": "Leading closed-source LLMs exceed 80% executable translation accuracy on NL2AGBench while even the largest open-source models fail to consistently preserve geometric constraints; few-shot prompting, fine-tuning and human-guided hinting measurably improve results.", "recommendation": {"advice": "Verify auto-formalization by executing translations in the target DSL rather than scoring textual similarity, and add few-shot prompting or fine-tuning when open-source models drop geometric constraints.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry", "markdown": "https://wpnews.pro/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry.md", "text": "https://wpnews.pro/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry.txt", "jsonld": "https://wpnews.pro/news/nl2agbench-benchmarking-llm-auto-formalization-for-alphageometry.jsonld"}}