{"slug": "beyond-compilation-evaluating-faithful-natural-language-to-lean-statement", "title": "Beyond Compilation: Evaluating Faithful Natural-Language-to-Lean Statement Formalization", "summary": "A new study on natural-language-to-Lean formalization reveals that compile-rate evaluation overestimates faithfulness, with a tool-augmented agent achieving 89.5% compilation but only 60.5% consensus faithfulness. The research introduces a protocol combining Lean compilation, cross-model semantic judging, and human calibration, finding that elaboration feedback is the largest validity intervention but also increases semantic failures. The findings suggest that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately.", "body_md": "arXiv:2606.31002v1 Announce Type: new\nAbstract: Theorem-proving benchmarks evaluate proof search against fixed formal statements, but natural-language-to-Lean formalization must generate the formal statement itself. In this setting, compilation is only a validity check: a Lean declaration may type-check while omitting hypotheses, changing domains, or expressing a vacuous claim. We study faithful statement formalization as both an evaluation problem and a bottleneck-attribution problem. On a 400-entry graduate-level benchmark spanning real analysis, complex analysis, topology, and algebra, our protocol combines Lean compilation, cross-model semantic judging, and human expert calibration. The resulting picture is different from compile-rate evaluation: a full tool-augmented agent reaches 89.5% compilation but only 60.5% consensus faithfulness, exposing a 29.0-point compile-pass but consensus-unfaithful gap. Targeted human audits support the metric as a conservative decision boundary: across available case-level audits, 96.0% of consensus-positive outputs are human-confirmed faithful, while 82.4% of compile-pass consensus-negative outputs are human-confirmed semantic failures. Under this metric, existing one-shot formalizer models and prover-oriented Lean models remain low, suggesting that formal validity, proof-oriented Lean competence, and faithful statement generation should be reported separately. We then use a full $2^3$ factorial design to decompose three recurring interventions in formalization pipelines: parametric expert drafting, Mathlib/context search, and Lean elaboration feedback. Elaboration feedback is the largest validity intervention, but it also exposes a larger compile-pass semantic-failure bucket; search mainly improves grounding and selectivity; and fine-tuned drafting is largely substitutable in this tool stack once feedback and grounding are available.", "url": "https://wpnews.pro/news/beyond-compilation-evaluating-faithful-natural-language-to-lean-statement", "canonical_source": "https://arxiv.org/abs/2606.31002", "published_at": "2026-07-01 04:00:00+00:00", "updated_at": "2026-07-01 04:25:23.955225+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "natural-language-processing", "ai-tools", "ai-agents"], "entities": ["Lean", "Mathlib", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/beyond-compilation-evaluating-faithful-natural-language-to-lean-statement", "markdown": "https://wpnews.pro/news/beyond-compilation-evaluating-faithful-natural-language-to-lean-statement.md", "text": "https://wpnews.pro/news/beyond-compilation-evaluating-faithful-natural-language-to-lean-statement.txt", "jsonld": "https://wpnews.pro/news/beyond-compilation-evaluating-faithful-natural-language-to-lean-statement.jsonld"}}