{"slug": "cost-effective-automated-judging-of-natural-language-mathematical-proofs", "title": "Cost-Effective Automated Judging of Natural-Language Mathematical Proofs", "summary": "A study on arXiv (2608.00004v1) found that three cheap open-weight models — GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B — agree with human pass/fail decisions on natural-language mathematical proof grading at rates statistically indistinguishable from frontier models Claude Opus 4.7 and Gemini 3.1 Pro, at up to 100x lower cost. On a 200-instance validation sample of IMO-GradingBench, the cheap judges matched frontier performance, and on the full 1000-instance benchmark, unanimous agreement (all-three-pass) achieved the highest pass-agreement and precision, though the rule was identified post-hoc and needs replication.", "body_md": "arXiv:2608.00004v1 Announce Type: new\nAbstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive. We ask whether cheap open-weight models can serve as reliable judges given a candidate proof, a ground-truth proof, and a human-grading rubric. On a 200-instance validation sample of IMO-GradingBench, three cheap judges (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) agree with human pass/fail decisions at rates statistically indistinguishable from Claude Opus 4.7 and Gemini 3.1 Pro, at up to $100\\times$ lower cost. We had expected a majority vote of the three to be the best budget option; it matched the frontier but did not improve on its strongest member. Extending to the full 1000-instance benchmark and exploring consensus rules, we found that requiring unanimous agreement (all-three-pass) reaches the highest pass-agreement and precision and, on four replicate runs, the smallest run-to-run spread. The headline finding is that cheap judges are competitive with the frontier at one to two orders of magnitude lower cost; as a deployable default we recommend all-three-pass, with the caveat that this rule was identified post-hoc and warrants independent replication.", "url": "https://wpnews.pro/news/cost-effective-automated-judging-of-natural-language-mathematical-proofs", "canonical_source": "https://arxiv.org/abs/2608.00004", "published_at": "2026-08-04 04:00:00+00:00", "updated_at": "2026-08-04 04:35:12.252117+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research"], "entities": ["arXiv", "IMO-GradingBench", "GPT-OSS 120B", "DeepSeek-V4 Flash", "Gemma-4 31B", "Claude Opus 4.7", "Gemini 3.1 Pro"], "alternates": {"html": "https://wpnews.pro/news/cost-effective-automated-judging-of-natural-language-mathematical-proofs", "markdown": "https://wpnews.pro/news/cost-effective-automated-judging-of-natural-language-mathematical-proofs.md", "text": "https://wpnews.pro/news/cost-effective-automated-judging-of-natural-language-mathematical-proofs.txt", "jsonld": "https://wpnews.pro/news/cost-effective-automated-judging-of-natural-language-mathematical-proofs.jsonld"}}