Cost-Effective Automated Judging of Natural-Language Mathematical Proofs

A study on arXiv (2608.00004v1) found that three cheap open-weight models — GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B — agree with human pass/fail decisions on natural-language mathematical proof grading at rates statistically indistinguishable from frontier models Claude Opus 4.7 and Gemini 3.1 Pro, at up to 100x lower cost. On a 200-instance validation sample of IMO-GradingBench, the cheap judges matched frontier performance, and on the full 1000-instance benchmark, unanimous agreement (all-three-pass) achieved the highest pass-agreement and precision, though the rule was identified post-hoc and needs replication.

arXiv:2608.00004v1 Announce Type: new Abstract: Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier LLM judges are expensive. We ask whether cheap open-weight models can serve as reliable judges given a candidate proof, a ground-truth proof, and a human-grading rubric. On a 200-instance validation sample of IMO-GradingBench, three cheap judges GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B agree with human pass/fail decisions at rates statistically indistinguishable from Claude Opus 4.7 and Gemini 3.1 Pro, at up to $100\times$ lower cost. We had expected a majority vote of the three to be the best budget option; it matched the frontier but did not improve on its strongest member. Extending to the full 1000-instance benchmark and exploring consensus rules, we found that requiring unanimous agreement all-three-pass reaches the highest pass-agreement and precision and, on four replicate runs, the smallest run-to-run spread. The headline finding is that cheap judges are competitive with the frontier at one to two orders of magnitude lower cost; as a deployable default we recommend all-three-pass, with the caveat that this rule was identified post-hoc and warrants independent replication.