Originally published in Generative AI on Medium. I spent months building a system that wraps a language model in a checking loop so it can prove hard math theorems. Every step gets verified with real math tools, the system remembers what worked, and a second pass tries to break the proof before it counts.
Then I tested it on 300 hard math problems, and the results surprised me: the setup around the model matters as much as the model itself.
The harness is three things:
| Model | One pass | With the harness | Multiplier | Failures rescued |
|---|---|---|---|---|
| Qwen-3.8-27B (local, one RTX 5090) | 12.67% | 26.67% | 2.11x | 16% |
| Meta Muse 1.3 | 22.00% | 46.33% | 2.11x | 31% | | Gemini 3.8 Flash | 33.67% | 55.33% | 1.64x | 33% | | 3-model portfolio | 41.33% | 65.67% | 1.59x | 41% |
The honest headline is the last column: the loop fixed 16–33% of the problems each model got wrong on its own. Flash is the best model and improves least in multiples (1.64x, not 2x) but most in failures rescued.
The detail I'm proudest of: the local 27B catches factor-of-2 and boundary-case errors both big models miss. Five problems only Qwen solved; it beat Muse on 14 problems and Flash on 10. Flash is also 7.8x faster than Muse at the median solve (12.2 vs 95.0 minutes).
Proofs are graded by our own calibrated LLM judge ensemble, running in an isolated context — no human grading, no formal verification. All the eval logs are public so anyone can recount: certified counts reproduce exactly from the judgment files (Qwen 80, Muse 139, Flash 166).
The full article is published in Generative AI on Medium: https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7
The code and all test results are open: https://github.com/dionyziuz/proof_harness — eval logs: https://github.com/dionyziuz/proof_harness/tree/main/data/eval_results
If you try it or spot a flaw in the method, I'd like to hear it — the comments here or an issue on the repo.