{"slug": "scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning", "title": "Scale the Model, Scale the Harness: Unlocking a 2 Leap in Mathematical Reasoning", "summary": "A developer built an open-source proof-checking harness that wraps language models in a verification loop, re-checking each step with math tools and a second adversarial pass, and tested it on 300 hard math problems. Across Qwen-3.8-27B, Meta Muse 1.3, Gemini 3.8 Flash and a three-model portfolio, the harness roughly doubled accuracy (up to 2.11x) and rescued 16–33% of problems each model failed on its own, with the local 27B catching factor-of-2 and boundary-case errors both larger models missed. Proofs were graded by a calibrated LLM judge ensemble rather than human or formal verification, and the code and eval logs are public.", "body_md": "*Originally published in [Generative AI on Medium](https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7).*\n\nI spent months building a system that wraps a language model in a checking loop so it can prove hard math theorems. Every step gets verified with real math tools, the system remembers what worked, and a second pass tries to break the proof before it counts.\n\nThen I tested it on 300 hard math problems, and the results surprised me: the setup around the model matters as much as the model itself.\n\nThe harness is three things:\n\n| Model | One pass | With the harness | Multiplier | Failures rescued | \n|---|---|---|---|---|\n| Qwen-3.8-27B (local, one RTX 5090) | 12.67% | 26.67% | 2.11x | 16% | \n| Meta Muse 1.3 | 22.00% | 46.33% | 2.11x | 31% | \n| Gemini 3.8 Flash | 33.67% | 55.33% | 1.64x | 33% | \n| 3-model portfolio | 41.33% | 65.67% | 1.59x | 41% | \n\nThe honest headline is the last column: the loop fixed 16–33% of the problems each model got wrong on its own. Flash is the best model and improves least in multiples (1.64x, not 2x) but most in failures rescued.\n\nThe detail I'm proudest of: the local 27B catches factor-of-2 and boundary-case errors both big models miss. Five problems only Qwen solved; it beat Muse on 14 problems and Flash on 10. Flash is also 7.8x faster than Muse at the median solve (12.2 vs 95.0 minutes).\n\nProofs are graded by our own calibrated LLM judge ensemble, running in an isolated context — no human grading, no formal verification. All the eval logs are public so anyone can recount: certified counts reproduce exactly from the judgment files (Qwen 80, Muse 139, Flash 166).\n\nThe full article is published in Generative AI on Medium: [https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7](https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7)\n\nThe code and all test results are open: [https://github.com/dionyziuz/proof_harness](https://github.com/dionyziuz/proof_harness) — eval logs: [https://github.com/dionyziuz/proof_harness/tree/main/data/eval_results](https://github.com/dionyziuz/proof_harness/tree/main/data/eval_results)\n\nIf you try it or spot a flaw in the method, I'd like to hear it — the comments here or an issue on the repo.", "url": "https://wpnews.pro/news/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning", "canonical_source": "https://dev.to/edwin_jongsma_1739c848559/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-1i20", "published_at": "2026-11-19 01:39:16+00:00", "updated_at": "2026-11-19 01:41:46.499050+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-research", "ai-tools", "machine-learning"], "entities": ["Qwen-3.8-27B", "Meta Muse 1.3", "Gemini 3.8 Flash", "Medium", "GitHub"], "also_reported_by": [], "takeaway": "Wrapping models in a verified checking loop roughly doubled math accuracy (1.59x–2.11x) and rescued 16–33% of individually failed problems, with a local 27B uniquely solving 5 problems the larger models missed.", "recommendation": {"advice": "Run a second adversarial verification pass over each model output and log judgments so certified counts can be reproduced from the files.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning", "markdown": "https://wpnews.pro/news/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning.md", "text": "https://wpnews.pro/news/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning.txt", "jsonld": "https://wpnews.pro/news/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning.jsonld"}}