cd /news/large-language-models/scale-the-model-scale-the-harness-un… · home › topics › large-language-models › article
[ARTICLE · art-172247] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=↑ positive

Scale the Model, Scale the Harness: Unlocking a 2 Leap in Mathematical Reasoning

A developer built an open-source proof-checking harness that wraps language models in a verification loop, re-checking each step with math tools and a second adversarial pass, and tested it on 300 hard math problems. Across Qwen-3.8-27B, Meta Muse 1.3, Gemini 3.8 Flash and a three-model portfolio, the harness roughly doubled accuracy (up to 2.11x) and rescued 16–33% of problems each model failed on its own, with the local 27B catching factor-of-2 and boundary-case errors both larger models missed. Proofs were graded by a calibrated LLM judge ensemble rather than human or formal verification, and the code and eval logs are public.

recommendation

Run a second adversarial verification pass over each model output and log judgments so certified counts can be reproduced from the files.

more in playbook: Evals and testing for LLM apps
by read2 min views16 publishedNov 19, 2026

Originally published in Generative AI on Medium. I spent months building a system that wraps a language model in a checking loop so it can prove hard math theorems. Every step gets verified with real math tools, the system remembers what worked, and a second pass tries to break the proof before it counts.

Then I tested it on 300 hard math problems, and the results surprised me: the setup around the model matters as much as the model itself.

The harness is three things:

| Model | One pass | With the harness | Multiplier | Failures rescued |

|---|---|---|---|---|
| Qwen-3.8-27B (local, one RTX 5090) | 12.67% | 26.67% | 2.11x | 16% | 

| Meta Muse 1.3 | 22.00% | 46.33% | 2.11x | 31% | | Gemini 3.8 Flash | 33.67% | 55.33% | 1.64x | 33% | | 3-model portfolio | 41.33% | 65.67% | 1.59x | 41% |

The honest headline is the last column: the loop fixed 16–33% of the problems each model got wrong on its own. Flash is the best model and improves least in multiples (1.64x, not 2x) but most in failures rescued.

The detail I'm proudest of: the local 27B catches factor-of-2 and boundary-case errors both big models miss. Five problems only Qwen solved; it beat Muse on 14 problems and Flash on 10. Flash is also 7.8x faster than Muse at the median solve (12.2 vs 95.0 minutes).

Proofs are graded by our own calibrated LLM judge ensemble, running in an isolated context — no human grading, no formal verification. All the eval logs are public so anyone can recount: certified counts reproduce exactly from the judgment files (Qwen 80, Muse 139, Flash 166).

The full article is published in Generative AI on Medium: https://generativeai.pub/scale-the-model-scale-the-harness-unlocking-a-2-leap-in-mathematical-reasoning-231314ab3ea7

The code and all test results are open: https://github.com/dionyziuz/proof_harness — eval logs: https://github.com/dionyziuz/proof_harness/tree/main/data/eval_results

If you try it or spot a flaw in the method, I'd like to hear it — the comments here or an issue on the repo.

── more in #large-language-models 4 stories · sorted by recency
── more on @qwen-3.8-27b 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/scale-the-model-scal…] indexed:0 read:2min 2026-11-19 · —