Right, But It Wouldn't Stop: Turning Thinking Off in the Agent Loop

A benchmark harness called compare-models found that Qwen3.8-27B with thinking enabled scored a median 32/32 on a frozen 32-case Go PNG chunk decoder task but finished cleanly in 0 of 5 runs, hitting the 90-minute cap, while the same model with thinking disabled finished 3 of 5 runs in about 25 minutes at a median 30/32. In a separate comparison, the dense Qwen3.8-27B passed 32/32 anchors in roughly 20 turns and about 330K tokens, while the larger Qwen3-Next-80B-A3B MoE hit the agent's 100-turn cap at 11/32 anchors and roughly 9.85M tokens. The author, who built compare-models to freeze the agent, platform, and task so only the model varies, describes "correct, but won't stop" as a real failure mode for autonomous agent loops and cautions the results are small-N on one task, one agent, and one AMD Strix Halo box running Lemonade Server on llama.cpp.

Here’s a result that surprised me: the reasoning model wrote more correct code than the plain one — and still lost. It nailed all 32 test cases, then sat there and refused to declare itself done. Burned the whole time budget re-checking work that was already right. “Correct, but won’t stop” turns out to be a real failure mode, and it’s exactly the wrong one for an autonomous agent loop. So anyway. I’ve been building a little family of benchmark harnesses. compare-agents https://github.com/gherlein/compare-agents pins the model and varies the agent. compare-platform https://github.com/gherlein/compare-platform pins the agent and varies the inference platform. This newest one — compare-models https://github.com/gherlein/compare-models — inverts both: freeze the agent, freeze the platform, freeze the task, and let the model be the only thing that moves. Then you can actually say something honest about the model, because everything else is nailed down. The task is frozen too: implement a Go PNG chunk decoder that has to pass a 32-case suite the model never gets to see. The platform is Lemonade Server https://github.com/lemonade-sdk/lemonade running llama.cpp on an AMD Strix Halo box. The agent is one I call hax , run non-interactively and isolated per trial. Correctness — did it pass the frozen anchor cases — is the primary metric. Everything else time, tokens, turns explains a result but never reorders the ranking. A fast model that writes wrong code loses to a slow one that writes correct code. Full method’s in VISION.md https://github.com/gherlein/compare-models/blob/main/VISION.md , and the whole write-up with the numbers is in FINDINGS.md https://github.com/gherlein/compare-models/blob/main/docs/FINDINGS.md . Standard disclaimer, same as on the repo: this works for me, that’s the entire guarantee. Small N. One task, one agent, one box. Don’t @ me — but do fork it. The practical winner is the boring one Before the fun result, the useful one. I ran Qwen3.8-27B https://blog.herlein.com/post/qwen38-27b-gx10-setup/ — the plain dense 27B — against the biggest MoE I could fit, Qwen3-Next-80B-A3B. Here’s the thing nobody selling you on “bigger is better” wants to hear: | model | anchor rate | finished cleanly | turns | tokens | |---|---|---|---|---| | Qwen3.8-27B dense | 32/32 | ✅ yes | ~20 | ~330K | | Qwen3-Next-80B-A3B | 11/32 | ❌ no | 100 cap | ~9.85M | The dense 27B solved it perfectly, twice, in about twenty turns each, and stopped . The 80B MoE ran the agent’s full 100-turn budget, burned nearly ten million tokens — two orders of magnitude more — and finished at 11 out of 32. It didn’t converge. It thrashed. Now, that’s N=1 on the 80B, so I’m not going to hang a law of physics on it. But it lines up with something I keep coming back to: on my own box, tokens are NRE, not COGS https://blog.herlein.com/post/tokens-are-nre-not-cogs/ — you pay for them once, in your own electricity, not per-call forever. And a model that spends 9.85M of them to be wrong is expensive no matter how you account for it. The boring dense model was the one I’d actually reach for. Bigger was not better here. Not even close. The sharp result: thinking on vs. off Qwen3 is a hybrid-reasoning model, which means you can toggle its thinking with a request flag and change nothing else . Same model. Same quant. Same sampling. Same task. Same agent. Reasoning is the only variable. I ran it five times each way — N=5, which is small but enough to trust a direction. | Qwen3.8-27B | anchor median | finished rate | wall median | output tokens | |---|---|---|---|---| | thinking OFF | 30/32 | 3/5 60% | ~25 min | ~25K | | thinking ON | 32/32 | 0/5 0% | 90 min the cap | ~75K | Look at those two columns and sit with them for a second. Thinking ON is more correct when it produces code — median 32/32 versus 30/32. The reasoning genuinely helps the answer . But three of the five thinking-ON runs that hit a perfect 32/32 timed out anyway . They wrote correct code and then never declared victory. The one thinking-ON run that actually stopped, stopped at 21/32 — below threshold. So: right, but wouldn’t stop. Thinking OFF was messier per-run — it swings from a clean 6-minute win to a 40-minute thrash — but it’s the only setting that reached a clean, correct, self-terminated result with any regularity. Three out of five. And about 3x cheaper in tokens and time. BOTH can be true here, and this is the part I want engineers to internalize: thinking’s payoff lands on the single-pass answer, not on driving a tool loop. Reasoning helps you solve the hard problem. It actively hurts the agent’s ability to know when it’s finished. That’s the “deliberate single pass for design vs. fast iteration for execution” split I keep waving my hands about — except now I’ve got numbers on it. So for an autonomous coding loop on this box, the recommendation is blunt: turn thinking off. Save the reasoning model for the design and planning phase, where a slow deliberate single pass is exactly what you want. Don’t put it on the execution treadmill. The caveat that ate the whole experiment Here’s where I have to be honest about what I’m actually measuring, because it ties straight back to harness engineering https://blog.herlein.com/post/harness-engineering-has-a-name/ . Every number above is “this model, as driven by hax , on this box.” That qualifier is not throat-clearing. An earlier run used a different agent — omp — and it failed structurally . omp auto-picked its text dialect instead of the platform’s native tools API, sent no tools , and the models just free-formed prose. They completed turns with zero tool calls . Never even opened REQUIREMENTS.md . Same models, same box, and the whole benchmark collapsed — not because the models were bad, but because the agent and the platform didn’t shake hands. I proved the platform itself does native tool-calling with a raw /v1/chat/completions probe, swapped in hax which is native-tools-only , and everything worked. Same models. Opposite outcome. The choice of agent decided whether the models could be measured at all. The tooling is the new model. I keep saying it because the data keeps saying it. One more exclusion in the same spirit: I had to drop Qwen3-Coder-30B entirely. Not a coding verdict — a serving bug. That model emits tool calls in a <function=… XML format, and this particular llama.cpp build doesn’t parse it into an OpenAI tool calls field. So the server never emits a tool call, no agent can drive it, and its actual coding ability stays unmeasured here. Confirmed with a raw API probe before I pulled it. The kind of thing that looks like “the model is dumb” and is really “two pieces of your stack disagree about a wire format.” What I got wrong at N=1 My very first thinking-off run was a clean 6-minute, 14-turn, 30/32. My first thinking-on run thrashed. If I’d stopped there — N=1 — I’d have told you thinking-off was a 15x win and moved on smug. N=5 corrected me. The direction held off finishes, on doesn’t , but the magnitude was badly overstated by that lucky first draw. Thinking-off is high-variance too — it has its own 100-turn thrashes in the five. This is the whole reason I build harnesses instead of eyeballing one run and tweeting about it. One trial is an anecdote wearing a lab coat. Conclusion For an agentic coding loop on a local Strix Halo box, driven by hax: - Run the dense Qwen3.8-27B with thinking OFF. Correct, convergent, cheap, and it stops . - Don’t reach for the big MoE for autonomous execution — at least not this one, on this stack. It thrashed instead of converging. - Save the reasoning model for design and planning , where a deliberate single pass earns its keep. Reasoning helps the answer and wrecks the loop. - And remember every number is conditional on the agent. Swap the harness and you can erase the whole result. It’s early — small N, one task, one platform. Next I want to run the rest of the configured Qwen3 set the 8B through the 35B-A3B pair to find out whether smaller MoEs also refuse to converge, or whether that’s an 80B-specific problem. And I want to re-run the whole thing under a second agent, because I already know that number will move. The harness, the frozen task, the 32-case suite, and the full findings are all up at github.com/gherlein/compare-models https://github.com/gherlein/compare-models . Fork it, break it, tell me where my N is too small. That’s the point of publishing it. If this was useful, or you think I’m measuring the wrong thing, drop me a note on LinkedIn https://www.linkedin.com/in/gherlein/ . And yes — pay it forward.