# Right, But It Wouldn't Stop: Turning Thinking Off in the Agent Loop

> Source: <https://blog.herlein.com/post/compare-models-thinking-off/>
> Published: 2026-09-09 08:00:01+00:00

Here’s a result that surprised me: the reasoning model wrote *more* correct code than the plain one — and still lost. It nailed all 32 test cases, then sat there and refused to declare itself done. Burned the whole time budget re-checking work that was already right. “Correct, but won’t stop” turns out to be a real failure mode, and it’s exactly the wrong one for an autonomous agent loop.

So anyway. I’ve been building a little family of benchmark harnesses. [compare-agents](https://github.com/gherlein/compare-agents) pins the model and varies the agent. [compare-platform](https://github.com/gherlein/compare-platform) pins the agent and varies the inference platform. This newest one — [**compare-models**](https://github.com/gherlein/compare-models) — inverts both: freeze the agent, freeze the platform, freeze the task, and let the **model** be the only thing that moves. Then you can actually say something honest about the model, because everything else is nailed down.

The task is frozen too: implement a Go PNG chunk decoder that has to pass a 32-case suite the model never gets to see. The platform is [Lemonade Server](https://github.com/lemonade-sdk/lemonade) running llama.cpp on an AMD Strix Halo box. The agent is one I call `hax`, run non-interactively and isolated per trial. Correctness — did it pass the frozen anchor cases — is the primary metric. Everything else (time, tokens, turns) explains a result but never reorders the ranking. A fast model that writes wrong code loses to a slow one that writes correct code. Full method’s in [VISION.md](https://github.com/gherlein/compare-models/blob/main/VISION.md), and the whole write-up with the numbers is in [FINDINGS.md](https://github.com/gherlein/compare-models/blob/main/docs/FINDINGS.md).

Standard disclaimer, same as on the repo: this works for me, that’s the entire guarantee. Small N. One task, one agent, one box. Don’t @ me — but do fork it.

## The practical winner is the boring one

Before the fun result, the useful one.

I ran [Qwen3.8-27B](https://blog.herlein.com/post/qwen38-27b-gx10-setup/) — the plain **dense** 27B — against the biggest MoE I could fit, Qwen3-Next-80B-A3B. Here’s the thing nobody selling you on “bigger is better” wants to hear:

| model | anchor rate | finished cleanly | turns | tokens | 
|---|---|---|---|---|
| **Qwen3.8-27B (dense)** | **32/32** | ✅ yes | ~20 | ~330K | 
| Qwen3-Next-80B-A3B | 11/32 | ❌ no | **100 (cap)** | **~9.85M** | 

The dense 27B solved it perfectly, twice, in about twenty turns each, and *stopped*. The 80B MoE ran the agent’s full 100-turn budget, burned nearly **ten million tokens** — two orders of magnitude more — and finished at 11 out of 32. It didn’t converge. It thrashed.

Now, that’s N=1 on the 80B, so I’m not going to hang a law of physics on it. But it lines up with something I keep coming back to: on my own box, [tokens are NRE, not COGS](https://blog.herlein.com/post/tokens-are-nre-not-cogs/) — you pay for them once, in your own electricity, not per-call forever. And a model that spends 9.85M of them to be wrong is expensive no matter how you account for it. The boring dense model was the one I’d actually reach for. Bigger was not better here. Not even close.

## The sharp result: thinking on vs. off

Qwen3 is a hybrid-reasoning model, which means you can toggle its thinking with a request flag and change *nothing else*. Same model. Same quant. Same sampling. Same task. Same agent. Reasoning is the only variable. I ran it five times each way — N=5, which is small but enough to trust a direction.

| Qwen3.8-27B | anchor (median) | **finished rate** | wall (median) | output tokens | 
|---|---|---|---|---|
| **thinking OFF** | 30/32 | **3/5 (60%)** | ~25 min | ~25K | 
| **thinking ON** | 32/32 | **0/5 (0%)** | 90 min (the cap) | ~75K | 

Look at those two columns and sit with them for a second.

Thinking ON is **more correct** when it produces code — median 32/32 versus 30/32. The reasoning genuinely helps the *answer*. But three of the five thinking-ON runs that hit a perfect 32/32 **timed out anyway**. They wrote correct code and then never declared victory. The one thinking-ON run that actually stopped, stopped at 21/32 — below threshold. So: right, but wouldn’t stop.

Thinking OFF was messier per-run — it swings from a clean 6-minute win to a 40-minute thrash — but it’s the only setting that reached a clean, correct, self-terminated result with any regularity. Three out of five. And about 3x cheaper in tokens and time.

BOTH can be true here, and this is the part I want engineers to internalize: **thinking’s payoff lands on the single-pass answer, not on driving a tool loop.** Reasoning helps you solve the hard problem. It actively hurts the agent’s ability to know when it’s finished. That’s the “deliberate single pass for design vs. fast iteration for execution” split I keep waving my hands about — except now I’ve got numbers on it.

So for an autonomous coding loop on this box, the recommendation is blunt: **turn thinking off.** Save the reasoning model for the design and planning phase, where a slow deliberate single pass is exactly what you want. Don’t put it on the execution treadmill.

## The caveat that ate the whole experiment

Here’s where I have to be honest about what I’m actually measuring, because it ties straight back to [harness engineering](https://blog.herlein.com/post/harness-engineering-has-a-name/).

Every number above is “this model, *as driven by hax*, on *this* box.” That qualifier is not throat-clearing. An earlier run used a different agent — omp — and it failed *structurally*. omp auto-picked its text dialect instead of the platform’s native tools API, sent no `tools`, and the models just free-formed prose. They completed turns with **zero tool calls**. Never even opened `REQUIREMENTS.md`. Same models, same box, and the whole benchmark collapsed — not because the models were bad, but because the agent and the platform didn’t shake hands.

I proved the platform itself does native tool-calling with a raw `/v1/chat/completions` probe, swapped in `hax` (which is native-tools-only), and everything worked. Same models. Opposite outcome. **The choice of agent decided whether the models could be measured at all.** The tooling is the new model. I keep saying it because the data keeps saying it.

One more exclusion in the same spirit: I had to drop Qwen3-Coder-30B entirely. Not a coding verdict — a *serving* bug. That model emits tool calls in a `<function=…>` XML format, and this particular llama.cpp build doesn’t parse it into an OpenAI `tool_calls` field. So the server never emits a tool call, no agent can drive it, and its actual coding ability stays unmeasured here. Confirmed with a raw API probe before I pulled it. The kind of thing that looks like “the model is dumb” and is really “two pieces of your stack disagree about a wire format.”

## What I got wrong (at N=1)

My very first thinking-off run was a clean 6-minute, 14-turn, 30/32. My first thinking-on run thrashed. If I’d stopped there — N=1 — I’d have told you thinking-off was a 15x win and moved on smug.

N=5 corrected me. The *direction* held (off finishes, on doesn’t), but the *magnitude* was badly overstated by that lucky first draw. Thinking-off is high-variance too — it has its own 100-turn thrashes in the five. This is the whole reason I build harnesses instead of eyeballing one run and tweeting about it. One trial is an anecdote wearing a lab coat.

## Conclusion

For an agentic coding loop on a local Strix Halo box, driven by hax:

- **Run the dense Qwen3.8-27B with thinking OFF.** Correct, convergent, cheap, and it*stops* .
- **Don’t reach for the big MoE for autonomous execution** — at least not this one, on this stack. It thrashed instead of converging.
- **Save the reasoning model for design and planning** , where a deliberate single pass earns its keep. Reasoning helps the answer and wrecks the loop.
- **And remember every number is conditional on the agent.** Swap the harness and you can erase the whole result.

It’s early — small N, one task, one platform. Next I want to run the rest of the configured Qwen3 set (the 8B through the 35B-A3B pair) to find out whether smaller MoEs also refuse to converge, or whether that’s an 80B-specific problem. And I want to re-run the whole thing under a second agent, because I already know that number will move.

The harness, the frozen task, the 32-case suite, and the full findings are all up at [github.com/gherlein/compare-models](https://github.com/gherlein/compare-models). Fork it, break it, tell me where my N is too small. That’s the point of publishing it.

If this was useful, or you think I’m measuring the wrong thing, [drop me a note on LinkedIn](https://www.linkedin.com/in/gherlein/). And yes — pay it forward.