I run a small local model — Qwen3.5 4B via Ollama — as one deterministic-checked step inside some tooling. This is what still reproduces today, on named versions, with the scripts to check it yourself. Where a result did not survive re-running, that is here too, because that turned out to be the most useful part.
Versions, so you can pin them: Ollama 0.32.6, model qwen3.5:4b
, digest 2a654d98e6fb
, Q4_K_M.
None of this is a new technique. The underlying pattern is Best-of-N with verifiers, which is well established. These are measurements inside a known pattern.
think: true
can hand you an empty answer
With thinking enabled, the chain-of-thought and the final answer share one token budget. When the thinking consumes it, response
comes back empty while thinking
holds the content. Code that reads only response
sees ""
and concludes the call failed. It didn't.
Measured today, one-line question, num_predict: 128
:
think:false response=200 thinking=0
think:true response=0 thinking=483
Fix: for extraction and structured tasks, read both fields, or turn thinking off. Whether it triggers depends on your budget — tighten num_predict
to make it likely.
(Separately: there is evidence that pushing long chain-of-thought onto small models can hurt accuracy — Luo et al., "Through the Valley: Path to Effective Long CoT Training for Small Language Models," EMNLP 2025. That is a training-time effect, not this API field-routing quirk. Different problem; noting it so the two don't get conflated.)
I asked it to critique its own answer. It verbally retracted the wrong fix — "Remove the check proposed above" — and then re-emitted that same wrong fix in code, in the same reply.
One anecdotal example of what Huang et al. measure at scale in "Large Language Models Cannot Self-Correct Reasoning Yet" (arXiv:2310.01798, ICLR 2024): on GSM8K, GPT-4's own accuracy drops 95.5% → 91.5% → 89.0% under intrinsic self-critique.
Self-critique is not a free reliability layer. You need an external judge, not the model grading itself.
Same input, two prompts. Asking for output keys that match the source's naming literally moved accuracy 7/10 → 10/10. A numeric spec — "each row has exactly 3 cells and 4 dashes" — fixed a table that a descriptive spec — "a well-formed separator row" — kept producing broken.
Well-worn prompt-engineering advice. Included only because I have the before/after on the same input.
Here is the part I got wrong for a while, and it is the expensive one.
"Put a deterministic external check on the output" is half an instruction. I had that check. It passed a fabrication.
A merge task, two columns into one, 81 rows. The model invented 5 values that were never in the input. The check returned PASS
, autoVerified: true
, and exit code 0. It was not bypassed and it did not crash. Every invented value was drawn from a closed vocabulary — every value that could appear did appear somewhere in the input — so every token-level assertion the check ran was satisfied. The judge was asked a question that could not tell the two cases apart.
So the rule I would give someone starting this:
A judge that has never been made to fail has not been shown to pass.
Build the check, then feed it material it must reject. If it stays green, you don't have a judge — you have a second thing that agrees with the model. And when the output is a pure function of the input, compare it to the input mechanically before accepting it. An exit code is not evidence.
An earlier note of mine reported that this model drops the space at a Latin↔Arabic boundary — ask for hello مرحبا
, get back helloمرحبا
. I had it in 2 of 2 runs and treated it as a real defect.
Re-running it today: 53 generations. Zero reproductions. Seven prompt shapes on the pinned model, six of those shapes repeated on two other local 4B variants, plus the original echo prompt.
I can't tell you it was fixed, and I'm not going to pretend otherwise — the model file on disk is the same one that was there when I first measured it. Prompt shape doesn't explain it either; I tried the transform-style tasks that were closer to the original context and they came back clean too.
What I can tell you is that a result I would have published as a measured defect was, six weeks later, unreproducible — and I only know that because re-running it was one command. Both scripts are in the repo; the boundary one is currently a null result, and it stays in, because a null result you can run is worth more than a claim you can't.
The check itself is four lines and still worth having if you ship bilingual text — English-centric output assertions never look at the script boundary, so nothing else in your test suite is watching that seam.
A small model doesn't get better when you ask it to think more. It gets better when you reduce what it has to decide, put a deterministic external judge on its output — and then prove the judge can fail.
That framing isn't mine. DSPy's BestOfN
with a custom reward_fn
reproduces most of this shape; CodeT and AlphaCode filter samples by executing tests; Guardrails' on_fail=reask
and Instructor drive retry-on-failure. Even graduated exit codes for agent steps are the Nagios convention (0
OK / 1
WARNING / 2
CRITICAL), about 25 years old.
Reproduction scripts, with versions pinned: https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro