# Small local models: what actually holds up when you re-run the measurement > Source: > Published: 2026-08-21 18:18:08+00:00 I run a small local model — **Qwen3.5 4B via Ollama** — as one deterministic-checked step inside some tooling. This is what still reproduces today, on named versions, with the scripts to check it yourself. Where a result did *not* survive re-running, that is here too, because that turned out to be the most useful part. Versions, so you can pin them: **Ollama 0.32.6**, model `qwen3.5:4b` , digest `2a654d98e6fb` , Q4_K_M. None of this is a new technique. The underlying pattern is **Best-of-N with verifiers**, which is well established. These are measurements inside a known pattern. `think: true` can hand you an empty answer With thinking enabled, the chain-of-thought and the final answer share one token budget. When the thinking consumes it, `response` comes back empty while `thinking` holds the content. Code that reads only `response` sees `""` and concludes the call failed. It didn't. Measured today, one-line question, `num_predict: 128` : ``` think:false response=200 thinking=0 think:true response=0 thinking=483 ``` Fix: for extraction and structured tasks, read both fields, or turn thinking off. Whether it triggers depends on your budget — tighten `num_predict` to make it likely. (Separately: there is evidence that pushing long chain-of-thought onto small models can *hurt* accuracy — Luo et al., *"Through the Valley: Path to Effective Long CoT Training for Small Language Models,"* EMNLP 2025. That is a training-time effect, not this API field-routing quirk. Different problem; noting it so the two don't get conflated.) I asked it to critique its own answer. It verbally retracted the wrong fix — "Remove the check proposed above" — and then re-emitted that same wrong fix in code, in the same reply. One anecdotal example of what **Huang et al.** measure at scale in *"Large Language Models Cannot Self-Correct Reasoning Yet"* (arXiv:2310.01798, ICLR 2024): on GSM8K, GPT-4's own accuracy drops **95.5% → 91.5% → 89.0%** under intrinsic self-critique. Self-critique is not a free reliability layer. You need an external judge, not the model grading itself. Same input, two prompts. Asking for output keys that match the source's naming *literally* moved accuracy **7/10 → 10/10**. A numeric spec — "each row has exactly 3 cells and 4 dashes" — fixed a table that a descriptive spec — "a well-formed separator row" — kept producing broken. Well-worn prompt-engineering advice. Included only because I have the before/after on the same input. Here is the part I got wrong for a while, and it is the expensive one. "Put a deterministic external check on the output" is half an instruction. I had that check. It passed a fabrication. A merge task, two columns into one, 81 rows. The model invented 5 values that were never in the input. The check returned `PASS` , `autoVerified: true` , and **exit code 0**. It was not bypassed and it did not crash. Every invented value was drawn from a closed vocabulary — every value that *could* appear did appear somewhere in the input — so every token-level assertion the check ran was satisfied. The judge was asked a question that could not tell the two cases apart. So the rule I would give someone starting this: A judge that has never been made to fail has not been shown to pass. Build the check, then feed it material it *must* reject. If it stays green, you don't have a judge — you have a second thing that agrees with the model. And when the output is a pure function of the input, compare it *to the input* mechanically before accepting it. An exit code is not evidence. An earlier note of mine reported that this model drops the space at a Latin↔Arabic boundary — ask for `hello مرحبا` , get back `helloمرحبا` . I had it in 2 of 2 runs and treated it as a real defect. Re-running it today: **53 generations. Zero reproductions.** Seven prompt shapes on the pinned model, six of those shapes repeated on two other local 4B variants, plus the original echo prompt. I can't tell you it was fixed, and I'm not going to pretend otherwise — the model file on disk is the same one that was there when I first measured it. Prompt shape doesn't explain it either; I tried the transform-style tasks that were closer to the original context and they came back clean too. What I can tell you is that a result I would have published as a measured defect was, six weeks later, unreproducible — and I only know that because re-running it was one command. Both scripts are in the repo; the boundary one is currently a null result, and it stays in, because a null result you can run is worth more than a claim you can't. The check itself is four lines and still worth having if you ship bilingual text — English-centric output assertions never look at the script boundary, so nothing else in your test suite is watching that seam. A small model doesn't get better when you ask it to think more. It gets better when you **reduce what it has to decide**, put a **deterministic external judge** on its output — and then **prove the judge can fail**. That framing isn't mine. DSPy's `BestOfN` with a custom `reward_fn` reproduces most of this shape; CodeT and AlphaCode filter samples by executing tests; Guardrails' `on_fail=reask` and Instructor drive retry-on-failure. Even graduated exit codes for agent steps are the **Nagios** convention (`0` OK / `1` WARNING / `2` CRITICAL), about 25 years old. Reproduction scripts, with versions pinned: [https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro](https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro)