{"slug": "small-local-models-what-actually-holds-up-when-you-re-run-the-measurement", "title": "Small local models: what actually holds up when you re-run the measurement", "summary": "A developer testing Qwen3.5 4B via Ollama found that enabling chain-of-thought can return an empty response while the thinking field holds content, and that self-critique can degrade accuracy, citing research by Huang et al. The developer also discovered that a deterministic check passed fabricated values in a merge task, highlighting that a judge that has never been made to fail has not been shown to pass. Re-running an earlier reported defect about missing spaces at Latin-Arabic boundaries failed to reproduce across 53 generations.", "body_md": "I run a small local model — **Qwen3.5 4B via Ollama** — as one deterministic-checked step inside some tooling. This is what still reproduces today, on named versions, with the scripts to check it yourself. Where a result did *not* survive re-running, that is here too, because that turned out to be the most useful part.\n\nVersions, so you can pin them: **Ollama 0.32.6**, model `qwen3.5:4b`\n\n, digest `2a654d98e6fb`\n\n, Q4_K_M.\n\nNone of this is a new technique. The underlying pattern is **Best-of-N with verifiers**, which is well established. These are measurements inside a known pattern.\n\n`think: true`\n\ncan hand you an empty answer\nWith thinking enabled, the chain-of-thought and the final answer share one token budget. When the thinking consumes it, `response`\n\ncomes back empty while `thinking`\n\nholds the content. Code that reads only `response`\n\nsees `\"\"`\n\nand concludes the call failed. It didn't.\n\nMeasured today, one-line question, `num_predict: 128`\n\n:\n\n```\nthink:false    response=200   thinking=0\nthink:true     response=0     thinking=483\n```\n\nFix: for extraction and structured tasks, read both fields, or turn thinking off. Whether it triggers depends on your budget — tighten `num_predict`\n\nto make it likely.\n\n(Separately: there is evidence that pushing long chain-of-thought onto small models can *hurt* accuracy — Luo et al., *\"Through the Valley: Path to Effective Long CoT Training for Small Language Models,\"* EMNLP 2025. That is a training-time effect, not this API field-routing quirk. Different problem; noting it so the two don't get conflated.)\n\nI asked it to critique its own answer. It verbally retracted the wrong fix — \"Remove the check proposed above\" — and then re-emitted that same wrong fix in code, in the same reply.\n\nOne anecdotal example of what **Huang et al.** measure at scale in *\"Large Language Models Cannot Self-Correct Reasoning Yet\"* (arXiv:2310.01798, ICLR 2024): on GSM8K, GPT-4's own accuracy drops **95.5% → 91.5% → 89.0%** under intrinsic self-critique.\n\nSelf-critique is not a free reliability layer. You need an external judge, not the model grading itself.\n\nSame input, two prompts. Asking for output keys that match the source's naming *literally* moved accuracy **7/10 → 10/10**. A numeric spec — \"each row has exactly 3 cells and 4 dashes\" — fixed a table that a descriptive spec — \"a well-formed separator row\" — kept producing broken.\n\nWell-worn prompt-engineering advice. Included only because I have the before/after on the same input.\n\nHere is the part I got wrong for a while, and it is the expensive one.\n\n\"Put a deterministic external check on the output\" is half an instruction. I had that check. It passed a fabrication.\n\nA merge task, two columns into one, 81 rows. The model invented 5 values that were never in the input. The check returned `PASS`\n\n, `autoVerified: true`\n\n, and **exit code 0**. It was not bypassed and it did not crash. Every invented value was drawn from a closed vocabulary — every value that *could* appear did appear somewhere in the input — so every token-level assertion the check ran was satisfied. The judge was asked a question that could not tell the two cases apart.\n\nSo the rule I would give someone starting this:\n\nA judge that has never been made to fail has not been shown to pass.\n\nBuild the check, then feed it material it *must* reject. If it stays green, you don't have a judge — you have a second thing that agrees with the model. And when the output is a pure function of the input, compare it *to the input* mechanically before accepting it. An exit code is not evidence.\n\nAn earlier note of mine reported that this model drops the space at a Latin↔Arabic boundary — ask for `hello مرحبا`\n\n, get back `helloمرحبا`\n\n. I had it in 2 of 2 runs and treated it as a real defect.\n\nRe-running it today: **53 generations. Zero reproductions.** Seven prompt shapes on the pinned model, six of those shapes repeated on two other local 4B variants, plus the original echo prompt.\n\nI can't tell you it was fixed, and I'm not going to pretend otherwise — the model file on disk is the same one that was there when I first measured it. Prompt shape doesn't explain it either; I tried the transform-style tasks that were closer to the original context and they came back clean too.\n\nWhat I can tell you is that a result I would have published as a measured defect was, six weeks later, unreproducible — and I only know that because re-running it was one command. Both scripts are in the repo; the boundary one is currently a null result, and it stays in, because a null result you can run is worth more than a claim you can't.\n\nThe check itself is four lines and still worth having if you ship bilingual text — English-centric output assertions never look at the script boundary, so nothing else in your test suite is watching that seam.\n\nA small model doesn't get better when you ask it to think more. It gets better when you **reduce what it has to decide**, put a **deterministic external judge** on its output — and then **prove the judge can fail**.\n\nThat framing isn't mine. DSPy's `BestOfN`\n\nwith a custom `reward_fn`\n\nreproduces most of this shape; CodeT and AlphaCode filter samples by executing tests; Guardrails' `on_fail=reask`\n\nand Instructor drive retry-on-failure. Even graduated exit codes for agent steps are the **Nagios** convention (`0`\n\nOK / `1`\n\nWARNING / `2`\n\nCRITICAL), about 25 years old.\n\nReproduction scripts, with versions pinned: [https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro](https://github.com/ahmadyaseen35-coder/local-model-field-notes-repro)", "url": "https://wpnews.pro/news/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement", "canonical_source": "https://dev.to/ahmadammar/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement-5590", "published_at": "2026-08-21 18:18:08+00:00", "updated_at": "2026-08-21 18:45:11.418646+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "developer-tools"], "entities": ["Qwen3.5", "Ollama", "Huang et al.", "Luo et al.", "GPT-4", "GSM8K", "EMNLP 2025", "ICLR 2024"], "alternates": {"html": "https://wpnews.pro/news/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement", "markdown": "https://wpnews.pro/news/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement.md", "text": "https://wpnews.pro/news/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement.txt", "jsonld": "https://wpnews.pro/news/small-local-models-what-actually-holds-up-when-you-re-run-the-measurement.jsonld"}}