{"slug": "a-1-5b-model-is-a-triage-signal-not-a-verdict", "title": "A 1.5B model is a triage signal, not a verdict", "summary": "A developer built a local code-auditing tool that runs five deterministic checks over trading-strategy repositories and then asks a locally hosted Qwen2.5-Coder-1.5B-Instruct Q4_K_M model, via llama.cpp on a laptop CPU, a single yes/no question about each finding's evidence line. Measuring the pipeline showed roughly 12 seconds per model question (48–50 seconds for four checks versus under a second for the deterministic pass), and re-asking Gemma 2 2B IT the same question three times produced NO, NO, YES — leading the developer to conclude that at 2B the verdict is a triage hint rather than a stable measurement, with file locations always coming from pattern-matching code rather than the model.", "body_md": "I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong.\n\nThe tool runs five deterministic checks over the source, then hands each finding to a local model with exactly six lines of context and one yes/no question: *is this actually true here?* Nothing leaves the machine. No API key, no upload, no account. The model is Qwen2.5-Coder-1.5B-Instruct Q4_K_M, loaded through llama.cpp, on a laptop CPU.\n\nThe interesting part is not that it works. It is where it stops working, and what I changed once I measured that.\n\nFour model checks on the bundled sample take about 48 to 50 seconds on this laptop. That is roughly twelve seconds per question. The deterministic pass, `--no-llm`, takes under a second.\n\nTwelve seconds per question is the fact that shaped everything else. You cannot put that inside a loop over every file in a repository. You cannot put it behind a save button. What you can do is run it at night, or while you make coffee, and then read a report that is already sorted.\n\nLatency also removes a whole class of bad ideas for free. I stopped considering \"ask the model to look around and tell me what seems off\" not because it is philosophically wrong but because it costs twelve seconds and returns a paragraph, and I do not want a paragraph at that price. I want a pointer.\n\nThe model never sees the repository. It sees the finding's evidence line plus five lines around it, and it can answer YES, NO, or UNSURE. It is never asked to name a file, never asked for a line number, never asked for a summary.\n\nThat last restriction came from a failure. In an early build I asked it to summarise the file. The output was fluent, specific, and wrong about which function did what — the kind of answer that is expensive to read because you have to verify every clause. A checker that points at line 9 and a model that confirms or denies a claim about line 9 is a pipeline. A model that writes prose about code is a different product, and not one a 1.5B can deliver.\n\nSo the division is blunt:\n\nHere is the measurement that changed my design.\n\nI asked Gemma 2 2B IT Q4_K_M (temperature 0.2) the same yes/no question about the same code, three times, in isolated contexts. The answers were **NO, NO, YES**.\n\nSame weights, same prompt, same evidence line. Two of three said the cost bug was not there. It was there; the checker had already printed the ledger line with no fee term in it.\n\nI also run both models and print their answers side by side, as a confidence signal. Two runs of the same audit gave **3 of 4** agreement and then **4 of 4**. It would be tidy to call that sampling variance, but it is not: the code changed between the runs — I moved the context window to the evidence line — so the difference is confounded. I would rather write that down than round it into a nicer story.\n\nBoth runs are committed in the repository, and so are the three re-asks. That is the only claim I am willing to make: at 2B, on this task, **the verdict is not a stable measurement**. It is a hint about where to look, from a reader who has read six lines.\n\nOnce you accept that, the design stops fighting the model.\n\n`path:line` in the report comes from code that matched a pattern. The model cannot create a finding, move one, or delete one.\nThe blind spot is written in fixed text, not generated: the tool does not run the strategy, does not see your data, and cannot tell you whether the edge is real. All it can do is point at the places where a backtest usually lies and ask a small model whether the pointer is correct.\n\nThe honest gap is the threshold. Right now a finding either fires or it does not, and the model's answer is read by a human. If I ever wanted to automate the decision — skip the ones the model denies, say — I would need two numbers I do not have:\n\nBoth are cheap to measure and both are missing from this post. That is the difference between a demo and something I would let run unattended on a repository I care about. I would rather say that here than discover it in production.\n\n`--no-llm` runs the deterministic checks only, instantly, and it is the right mode more often than you would think:\n\nThe model is worth its twelve seconds when the finding is ambiguous and you want a second reader at 3am. It is not worth it when you already know what to look at.\n\n```\npip install -e .\nslothaudit audit samples/broken_lookahead            # ~50s, four model checks\nslothaudit audit samples/broken_lookahead --compare   # both models, one report\nslothaudit audit samples/broken_lookahead --no-llm    # deterministic only, instant\npython3 -m slothaudit audit samples/broken_lookahead  # works without install\n```\n\nA clean repository gets a deliberately un-reassuring answer: *no v0 check fired; that means this run had nothing to point at, not that the strategy survives an audit.*\n\nCode: [github.com/rytjy/slothaudit](https://github.com/rytjy/slothaudit)\n\n*Disclosure: I used an AI assistant to help draft and edit this post. The measurements, the code, and the command output are mine, and the output above is pasted verbatim from runs on this machine.*", "url": "https://wpnews.pro/news/a-1-5b-model-is-a-triage-signal-not-a-verdict", "canonical_source": "https://dev.to/rytjy/a-15b-model-is-a-triage-signal-not-a-verdict-2j63", "published_at": "2026-10-24 06:03:08+00:00", "updated_at": "2026-10-24 06:12:08.216641+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Qwen2.5-Coder-1.5B-Instruct", "Gemma 2 2B IT", "llama.cpp"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-1-5b-model-is-a-triage-signal-not-a-verdict", "markdown": "https://wpnews.pro/news/a-1-5b-model-is-a-triage-signal-not-a-verdict.md", "text": "https://wpnews.pro/news/a-1-5b-model-is-a-triage-signal-not-a-verdict.txt", "jsonld": "https://wpnews.pro/news/a-1-5b-model-is-a-triage-signal-not-a-verdict.jsonld"}}