# A 1.5B model is a triage signal, not a verdict > Source: > Published: 2026-10-24 06:03:08+00:00 I run an auditor that reads trading-strategy repos while I sleep. A 1.5B model is part of it, and for a long time I described that part wrong. The tool runs five deterministic checks over the source, then hands each finding to a local model with exactly six lines of context and one yes/no question: *is this actually true here?* Nothing leaves the machine. No API key, no upload, no account. The model is Qwen2.5-Coder-1.5B-Instruct Q4_K_M, loaded through llama.cpp, on a laptop CPU. The interesting part is not that it works. It is where it stops working, and what I changed once I measured that. Four model checks on the bundled sample take about 48 to 50 seconds on this laptop. That is roughly twelve seconds per question. The deterministic pass, `--no-llm`, takes under a second. Twelve seconds per question is the fact that shaped everything else. You cannot put that inside a loop over every file in a repository. You cannot put it behind a save button. What you can do is run it at night, or while you make coffee, and then read a report that is already sorted. Latency also removes a whole class of bad ideas for free. I stopped considering "ask the model to look around and tell me what seems off" not because it is philosophically wrong but because it costs twelve seconds and returns a paragraph, and I do not want a paragraph at that price. I want a pointer. The model never sees the repository. It sees the finding's evidence line plus five lines around it, and it can answer YES, NO, or UNSURE. It is never asked to name a file, never asked for a line number, never asked for a summary. That last restriction came from a failure. In an early build I asked it to summarise the file. The output was fluent, specific, and wrong about which function did what — the kind of answer that is expensive to read because you have to verify every clause. A checker that points at line 9 and a model that confirms or denies a claim about line 9 is a pipeline. A model that writes prose about code is a different product, and not one a 1.5B can deliver. So the division is blunt: Here is the measurement that changed my design. I asked Gemma 2 2B IT Q4_K_M (temperature 0.2) the same yes/no question about the same code, three times, in isolated contexts. The answers were **NO, NO, YES**. Same weights, same prompt, same evidence line. Two of three said the cost bug was not there. It was there; the checker had already printed the ledger line with no fee term in it. I also run both models and print their answers side by side, as a confidence signal. Two runs of the same audit gave **3 of 4** agreement and then **4 of 4**. It would be tidy to call that sampling variance, but it is not: the code changed between the runs — I moved the context window to the evidence line — so the difference is confounded. I would rather write that down than round it into a nicer story. Both runs are committed in the repository, and so are the three re-asks. That is the only claim I am willing to make: at 2B, on this task, **the verdict is not a stable measurement**. It is a hint about where to look, from a reader who has read six lines. Once you accept that, the design stops fighting the model. `path:line` in the report comes from code that matched a pattern. The model cannot create a finding, move one, or delete one. The blind spot is written in fixed text, not generated: the tool does not run the strategy, does not see your data, and cannot tell you whether the edge is real. All it can do is point at the places where a backtest usually lies and ask a small model whether the pointer is correct. The honest gap is the threshold. Right now a finding either fires or it does not, and the model's answer is read by a human. If I ever wanted to automate the decision — skip the ones the model denies, say — I would need two numbers I do not have: Both are cheap to measure and both are missing from this post. That is the difference between a demo and something I would let run unattended on a repository I care about. I would rather say that here than discover it in production. `--no-llm` runs the deterministic checks only, instantly, and it is the right mode more often than you would think: The model is worth its twelve seconds when the finding is ambiguous and you want a second reader at 3am. It is not worth it when you already know what to look at. ``` pip install -e . slothaudit audit samples/broken_lookahead # ~50s, four model checks slothaudit audit samples/broken_lookahead --compare # both models, one report slothaudit audit samples/broken_lookahead --no-llm # deterministic only, instant python3 -m slothaudit audit samples/broken_lookahead # works without install ``` A clean repository gets a deliberately un-reassuring answer: *no v0 check fired; that means this run had nothing to point at, not that the strategy survives an audit.* Code: [github.com/rytjy/slothaudit](https://github.com/rytjy/slothaudit) *Disclosure: I used an AI assistant to help draft and edit this post. The measurements, the code, and the command output are mine, and the output above is pasted verbatim from runs on this machine.*