What broke when we ran agents on small local models

The team behind Bees, a system that runs a team of AI agents locally, documented five failures encountered while getting agents to work on 3-4B parameter models via llama.cpp, including a parser lock that silently dropped second tool calls and a prompt that contradicted its own tool list. The team found that most failures were harness bugs rather than model weakness, and switched its local agent model to Qwen3 4B Instruct while moving core tools into the always-visible set.

Bees runs a team of AI agents on your own computer. You can point it at Claude Code, Codex or an API key, but we also ship a local model so you can run it without paying for an API. That model runs through llama.cpp on a laptop, and it is small: 3 to 4 billion parameters. Getting agents to do real work on a model that size took us most of August and September. Almost none of the failures were "the model is too dumb". Most of them were our code, or a mismatch between our code and the model, and they looked exactly like a dumb model from the outside. Here are the five that cost us the most time. The first local model we tried was Nanbeige 4.2 3B. On the first turn it called tools perfectly. Then the run would just end. No error. The agent said it was done, and no file was written, no search was run. We first read this as "a 3B model is too weak for agents" and started planning around it: one tool call per agent, a bigger model for some roles. That was the wrong fix for the wrong problem. The real cause took a two-turn request to reproduce. Nanbeige sends its first tool call as JSON. Once a tool result is in the history, it switches to an XML style: <tool call <function=write file <parameter=path notes.md</parameter ... llama.cpp locks its parser to the first format it detects. So the second call never got parsed. It came back as plain text in content , with finish reason: stop and tool calls: null . To our agent loop that is a model saying "I'm finished". Nothing ran because, as far as the code could tell, nothing was asked for. It wasn't a missing --jinja flag, which is on by default in our llama.cpp build. Forcing the XML parser would just have broken turn one instead. The same model had a second problem hiding behind the first. Our catalog pinned it to an 8k context window, and our own agent prompt measures 8,567 tokens. So the code that sizes the context for the prompt never ran. What fixed it: we stopped offering models that can't hold one tool-call format for a whole run. Bees now offers Qwen3 4B Instruct as its local model for agents, and dropped the ones that couldn't run one. The lesson we took: when an agent run "does nothing", look at the raw reply before you blame the prompt or the model. If you see <tool call or <function= sitting in the text, the model did its job and the parser didn't. Bees has a lot of tools. One planning agent's first request was about 22k tokens, and 15.6k of that was the schemas for 79 tools. On a laptop running a 16k context that does not fit at all. So most schemas sit behind a search tool. The model sees a small base set and calls the search tool to load the rest. It loads 4 matching schemas at a time. Two things went wrong with small models. First, a small model searches for its task words, not for a capability. Asked to fetch the top 5 news stories, it searched, matched nothing, and decided the run had no tools at all. In another run it searched "nepal news in nepali api", matched an unrelated add-on, and reported the same. Both times it gave up. A big model would try a second search. A 4B model takes the first answer as the truth. We moved reading, writing and the web back into the always-visible set, and an empty search now lists the tool names it could have asked for. Second, and worse, our own prompt contradicted the tool list. It said "the tools listed are ready to call, only search when nothing listed can do the job". Then, two lines down, it told the model to call our delegation and knowledge tools. Those weren't listed. A big model notices and goes looking. A small model believes the harness, sees the tool is missing, and stops. That bug made eight local models "fail" one of our internal team tests on a tool they were never given. Every failing result we had recorded for those test tasks up to that day was measured this way. We were grading models on whether they would second-guess our prompt. What fixed it: a sweep. We grep every tool name that our prompts and error messages mention and compare it against the set the model can actually see. Anything named but hidden is a bug. We also hide a tool entirely when it can't work. Our web search needs a key, and without it every call failed, yet the model kept calling it because it was listed. Now it only shows up when the key is there. The search tool keeps the last 8 loaded schemas and drops the oldest. The tool itself still works after its schema is dropped. Only its description leaves the prompt. On a long email run, after 14 searches, the model called a browser tool it had loaded much earlier. Its schema had already been dropped. Our adapter checked the name against that turn's list, didn't find it, threw "unknown tool" and killed the run. This one happened on a big model through Claude Code, but it would hit any model. The model wasn't wrong here. It remembered a real tool. We had treated a cache miss in the prompt as a protocol violation. What fixed it: the adapter now passes the call through. If the tool exists, it runs. If the model made the name up, it gets back a normal tool error it can recover from. A long run with many tools no longer fails at random. Bees agents keep a memory so they can skip things they already handled and stick to daily limits. At one point every memory save failed with "Extraction failed". Agents saw saves time out and recall return nothing. The memory service uses the local model to pull facts out of text. That was Qwen3.5 4B, and with the chat template on, it thinks by default. It wrote 2,000 to 8,500 tokens of thinking for each fact extraction, at about 30 tokens a second. That is well past the memory service's 300 second timeout. And llama.cpp had one slot, so every other save queued behind it. When the laptop was deep in swap, output fell to about 1 token a second. What fixed it was one field. Our bridge to the model now sends chat template kwargs: { enable thinking: false } for memory calls. The same extraction took 1.8 seconds and 43 tokens. If you run a thinking model locally behind anything with a timeout, check what it does by default. A reasoning model's real cost is the tokens it spends before it answers, and on a laptop those tokens are seconds. After the fixes above, small models still took shortcuts. A run would say it wrote a report when the file was empty. A summary would describe a search that never happened. A model would ask the user a question with only one possible answer and stall the run on it. A model stuck in a loop would hand out the exact same batch of work again. We had rules against all of this in the prompt. A small model reads a rule as a suggestion. So the step where an agent hands in its result now checks the work instead of trusting the description: The interesting part: a much bigger model tried every one of those shortcuts too. The checks bounced it, and then it did the work. So these aren't small-model crutches. They are how the harness should have worked from the start. Bees is free and open source, on Mac, Windows and Linux, and runs agents on Claude Code, Codex, an API key or a local model. Download it at bees.bot https://bees.bot/download/ , and the code is at github.com/Bees-bot/bees-desktop https://github.com/Bees-bot/bees-desktop .