{"slug": "claude-code-using-a-local-llm", "title": "Claude Code using a local LLM", "summary": "A developer tested whether splitting a coding task between Claude Code as an orchestrator and a local Qwen3.8 27B Q4_K_M model running on llama.cpp as a worker beats giving the local model the whole job. The orchestrated runs scored 78/78 and 77/78 on a hidden-test suite, while single-prompt runs scored 76, 74 and 71 out of 78, with the developer noting that \"the review is where the defects died\" and that the setup cannot isolate how much the splitting alone contributes.", "body_md": "I wanted a local model for the privacy it gives me. And if I do have to use a frontier model as the master control node, the problem is cost: frontier models are expensive, and the subscription usage limits seem keep getting smaller and smaller.\n\nSo the question is whether the expensive model can do the thinking while the cheap one does the typing, and whether anything worth having comes out the other end. (For this test the \"local\" model ran on a rented RunPod card, which is not private from RunPod. It's the same 16.8 GB file that runs on any 24 GB card you own.)\n\nThere are already plenty of posts on wiring Claude Code to Ollama or a local endpoint. What I couldn't find was anyone measuring whether the split actually beats just giving the local model the whole job. So I had Claude set up a small project with hidden tests, ran it both ways on the same card, and graded it with the same script. This is what came out, including the parts that don't flatter the idea.\n\n**TL;DR.** Orchestrated (Claude writes five small task prompts, reviews every result against the spec, sends defects back with a reproduction): **78/78 and 77/78** on the full test suite, 58 of those tests hidden from the worker. One big prompt to the same model: **76, 74 and 71 out of 78**, and the 71 was the run with the most turns. Every one shot run shipped the same bug (replaying the log wrote it out again), and both orchestrated runs had it too, until review caught it. The worker was Qwen3.8 27B Q4_K_M on llama.cpp, one RTX PRO 4000 Blackwell at $0.57/hr, 32K context, and the whole session cost about forty cents of GPU. The review is where the defects died. This setup can't say how much the splitting alone buys.\n\nTwo roles. \"Claude Code with a local model\" gets used for two different setups, so here is which is which.\n\nThe first setup points Claude Code itself at a local endpoint, so the local model becomes the whole agent. That's not this. Here Claude stays Claude. It plans, writes the task prompts, reads the code that comes back, runs probes against it, and decides what goes back for rework. The local model is a worker it calls through a small harness, one stateless conversation per task, with four tools and a workspace folder.\n\n| Orchestrator | Claude, in a Claude Code session on my Mac | \n| Worker | `Qwen3.8-27B-Uncensored-Q4_K_M.gguf` (16.8 GB), llama.cpp`llama-server` | \n| GPU | 1x RTX PRO 4000 Blackwell, 24 GB, RunPod Secure, $0.57/hr | \n| Context | 32,768 tokens, `--jinja` , flash attention on, thinking off, temperature 0.2 | \n| Reach | SSH tunnel from the Mac to the pod's port 8080 | \n\nI wanted the RTX A5000 at $0.27/hr, which is what the first round of this experiment ran on, but RunPod had none in stock when I asked. The PRO 4000 was next on the list. The card changes speed. The model file and settings are identical, but the harness changed between the rounds, so the two aren't a card comparison.\n\nThe server on the pod is one line:\n\n```\nllama-server -m /workspace/gguf/Qwen3.8-27B-Uncensored-Q4_K_M.gguf \\\n  --host 127.0.0.1 --port 8080 -c 32768 -ngl 99 --jinja -fa on --alias local\n```\n\nIt listens on localhost only, and I reach it through a tunnel instead of RunPod's public proxy:\n\n```\nssh -f -N -p <pod-ssh-port> -i ~/.runpod/ssh/runpodctl-ssh-key \\\n  -o ExitOnForwardFailure=yes -L 8091:127.0.0.1:8080 root@<pod-ip>\ncurl -s http://127.0.0.1:8091/health\n```\n\nThe prebuilt llama.cpp comes from my own Cloudflare R2 bucket. The model came from Hugging Face for the measured runs and is now mirrored to the same bucket. For the screenshot run below, a fresh pod pulled both from R2 and was serving in 379 seconds, a little over six minutes, five of them the 15.7 GiB model download. If you're starting from scratch, [the vLLM post](https://peculiarengineer.com/blog/self-host-qwen-runpod-vllm/) covers renting the card and what RunPod bills you for.\n\nThe pod screenshots come from a fresh pod after the runs above, so the times and token counts in them differ from the tables.\n\nA toy project is only useful if it has corners. This one is `minikv`, a key value store in plain Python with five modules that depend on each other:\n\n| Module | What it does | The corner that bites | \n|---|---|---|\n| `store.py` | keys, values, expiry | `ttl` rounding, expired keys still in the dict | \n| `txn.py` | nested transactions on top of the store | a key deleted in a transaction must read as missing, even if the store still has it | \n| `protocol.py` | parse and format text commands | quoting rules, error order, `KEYS` takes no arguments | \n| `aof.py` | append only log file | a torn last line is ignored, a blank line is kept | \n| `session.py` | ties it together, replays the log at startup | replayed lines must not be logged again; transactions buffer their lines | \n\nThe spec is one page. There are 78 `unittest` tests: 20 the worker can see (one file per module) and 58 it never sees, covering edge cases and cross module scenarios. A reference solution written from the spec alone passes 78/78, so the tests are satisfiable. The grader takes the denominator from the test files, never from the submission, and only copies the five spec modules into its sandbox, so a stray file in the workspace can't shadow anything. It also runs Python with `-I` and counts results from the runner itself, so a fake `unittest.py` or a doctored summary gains nothing.\n\nThe worker is a Python script, `worker.py`, that holds one conversation with the model through the OpenAI style `/v1/chat/completions` endpoint and gives it four tools:\n\n`list_files` and `read_file`, confined to the workspace folder` write_file`, confined the same way (absolute paths, `..` and symlinks out are refused)`run_command`, limited to `python3` scripts, `python3 -c`, `python3 -m unittest`, `ls`, `cat` and `pwd`, relative paths only\nIt also has a repeat guard: when the same sequence of one to three turns, tool calls and results included, repeats four times in a row, it gets a warning, and if it carries on, the run stops with `repeat-stop`. Every call prints one JSON line (trimmed here; a `done` run also carries the model's summary):\n\n```\n{\"turns\": 6, \"calls\": 6, \"bad_calls\": 0, \"completion_tokens\": 693, \"max_prompt_tokens\": 2060, \"total_prompt_tokens\": 9455, \"repeat_warnings\": 0, \"finish\": \"done\", \"secs\": 55.1}\n```\n\n`finish` is the field that matters. `done` means the model said it was finished. `max-turns` means it ran out of turns, and in my runs that usually meant it was going in circles.\n\nFive tasks, one per module, in dependency order:\n\n`store.py`, `protocol.py`, `aof.py`. Nothing depends on anything yet. Three tasks against one `llama-server`, and the slowest of them finished in 87 seconds.` txn.py` (needs the store), then `session.py` (needs everything).\nEach task prompt is the spec text for that module, the interfaces of the modules it uses, the test command, and the line doing the most work: \"the final grading uses extra tests you cannot see, so follow the requirements exactly\". Here is the store prompt, trimmed:\n\n```\nWrite store.py (test file: test_store).\nThe workspace contains clock.py and a visible test file for your module. Plain Python 3 standard\nlibrary only. Run the test file with `python3 -m unittest <testfile without .py>` and fix failures.\nThe final grading uses extra tests you cannot see, so follow the requirements exactly, including\nedge cases and error messages. Keep to the file named below; do not edit other files or the tests.\n\nclass Store(clock); attribute `clock` holds the clock. ...\n- ttl(key) -> -2 if missing or expired, -1 if the key has no expiry, otherwise the remaining\n  seconds rounded UP to an int (math.ceil(expire_at - now)).\n```\n\nA rework prompt states the defect, gives a short reproduction, and quotes the spec. No hint at the cause. This one, trimmed:\n\n```\nReview of the current txn.py against the spec found that, inside a transaction, a key that was\ndeleted or has expired is still treated as present by some methods.\n\n    t.delete(\"a\")      # True, correct\n    t.delete(\"a\")      # returns True, required False (the key is no longer visible to this transaction)\n    t.ttl(\"a\")         # returns -1, required -2 (missing)\n```\n\nThe one shot condition got the whole spec in a single prompt, all five modules at once, with the same tools, the same visible tests and a budget of 90 turns.\n\nAll runs on the same pod, same model, same harness, same grader.\n\n| Run | Score (58 hidden + 20 visible) | Worker turns | Completion tokens | Worker time | Largest prompt | How it ended | \n|---|---|---|---|---|---|---|\n| **Orchestrated, run 1** | **78/78** | 84 | 12,594 | 672 s | 7,832 | all tasks done | \n| **Orchestrated, run 2** | **77/78** | 69 | 11,852 | 673 s | 6,014 | one rework ran out of turns | \n| One shot, run 1 | 74/78 | 23 | 5,395 | 204 s | 10,469 | repeat guard stopped it | \n| One shot, run 2 | 76/78 | 11 | 4,497 | 221 s | 8,568 | said it was done | \n| One shot, guard off, 90 turns | 71/78 | 90 | 22,442 | 935 s | 32,001 | out of turns, context nearly full | \n\nBoth graded again for the screenshots, from the workspaces those runs left behind.\n\nWorker time for the orchestrated runs is summed across tasks. Three of them ran in parallel, so the wall clock was shorter.\n\nThe headline is the gap, 77 to 78 against 71 to 76, but the more useful number is in the Largest prompt column. No orchestrated prompt got past 8K tokens. The one shot runs went to 10K and 8.5K and the long one ran to within 800 tokens of the 32K limit. The small tasks keep the model in the part of its context where it still pays attention.\n\nThe cost of that is turns. Orchestration took three to eight times as many worker turns as the one shot runs that stopped on their own. On a card that bills by the hour, that's a few extra minutes: the whole session, both conditions and all five runs, was 42 minutes of pod time, about $0.40.\n\nEvery defect below passed the visible tests. That's the point of having hidden ones.\n\n| Module | Defect | Run 1 | Run 2 | \n|---|---|---|---|\n| `protocol.py` | `KEYS` not implemented at all;`parse(\"KEYS\")` raised \"unknown command\" | fixed in one rework | fixed in one rework | \n| `txn.py` | a deleted or expired key still counted as found: second `delete` returned True,`ttl` gave -1, an expired key stayed in`keys()` | fixed in one rework | fixed in one rework | \n| `txn.py` | a key deleted in an outer transaction was still visible inside a nested one | not present | still there when the rework rounds ran out | \n| `session.py` | replaying the log at startup wrote every line out again, so the file doubled on each restart | fixed in round two | fixed in one rework | \n| `session.py` | an inner `ROLLBACK` kept that transaction's lines, so a rolled back`SET` reached the log | fixed in round two | not present | \n\nThe replay bug is the one I'd frame. Every one shot run shipped it: all three fail the hidden test `test_replay_does_not_relog`. The orchestrated runs wrote it too, both times. The only difference is that Claude read `__init__`, saw the replay loop call `execute()`, and knew `execute()` logs. It's an easy bug to write and an easy bug to see, as long as somebody is looking.\n\nThe same goes for `txn.py`. The model wrote the same \"deleted means found\" bug in both orchestrated runs, and the first round of this experiment, the day before, hit it too. It's a stable blind spot.\n\nI wouldn't trust it on its own, like any model. I would need to review its output. It will tell you it's done with the replay bug still in there, so the shape that works is well specified modules going out and something else doing the reading.\n\nThe second orchestrated run lost one test, and it's a fair hit. The spec says to quote an argument when it is empty or contains whitespace or a double quote. The worker also quoted values with a backslash in them, so `format_command` produced `SET k \"a\\\\b\"` where the hidden test wanted `SET k a\\b`. Both forms parse back to the same command, so a round trip check passes either way. Catching it means checking what `format_command` writes, character by character, and the review didn't.\n\nAnd the nested transaction bug in run 2 was still there when its rework rounds ran out (only the second one was about it), but none of the 58 hidden tests exercise that case, so it cost nothing on the scoreboard. The review found a real defect the tests can't see, and the tests found one the review missed. Neither is a complete check on its own.\n\nThe obvious objection to the first round of this experiment was budget: the orchestrated run used 88 turns and the one shot run only 40. So this time one shot got 90 turns, and one run had the repeat guard switched off so it could use all of them.\n\nIt used all of them and scored the worst of the five, 71/78. By the end its prompt was 32,001 tokens, within 800 of the 32,768 limit, because every turn appends the tool results to the conversation. The shorter runs failed two and four hidden tests. This one failed seven, including the transaction logging tests the shorter runs passed. More rope, same model, worse code. The runs with the guard on stopped themselves at 23 and 11 turns and did better.\n\nSo the gap isn't that orchestration gets more turns. It's that each turn happens in a short, focused conversation, with a reviewer deciding what the next one is about.\n\n**The command filter is not a sandbox.** In one session rework the worker spent twelve of its twenty turns running the same check script with a new file name each time: `/tmp/aof1.txt`, `/tmp/aof2.txt`, up to `/tmp/aof12.txt`. Two problems in one. The absolute path was inside a `python3 -c` string, which the filter doesn't parse, so it wrote eleven files into `/tmp` on my Mac. And because each turn used a different file name, no two turns were identical and the repeat guard never fired. The worker's code runs as you, on your machine. Give it a workspace with nothing in it that matters.\n\n**`finish: done` is the model's opinion.** Every task that ended `done` with a defect still in it came back with a cheerful summary saying all tests pass or everything is correct. They did pass. The visible ones.\n\n**`max-turns` means read the code.** Three worker calls ran out of turns across the orchestrated runs. One was the original session task, which left code that passed its visible tests with two defects in it. The other two were reworks that didn't fix what they were sent to fix. The status tells you it stopped, not what it left behind.\n\n**A second rework needs to say where.** The first session rework (defect plus reproduction plus spec) changed nothing at all. The second named the method and described the fix, and it was done in seven turns. That's the orchestrator doing more of the work, and it counts against the \"Claude only reviews\" story. It's in the honest limits below.\n\n**Budget for the review.** The worker's numbers are in the table. Claude's aren't: reading five modules, writing probes and writing rework prompts is real orchestrator time and tokens, and it isn't counted anywhere above.\n\n| Thing | Value | \n|---|---|\n| Model | `Qwen3.8-27B-Uncensored-Q4_K_M.gguf` , 16,810,714,496 bytes | \n| Card | 24 GB is enough at 32K context (RTX PRO 4000 Blackwell here, RTX A5000 in the first round) | \n| Server | `llama-server -c 32768 -ngl 99 --jinja -fa on --alias local` | \n| Worker settings | temperature 0.2, `enable_thinking: false` ,`max_tokens` 4096 | \n| Task size that worked | one module, 20 to 150 lines, its spec text, its interfaces, its test command | \n| Rework | defect, reproduction, spec text; at most two rounds | \n| Trust | `finish: done` and passing visible tests mean nothing until someone reads the code | \n\nThe cheap model is good enough for small, well specified tasks like this one. Would I trust it to touch my database? Probably not. But I can use it for automation tasks like Python or Bash scripts. Like with any model, check the output. Don't just run whatever it gives you.", "url": "https://wpnews.pro/news/claude-code-using-a-local-llm", "canonical_source": "https://dev.to/peculiarengineer/claude-code-using-a-local-llm-28nb", "published_at": "2026-10-16 19:39:13+00:00", "updated_at": "2026-10-16 19:50:36.311114+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "mlops", "ai-infrastructure"], "entities": ["Claude Code", "Qwen3.8 27B", "llama.cpp", "RunPod", "RTX PRO 4000 Blackwell", "Hugging Face", "Cloudflare R2", "Ollama"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/claude-code-using-a-local-llm", "markdown": "https://wpnews.pro/news/claude-code-using-a-local-llm.md", "text": "https://wpnews.pro/news/claude-code-using-a-local-llm.txt", "jsonld": "https://wpnews.pro/news/claude-code-using-a-local-llm.jsonld"}}