# How I Taught Claude Code to Offload Grunt Work to a Local 4B Model > Source: > Published: 2026-08-27 00:01:03+00:00 My GPU sits idle most of the day. Meanwhile, I keep burning Claude tokens on tasks a small model could finish in two seconds: summarizing a paragraph, rewording a commit message, suggesting five variable names, classifying a list of lines. It felt wasteful. So I tried to bridge them: keep Claude Code as my main driver, but let it delegate the trivial stuff to a Qwen3.5–4B running locally on llama.cpp. My first three instincts were all wrong. The fourth worked. This post is the whole arc — including a bug that left me staring at a blank terminal for twenty minutes — so you can skip straight to the working setup. If you’re already using Claude Code and have heard about local LLMs but never wired one in, this is for you. Before I show what works, here’s what most people try first. Skip these so you don’t waste your afternoon. **“I’ll just make a subagent that uses Qwen.”** You can’t. In Claude Code, subagents run on the same model as the main session. The model: field in subagent frontmatter accepts names like haiku or sonnet, but they all resolve against the same ANTHROPIC_BASE_URL. Subagents are context isolation, not model isolation. **“I’ll point ****ANTHROPIC_BASE_URL at my llama-server."** This *does* work — llama.cpp exposes an Anthropic-compatible /v1/messages endpoint, and Claude Code will happily talk to it. But it's all-or-nothing. The moment you set that env var, *everything* goes through Qwen, including the heavy reasoning tasks where you actually need Claude. You haven't built a hybrid; you've just downgraded your whole session. **“I’ll add multiple endpoints to settings.json, like opencode does.”** Opencode and qwen-code support per-model endpoints in their config. Claude Code does not. Don’t go looking for a flag — it isn’t there. The real answer is to stop thinking about this at the *model* level and start thinking at the *application* level. Claude stays in charge. Qwen becomes a tool Claude can call when it judges the task to be trivial. Claude Code has two primitives that, combined, give you exactly what you want: Glue them: write a Skill whose instructions are *“when you see a small text task, call this Python script via Bash; the script hits a local llama-server and returns the result.”* That’s it. No MCP server to maintain. No proxy. No config gymnastics. Claude makes the routing decision per-task, based on the skill’s description. Why not an MCP server? You could. MCP is the “right” answer if you want a persistent, structured tool. But for a single endpoint that takes text in and returns text out, a 60-line Python script is faster to write, easier to debug, and has fewer failure modes. MCP is overkill here. **Prerequisites** **Step 1: Run llama-server** Whatever flags you normally use are fine. For reference, mine on Windows: ``` llama-server.exe ^ -m gguf\Qwen3.5-4B\Qwen3.5-4B-UD-Q4_K_XL.gguf ^ --jinja ^ -c 32768 ^ --port 8080 ^ -a Qwen3.5-4B ``` The two flags that matter for this setup: Verify it’s alive: ``` curl http://127.0.0.1:8080/v1/models ``` **Step 2: Create the skill** Skills live at ~/.claude/skills// on macOS/Linux, or C:\Users\\.claude\skills\\ on Windows. Create the folder qwen-local-assistant/ with two files. **SKILL.md** — the file Claude reads to know when and how to delegate: ``` ---name: qwen-local-assistantdescription: Offload small text tasks (summarize, reword, translate, prose-to-list, name suggestions, classify, simple boilerplate) to a local Qwen3.5-4B running on llama.cpp at http://127.0.0.1:8080. ALWAYS use this skill when the user wants a self-contained text transformation that doesn't require codebase context or multi-step reasoning. Do NOT use for code edits, debugging, codebase-aware tasks, or anything where being wrong is costly.---# Qwen Local AssistantYou have a local Qwen3.5-4B model running on the user's machine athttp://127.0.0.1:8080. Use it as a cheap helper for low-stakes text workvia the `ask_qwen.py` script in this skill directory.## When to delegateDelegate when ALL of these are true:- Text in, text out (no file edits, no code semantics)- Self-contained - the prompt is short and stands on its own- Low stakes - a slightly off output is acceptable- It's tedious busywork## How to callPipe the prompt through stdin to avoid quote-escaping issues: echo "Summarize in 2 sentences: " | python /ask_qwen.py## After it respondsRead critically. Qwen3.5-4B can hallucinate. Sanity-check beforeintegrating. When showing the user, mention it came from the local model. ``` **ask_qwen.py** — the bridge script: ``` bash #!/usr/bin/env python3import sys, json, argparse, urllib.request, urllib.errorDEFAULT_URL = "http://127.0.0.1:8080/v1/chat/completions"DEFAULT_MODEL = "Qwen3.5-4B"def main(): p = argparse.ArgumentParser() p.add_argument("prompt", nargs="?", default=None) p.add_argument("--system", default=None) p.add_argument("--max-tokens", type=int, default=2048) p.add_argument("--temperature", type=float, default=0.3) p.add_argument("--url", default=DEFAULT_URL) p.add_argument("--model", default=DEFAULT_MODEL) p.add_argument("--timeout", type=int, default=180) p.add_argument("--enable-thinking", action="store_true") args = p.parse_args() prompt = args.prompt if args.prompt else sys.stdin.read() if not prompt.strip(): print("ERROR: empty prompt", file=sys.stderr); return 2 messages = [] if args.system: messages.append({"role": "system", "content": args.system}) messages.append({"role": "user", "content": prompt}) body = json.dumps({ "model": args.model, "messages": messages, "max_tokens": args.max_tokens, "temperature": args.temperature, "stream": False, "chat_template_kwargs": {"enable_thinking": args.enable_thinking}, }).encode("utf-8") req = urllib.request.Request( args.url, data=body, headers={"Content-Type": "application/json"}, method="POST") try: with urllib.request.urlopen(req, timeout=args.timeout) as r: data = json.loads(r.read().decode("utf-8")) except urllib.error.URLError as e: print(f"ERROR: {e}", file=sys.stderr); return 1 msg = data["choices"][0]["message"] content = (msg.get("content") or "").strip() if not content: reasoning = (msg.get("reasoning_content") or "").strip() if reasoning: print("WARNING: only reasoning returned", file=sys.stderr) print(reasoning); return 0 print("ERROR: empty response", file=sys.stderr); return 1 print(content)if __name__ == "__main__": sys.exit(main()) ``` **Step 3: Sanity-check from the terminal** Before going through Claude Code, confirm the script works on its own: ``` echo "Summarize in one sentence: the cat climbed on the roof and meowed all night." | python ~/.claude/skills/qwen-local-assistant/ask_qwen.py ``` You should get a one-line summary in about two seconds. If you get nothing, jump to the next section — you probably hit the bug I hit. **Step 4: Use it in Claude Code** Restart Claude Code so it picks up the new skill. Then prompt it with something like: “Take the changelog below and produce three user-facing bullets. Use the Qwen local assistant skill.” You’ll see Claude invoke Bash, pipe the prompt to ask_qwen.py, get the response, and weave it into its reply. The first time it works it feels a little like cheating. **The bug that almost killed it** When I first wired this up, I ran the test command and got nothing. Blank line. The server log clearly showed a successful POST and 1024 tokens generated. But content in the response was empty. **The culprit:** Qwen3.5 has a reasoning mode. When --jinja is active and the chat template's enable_thinking flag is on (the default), the model wraps its output in a ... block before producing the actual answer. llama.cpp separates that into a reasoning_content field, leaving content empty until the tag appears. For a “summarize in one sentence” prompt, the 4B model happily burned all 1024 tokens *thinking about how to summarize* and never reached the actual answer. The block never closed. content stayed empty. My script printed the empty string. The fix is a single line in the request body: ``` "chat_template_kwargs": {"enable_thinking": False} ``` That’s already in the script above. If you’re adapting the pattern for another reasoning-capable model, this is the gotcha to watch for. The symptom (response with tokens generated but empty content) is identical across most reasoning models I’ve tried. **What this is actually good for** I want to be honest, because Medium is full of “I replaced Claude with a local model” posts that are either lying or measuring the wrong thing. **This is good for:** summarizing pasted text, rewording sentences, converting prose to bullets, translating short snippets, generating naming variants, simple classification, and bulk operations where you’d otherwise spend Claude tokens on repetitive busywork. **This is not good for:** anything code-related, anything requiring multi-step reasoning, anything that depends on understanding your codebase, anything where being subtly wrong has consequences. Qwen3.5–4B is a 4B parameter model. It’s not Claude. It’s not even Haiku. Treat it accordingly. **What about cost savings?** Marginal, and probably not the right reason to do this. If you’re on a flat-rate Claude plan, you save nothing. If you’re on API and you delegate, say, 50 small tasks a day at maybe 500 tokens each, you’re saving cents. The real value isn’t dollars — it’s that your GPU does something useful, you get fast responses for trivial tasks (no network round-trip), and the pattern works offline. **Hidden cost:** the model has to be running. If you forget to start llama-server, the skill fails gracefully but Claude has to fall back to doing the task itself. Not a disaster, but worth knowing. **The honest pitch:** this is a worker pool for grunt work, not a Claude replacement. Anyone selling you the latter is either misinformed or measuring success on toy benchmarks. **Why the pattern matters beyond Qwen** Once you’ve built one of these bridges, you realize the Skill + Bash composition is surprisingly general. You can delegate to anything that listens on a port: The Skill description tells Claude when to invoke. The Bash tool gives it the means. localhost gives it the destination. No MCP, no proxy, no config gymnastics. You can wire a new external capability into Claude Code in about an hour. That’s the real takeaway. Qwen is just the example. **Repo and what to try first** I’ve published the skill files on GitHub: ``` git clone https://github.com/anmerino-pnd/qwen-local-assistant.git. ``` Clone, drop into ~/.claude/skills/, restart Claude Code. Two things to try once it’s running: If something breaks, the most likely culprits, in order: Tell me what breaks. The pattern is more interesting than the specific implementation, and there are probably better skill descriptions and better delegation heuristics than the ones I shipped. [How I Taught Claude Code to Offload Grunt Work to a Local 4B Model](https://pub.towardsai.net/claude-code-skill-local-qwen-3c6bb5d051bd) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.