My GPU sits idle most of the day. Meanwhile, I keep burning Claude tokens on tasks a small model could finish in two seconds: summarizing a paragraph, rewording a commit message, suggesting five variable names, classifying a list of lines.
It felt wasteful. So I tried to bridge them: keep Claude Code as my main driver, but let it delegate the trivial stuff to a Qwen3.5–4B running locally on llama.cpp.
My first three instincts were all wrong. The fourth worked. This post is the whole arc — including a bug that left me staring at a blank terminal for twenty minutes — so you can skip straight to the working setup.
If you’re already using Claude Code and have heard about local LLMs but never wired one in, this is for you.
Before I show what works, here’s what most people try first. Skip these so you don’t waste your afternoon.
“I’ll just make a subagent that uses Qwen.” You can’t. In Claude Code, subagents run on the same model as the main session. The model: field in subagent frontmatter accepts names like haiku or sonnet, but they all resolve against the same ANTHROPIC_BASE_URL. Subagents are context isolation, not model isolation.
**“I’ll point **ANTHROPIC_BASE_URL at my llama-server." This does work — llama.cpp exposes an Anthropic-compatible /v1/messages endpoint, and Claude Code will happily talk to it. But it's all-or-nothing. The moment you set that env var, everything goes through Qwen, including the heavy reasoning tasks where you actually need Claude. You haven't built a hybrid; you've just downgraded your whole session.
“I’ll add multiple endpoints to settings.json, like opencode does.” Opencode and qwen-code support per-model endpoints in their config. Claude Code does not. Don’t go looking for a flag — it isn’t there.
The real answer is to stop thinking about this at the model level and start thinking at the application level. Claude stays in charge. Qwen becomes a tool Claude can call when it judges the task to be trivial.
Claude Code has two primitives that, combined, give you exactly what you want:
Glue them: write a Skill whose instructions are “when you see a small text task, call this Python script via Bash; the script hits a local llama-server and returns the result.”
That’s it. No MCP server to maintain. No proxy. No config gymnastics. Claude makes the routing decision per-task, based on the skill’s description.
Why not an MCP server? You could. MCP is the “right” answer if you want a persistent, structured tool. But for a single endpoint that takes text in and returns text out, a 60-line Python script is faster to write, easier to debug, and has fewer failure modes. MCP is overkill here.
Prerequisites
Step 1: Run llama-server
Whatever flags you normally use are fine. For reference, mine on Windows:
llama-server.exe ^ -m gguf\Qwen3.5-4B\Qwen3.5-4B-UD-Q4_K_XL.gguf ^ --jinja ^ -c 32768 ^ --port 8080 ^ -a Qwen3.5-4B
The two flags that matter for this setup:
Verify it’s alive:
curl http://127.0.0.1:8080/v1/models
Step 2: Create the skill
Skills live at ~/.claude/skills/<name>/ on macOS/Linux, or C:\Users<you>.claude\skills<name>\ on Windows. Create the folder qwen-local-assistant/ with two files.
SKILL.md — the file Claude reads to know when and how to delegate:
---name: qwen-local-assistantdescription: Offload small text tasks (summarize, reword, translate, prose-to-list, name suggestions, classify, simple boilerplate) to a local Qwen3.5-4B running on llama.cpp at http://127.0.0.1:8080. ALWAYS use this skill when the user wants a self-contained text transformation that doesn't require codebase context or multi-step reasoning. Do NOT use for code edits, debugging, codebase-aware tasks, or anything where being wrong is costly.---# Qwen Local AssistantYou have a local Qwen3.5-4B model running on the user's machine athttp://127.0.0.1:8080. Use it as a cheap helper for low-stakes text workvia the `ask_qwen.py` script in this skill directory.## When to delegateDelegate when ALL of these are true:- Text in, text out (no file edits, no code semantics)- Self-contained - the prompt is short and stands on its own- Low stakes - a slightly off output is acceptable- It's tedious busywork## How to callPipe the prompt through stdin to avoid quote-escaping issues: echo "Summarize in 2 sentences: <text>" | python <skill_dir>/ask_qwen.py## After it respondsRead critically. Qwen3.5-4B can hallucinate. Sanity-check beforeintegrating. When showing the user, mention it came from the local model.
ask_qwen.py — the bridge script:
#!/usr/bin/env python3import sys, json, argparse, urllib.request, urllib.errorDEFAULT_URL = "http://127.0.0.1:8080/v1/chat/completions"DEFAULT_MODEL = "Qwen3.5-4B"def main(): p = argparse.ArgumentParser() p.add_argument("prompt", nargs="?", default=None) p.add_argument("--system", default=None) p.add_argument("--max-tokens", type=int, default=2048) p.add_argument("--temperature", type=float, default=0.3) p.add_argument("--url", default=DEFAULT_URL) p.add_argument("--model", default=DEFAULT_MODEL) p.add_argument("--timeout", type=int, default=180) p.add_argument("--enable-thinking", action="store_true") args = p.parse_args() prompt = args.prompt if args.prompt else sys.stdin.read() if not prompt.strip(): print("ERROR: empty prompt", file=sys.stderr); return 2 messages = [] if args.system: messages.append({"role": "system", "content": args.system}) messages.append({"role": "user", "content": prompt}) body = json.dumps({ "model": args.model, "messages": messages, "max_tokens": args.max_tokens, "temperature": args.temperature, "stream": False, "chat_template_kwargs": {"enable_thinking": args.enable_thinking}, }).encode("utf-8") req = urllib.request.Request( args.url, data=body, headers={"Content-Type": "application/json"}, method="POST") try: with urllib.request.urlopen(req, timeout=args.timeout) as r: data = json.loads(r.read().decode("utf-8")) except urllib.error.URLError as e: print(f"ERROR: {e}", file=sys.stderr); return 1 msg = data["choices"][0]["message"] content = (msg.get("content") or "").strip() if not content: reasoning = (msg.get("reasoning_content") or "").strip() if reasoning: print("WARNING: only reasoning returned", file=sys.stderr) print(reasoning); return 0 print("ERROR: empty response", file=sys.stderr); return 1 print(content)if __name__ == "__main__": sys.exit(main())
Step 3: Sanity-check from the terminal
Before going through Claude Code, confirm the script works on its own:
echo "Summarize in one sentence: the cat climbed on the roof and meowed all night." | python ~/.claude/skills/qwen-local-assistant/ask_qwen.py
You should get a one-line summary in about two seconds. If you get nothing, jump to the next section — you probably hit the bug I hit.
Step 4: Use it in Claude Code
Restart Claude Code so it picks up the new skill. Then prompt it with something like:
“Take the changelog below and produce three user-facing bullets. Use the Qwen local assistant skill.”
You’ll see Claude invoke Bash, pipe the prompt to ask_qwen.py, get the response, and weave it into its reply. The first time it works it feels a little like cheating.
The bug that almost killed it
When I first wired this up, I ran the test command and got nothing. Blank line. The server log clearly showed a successful POST and 1024 tokens generated. But content in the response was empty.
The culprit: Qwen3.5 has a reasoning mode. When --jinja is active and the chat template's enable_thinking flag is on (the default), the model wraps its output in a <think>...</think> block before producing the actual answer. llama.cpp separates that into a reasoning_content field, leaving content empty until the </think> tag appears.
For a “summarize in one sentence” prompt, the 4B model happily burned all 1024 tokens thinking about how to summarize and never reached the actual answer. The block never closed. content stayed empty. My script printed the empty string.
The fix is a single line in the request body:
"chat_template_kwargs": {"enable_thinking": False}
That’s already in the script above. If you’re adapting the pattern for another reasoning-capable model, this is the gotcha to watch for. The symptom (response with tokens generated but empty content) is identical across most reasoning models I’ve tried.
What this is actually good for
I want to be honest, because Medium is full of “I replaced Claude with a local model” posts that are either lying or measuring the wrong thing.
This is good for: summarizing pasted text, rewording sentences, converting prose to bullets, translating short snippets, generating naming variants, simple classification, and bulk operations where you’d otherwise spend Claude tokens on repetitive busywork.
This is not good for: anything code-related, anything requiring multi-step reasoning, anything that depends on understanding your codebase, anything where being subtly wrong has consequences. Qwen3.5–4B is a 4B parameter model. It’s not Claude. It’s not even Haiku. Treat it accordingly.
What about cost savings? Marginal, and probably not the right reason to do this. If you’re on a flat-rate Claude plan, you save nothing. If you’re on API and you delegate, say, 50 small tasks a day at maybe 500 tokens each, you’re saving cents. The real value isn’t dollars — it’s that your GPU does something useful, you get fast responses for trivial tasks (no network round-trip), and the pattern works offline.
Hidden cost: the model has to be running. If you forget to start llama-server, the skill fails gracefully but Claude has to fall back to doing the task itself. Not a disaster, but worth knowing.
The honest pitch: this is a worker pool for grunt work, not a Claude replacement. Anyone selling you the latter is either misinformed or measuring success on toy benchmarks.
Why the pattern matters beyond Qwen
Once you’ve built one of these bridges, you realize the Skill + Bash composition is surprisingly general. You can delegate to anything that listens on a port:
The Skill description tells Claude when to invoke. The Bash tool gives it the means. localhost gives it the destination. No MCP, no proxy, no config gymnastics. You can wire a new external capability into Claude Code in about an hour.
That’s the real takeaway. Qwen is just the example.
Repo and what to try first
I’ve published the skill files on GitHub:
git clone https://github.com/anmerino-pnd/qwen-local-assistant.git.
Clone, drop into ~/.claude/skills/, restart Claude Code.
Two things to try once it’s running:
If something breaks, the most likely culprits, in order:
Tell me what breaks. The pattern is more interesting than the specific implementation, and there are probably better skill descriptions and better delegation heuristics than the ones I shipped.
How I Taught Claude Code to Offload Grunt Work to a Local 4B Model was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.