{"slug": "how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model", "title": "How I Taught Claude Code to Offload Grunt Work to a Local 4B Model", "summary": "A developer detailed a method to offload trivial text tasks from Anthropic's Claude Code to a local Qwen3.5-4B model running on llama.cpp, using a custom skill that calls a Python script via Bash. The setup, which avoids MCP servers and proxy configurations, lets Claude decide per-task when to delegate, reducing token usage while keeping Claude for complex reasoning. The author noted that subagents cannot use different models and that setting ANTHROPIC_BASE_URL globally downgrades the entire session.", "body_md": "My GPU sits idle most of the day. Meanwhile, I keep burning Claude tokens on tasks a small model could finish in two seconds: summarizing a paragraph, rewording a commit message, suggesting five variable names, classifying a list of lines.\n\nIt felt wasteful. So I tried to bridge them: keep Claude Code as my main driver, but let it delegate the trivial stuff to a Qwen3.5–4B running locally on llama.cpp.\n\nMy first three instincts were all wrong. The fourth worked. This post is the whole arc — including a bug that left me staring at a blank terminal for twenty minutes — so you can skip straight to the working setup.\n\nIf you’re already using Claude Code and have heard about local LLMs but never wired one in, this is for you.\n\nBefore I show what works, here’s what most people try first. Skip these so you don’t waste your afternoon.\n\n**“I’ll just make a subagent that uses Qwen.”** You can’t. In Claude Code, subagents run on the same model as the main session. The model: field in subagent frontmatter accepts names like haiku or sonnet, but they all resolve against the same ANTHROPIC_BASE_URL. Subagents are context isolation, not model isolation.\n\n**“I’ll point ****ANTHROPIC_BASE_URL at my llama-server.\"** This *does* work — llama.cpp exposes an Anthropic-compatible /v1/messages endpoint, and Claude Code will happily talk to it. But it's all-or-nothing. The moment you set that env var, *everything* goes through Qwen, including the heavy reasoning tasks where you actually need Claude. You haven't built a hybrid; you've just downgraded your whole session.\n\n**“I’ll add multiple endpoints to settings.json, like opencode does.”** Opencode and qwen-code support per-model endpoints in their config. Claude Code does not. Don’t go looking for a flag — it isn’t there.\n\nThe real answer is to stop thinking about this at the *model* level and start thinking at the *application* level. Claude stays in charge. Qwen becomes a tool Claude can call when it judges the task to be trivial.\n\nClaude Code has two primitives that, combined, give you exactly what you want:\n\nGlue them: write a Skill whose instructions are *“when you see a small text task, call this Python script via Bash; the script hits a local llama-server and returns the result.”*\n\nThat’s it. No MCP server to maintain. No proxy. No config gymnastics. Claude makes the routing decision per-task, based on the skill’s description.\n\nWhy not an MCP server? You could. MCP is the “right” answer if you want a persistent, structured tool. But for a single endpoint that takes text in and returns text out, a 60-line Python script is faster to write, easier to debug, and has fewer failure modes. MCP is overkill here.\n\n**Prerequisites**\n\n**Step 1: Run llama-server**\n\nWhatever flags you normally use are fine. For reference, mine on Windows:\n\n```\nllama-server.exe ^  -m gguf\\Qwen3.5-4B\\Qwen3.5-4B-UD-Q4_K_XL.gguf ^  --jinja ^  -c 32768 ^  --port 8080 ^  -a Qwen3.5-4B\n```\n\nThe two flags that matter for this setup:\n\nVerify it’s alive:\n\n```\ncurl http://127.0.0.1:8080/v1/models\n```\n\n**Step 2: Create the skill**\n\nSkills live at ~/.claude/skills/<name>/ on macOS/Linux, or C:\\Users\\<you>\\.claude\\skills\\<name>\\ on Windows. Create the folder qwen-local-assistant/ with two files.\n\n**SKILL.md** — the file Claude reads to know when and how to delegate:\n\n```\n---name: qwen-local-assistantdescription: Offload small text tasks (summarize, reword, translate, prose-to-list, name suggestions, classify, simple boilerplate) to a local Qwen3.5-4B running on llama.cpp at http://127.0.0.1:8080. ALWAYS use this skill when the user wants a self-contained text transformation that doesn't require codebase context or multi-step reasoning. Do NOT use for code edits, debugging, codebase-aware tasks, or anything where being wrong is costly.---# Qwen Local AssistantYou have a local Qwen3.5-4B model running on the user's machine athttp://127.0.0.1:8080. Use it as a cheap helper for low-stakes text workvia the `ask_qwen.py` script in this skill directory.## When to delegateDelegate when ALL of these are true:- Text in, text out (no file edits, no code semantics)- Self-contained - the prompt is short and stands on its own- Low stakes - a slightly off output is acceptable- It's tedious busywork## How to callPipe the prompt through stdin to avoid quote-escaping issues:    echo \"Summarize in 2 sentences: <text>\" | python <skill_dir>/ask_qwen.py## After it respondsRead critically. Qwen3.5-4B can hallucinate. Sanity-check beforeintegrating. When showing the user, mention it came from the local model.\n```\n\n**ask_qwen.py** — the bridge script:\n\n``` bash\n#!/usr/bin/env python3import sys, json, argparse, urllib.request, urllib.errorDEFAULT_URL = \"http://127.0.0.1:8080/v1/chat/completions\"DEFAULT_MODEL = \"Qwen3.5-4B\"def main():    p = argparse.ArgumentParser()    p.add_argument(\"prompt\", nargs=\"?\", default=None)    p.add_argument(\"--system\", default=None)    p.add_argument(\"--max-tokens\", type=int, default=2048)    p.add_argument(\"--temperature\", type=float, default=0.3)    p.add_argument(\"--url\", default=DEFAULT_URL)    p.add_argument(\"--model\", default=DEFAULT_MODEL)    p.add_argument(\"--timeout\", type=int, default=180)    p.add_argument(\"--enable-thinking\", action=\"store_true\")    args = p.parse_args()    prompt = args.prompt if args.prompt else sys.stdin.read()    if not prompt.strip():        print(\"ERROR: empty prompt\", file=sys.stderr); return 2    messages = []    if args.system:        messages.append({\"role\": \"system\", \"content\": args.system})    messages.append({\"role\": \"user\", \"content\": prompt})    body = json.dumps({        \"model\": args.model,        \"messages\": messages,        \"max_tokens\": args.max_tokens,        \"temperature\": args.temperature,        \"stream\": False,        \"chat_template_kwargs\": {\"enable_thinking\": args.enable_thinking},    }).encode(\"utf-8\")    req = urllib.request.Request(        args.url, data=body,        headers={\"Content-Type\": \"application/json\"}, method=\"POST\")    try:        with urllib.request.urlopen(req, timeout=args.timeout) as r:            data = json.loads(r.read().decode(\"utf-8\"))    except urllib.error.URLError as e:        print(f\"ERROR: {e}\", file=sys.stderr); return 1    msg = data[\"choices\"][0][\"message\"]    content = (msg.get(\"content\") or \"\").strip()    if not content:        reasoning = (msg.get(\"reasoning_content\") or \"\").strip()        if reasoning:            print(\"WARNING: only reasoning returned\", file=sys.stderr)            print(reasoning); return 0        print(\"ERROR: empty response\", file=sys.stderr); return 1    print(content)if __name__ == \"__main__\":    sys.exit(main())\n```\n\n**Step 3: Sanity-check from the terminal**\n\nBefore going through Claude Code, confirm the script works on its own:\n\n```\necho \"Summarize in one sentence: the cat climbed on the roof and meowed all night.\" | python ~/.claude/skills/qwen-local-assistant/ask_qwen.py\n```\n\nYou should get a one-line summary in about two seconds. If you get nothing, jump to the next section — you probably hit the bug I hit.\n\n**Step 4: Use it in Claude Code**\n\nRestart Claude Code so it picks up the new skill. Then prompt it with something like:\n\n“Take the changelog below and produce three user-facing bullets. Use the Qwen local assistant skill.”\n\nYou’ll see Claude invoke Bash, pipe the prompt to ask_qwen.py, get the response, and weave it into its reply. The first time it works it feels a little like cheating.\n\n**The bug that almost killed it**\n\nWhen I first wired this up, I ran the test command and got nothing. Blank line. The server log clearly showed a successful POST and 1024 tokens generated. But content in the response was empty.\n\n**The culprit:** Qwen3.5 has a reasoning mode. When --jinja is active and the chat template's enable_thinking flag is on (the default), the model wraps its output in a <think>...</think> block before producing the actual answer. llama.cpp separates that into a reasoning_content field, leaving content empty until the </think> tag appears.\n\nFor a “summarize in one sentence” prompt, the 4B model happily burned all 1024 tokens *thinking about how to summarize* and never reached the actual answer. The block never closed. content stayed empty. My script printed the empty string.\n\nThe fix is a single line in the request body:\n\n```\n\"chat_template_kwargs\": {\"enable_thinking\": False}\n```\n\nThat’s already in the script above. If you’re adapting the pattern for another reasoning-capable model, this is the gotcha to watch for. The symptom (response with tokens generated but empty content) is identical across most reasoning models I’ve tried.\n\n**What this is actually good for**\n\nI want to be honest, because Medium is full of “I replaced Claude with a local model” posts that are either lying or measuring the wrong thing.\n\n**This is good for:** summarizing pasted text, rewording sentences, converting prose to bullets, translating short snippets, generating naming variants, simple classification, and bulk operations where you’d otherwise spend Claude tokens on repetitive busywork.\n\n**This is not good for:** anything code-related, anything requiring multi-step reasoning, anything that depends on understanding your codebase, anything where being subtly wrong has consequences. Qwen3.5–4B is a 4B parameter model. It’s not Claude. It’s not even Haiku. Treat it accordingly.\n\n**What about cost savings?** Marginal, and probably not the right reason to do this. If you’re on a flat-rate Claude plan, you save nothing. If you’re on API and you delegate, say, 50 small tasks a day at maybe 500 tokens each, you’re saving cents. The real value isn’t dollars — it’s that your GPU does something useful, you get fast responses for trivial tasks (no network round-trip), and the pattern works offline.\n\n**Hidden cost:** the model has to be running. If you forget to start llama-server, the skill fails gracefully but Claude has to fall back to doing the task itself. Not a disaster, but worth knowing.\n\n**The honest pitch:** this is a worker pool for grunt work, not a Claude replacement. Anyone selling you the latter is either misinformed or measuring success on toy benchmarks.\n\n**Why the pattern matters beyond Qwen**\n\nOnce you’ve built one of these bridges, you realize the Skill + Bash composition is surprisingly general. You can delegate to anything that listens on a port:\n\nThe Skill description tells Claude when to invoke. The Bash tool gives it the means. localhost gives it the destination. No MCP, no proxy, no config gymnastics. You can wire a new external capability into Claude Code in about an hour.\n\nThat’s the real takeaway. Qwen is just the example.\n\n**Repo and what to try first**\n\nI’ve published the skill files on GitHub:\n\n```\ngit clone https://github.com/anmerino-pnd/qwen-local-assistant.git.\n```\n\nClone, drop into ~/.claude/skills/, restart Claude Code.\n\nTwo things to try once it’s running:\n\nIf something breaks, the most likely culprits, in order:\n\nTell me what breaks. The pattern is more interesting than the specific implementation, and there are probably better skill descriptions and better delegation heuristics than the ones I shipped.\n\n[How I Taught Claude Code to Offload Grunt Work to a Local 4B Model](https://pub.towardsai.net/claude-code-skill-local-qwen-3c6bb5d051bd) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model", "canonical_source": "https://pub.towardsai.net/claude-code-skill-local-qwen-3c6bb5d051bd?source=rss----98111c9905da---4", "published_at": "2026-08-27 00:01:03+00:00", "updated_at": "2026-08-27 00:20:07.681979+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models"], "entities": ["Anthropic", "Claude Code", "Qwen3.5-4B", "llama.cpp", "MCP"], "alternates": {"html": "https://wpnews.pro/news/how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model", "markdown": "https://wpnews.pro/news/how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model.md", "text": "https://wpnews.pro/news/how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model.txt", "jsonld": "https://wpnews.pro/news/how-i-taught-claude-code-to-offload-grunt-work-to-a-local-4b-model.jsonld"}}