Point the Claude Code CLI at a self-hosted, abliterated Qwen model instead of the Anthropic API. The model runs on a rented Vast.ai GPU and is served by llama-server. There is no router or proxy in between: llama-server --jinja speaks the Anthropic Messages API (/v1/messages) directly, which is what Claude Code talks.
Result: full Claude Code (tool calling, multi-turn, 262k context) on an uncensored 35B MoE model. Intended for authorized pentest work. Measured on an A100 PCIE 40 GB: 141 tok/s generation at short context, 112 tok/s at 104k context, 4505 tok/s prompt eval.
Files in this gist:
README.md- this guide, setup steps, and a troubleshooting table for every error found while building thisqwen-chat-template.jinja- the patched chat template (required, do not skip it)claude-qwen.zsh- the zsh functions that manage everything
Claude Code CLI ----HTTP /v1/messages----> ssh tunnel =====ssh=====> llama-server (Vast.ai box)
(local machine) 127.0.0.1:11435 11435 -> 8080 :8080, model in RAM
llama-serverlistens on the box on port 8080.- A local ssh tunnel forwards
127.0.0.1:11435to that port. - Claude Code talks to
http://127.0.0.1:11435as if it were the Anthropic API. llama-serveris started with--jinja, which makes it accept the Anthropic/v1/messagesrequest and response format directly. No translation layer.
Every placeholder in this guide is one of these, and each section says where to get it:
| Placeholder | Meaning | How to get it |
|---|---|---|
<MACHINE_ID> |
The GPU host you rent on Vast | vastai search offers , or the machine card in the Vast web UI |
<INSTANCE_ID> |
Your rented instance (one machine can host many) | The id Vast prints after vastai create instance succeeds |
<HOST> ,<PORT> |
ssh address of that instance | vastai ssh-url <INSTANCE_ID> printsssh://root@<HOST>:<PORT> |
Local machine needs these. Run each check and make sure it prints something.
zsh --version # the function file is zsh
ssh -V # for the tunnel and remote commands
scp -V # for the template upload (note: -V prints usage, that is fine)
curl -V # for the /health checks
python3 -V # the function parses JSON with python3 (no jq needed)
vastai --version # the Vast CLI
vastai also needs a login:
pip install vastai # or pipx install vastai
vastai set api-key <KEY> # key from the Vast web UI -> Account -> API Keys
vastai show instances # must print a (possibly empty) table, not an auth error
If you do not have pip, any recent Python includes it via python3 -m pip.
No HuggingFace token is needed: the model repo used below is ungated.
The model is ~31 GB (Q6_K_P), so rent a 40 GB GPU or larger. An A100 PCIE 40 GB is the one this was measured on, and it loads at 37.1 of 40.9 GB, so 40 GB is the practical floor.
vastai create instance <MACHINE_ID> \
--image ghcr.io/ggml-org/llama.cpp:server-cuda \
--disk 60 --ssh --direct --label claude-qwen
- Find
<MACHINE_ID>on the Vast web UI (search "A100", pick a machine with 40 GB or more) or withvastai search offers '.gpu_ram >= 40'(seevastai search offers --helpfor filter syntax in your CLI version). Sort bygpu_mem_bwrather than by GPU name: Vast measures that field per machine, and it is a benchmark, not the datasheet number. --label claude-qwenis required: the function finds the instance by this exact label.--sshand--directgive a direct ssh port, which the function needs.
When it finishes, get the address:
vastai ssh-url <INSTANCE_ID>
That URL is <HOST> = 12.34.56.78, <PORT> = 12345.
Known trap: Vast's --ssh wrapper ignores the --args you pass at create time. It never auto-launches the server, no matter what --args you gave. Plan to launch it yourself, which step 3 does.
This file is required. The stock Qwen template breaks (see troubleshooting row 1).
scp -P <PORT> qwen-chat-template.jinja root@<HOST>:/root/qwen.jinja
The model has to keep serving after you disconnect. Use tmux, which is what the function does. A plain nohup ... & over ssh gets killed when the session closes. setsid nohup ... < /dev/null & does survive, but tmux is easier to attach to and read the log from.
Write the launch command to a script on the box first (the trailing heredoc goes to ssh's stdin), then start it in tmux:
ssh -p <PORT> root@<HOST> 'cat > /root/run-llama.sh' <<'EOF'
#!/bin/sh
cd /app || exit 1
export LLAMA_CACHE=/root/models
./llama-server \
-hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_P \
--jinja --chat-template-file /root/qwen.jinja \
--reasoning-budget 0 --reasoning-format none \
-ngl 999 -c 262144 -fa on \
-ctk f16 -ctv f16 \
-b 8192 -ub 2048 -np 1 \
--host 0.0.0.0 --port 8080 2>&1 | tee /root/llama.log
EOF
ssh -p <PORT> root@<HOST> 'command -v tmux >/dev/null || apt-get install -y -qq tmux; tmux has-session -t llama 2>/dev/null || tmux new-session -d -s llama sh /root/run-llama.sh'
This is exactly what claude-qwen-server does, so once you source the function you can skip this by-hand step.
Confirm it is up (this also happens automatically when you run claude-qwen):
ssh -p <PORT> root@<HOST> 'curl -sf http://localhost:8080/health'
If this is empty or fails, watch the log: ssh -p <PORT> root@<HOST> 'tail -f /root/llama.log'.
Notes on the flags:
-hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_Pis ungated, so no HuggingFace token.Q6_K_Pis a "Perfect" quant: a profile tuned for that one model, 7.07 bits per weight against 4.88 for the plainQ4_K_M.- The model downloads on first launch into
/root/models(LLAMA_CACHE). 31 GB takes about 8 minutes. Wait for/healthto return. --reasoning-budget 0 --reasoning-format nonedisables the model's thinking output (thethinkingsection). Claude Code does not read it.-c 262144is the context size. Step 5 sets the matching client variable. llama.cpp allocates the KV cache for the whole context at load, so this is a load-time VRAM cost, not something that grows while you chat.-fa onis flash attention; note it now needs an explicit value,-faalone errors.-ngl 999puts all layers on the GPU.-ctk f16 -ctv f16keeps the KV cache at full precision. This is the counter-intuitive one: the 8 bit cache is the SLOWER one. At 104k context f16 gives 106.0 tok/s,q8_0gives 74.3,q4_0gives 70.1. The attention kernel dequantizes the cache on every token, and that costs more than the bytes it saves. Weights plus f16 KV load at 37.1 of 40.9 GB, so it fits on a 40 GB card.-b 8192 -ub 2048raise the prompt-processing batch size. Prompt processing is compute bound, so bigger batches help: +11% at 104k context. It does nothing for generation.-np 1keeps one slot. The default is 4 slots, which splits-c 262144into 4 x 65536 while Claude Code is still told 262144, so a long session overflows its slot.
For a dense model, generation is memory-bandwidth bound, not compute bound. Each generated token reads every weight once plus the whole KV cache, so tokens per second is roughly bandwidth / (weights + KV), and a card running at its rated bandwidth is already near its ceiling. Measured on a dense 27B Q6_K (23.46 GB):
| GPU | measured bandwidth | generation | prompt processing |
|---|---|---|---|
| RTX A6000 48 GB | 572.5 GB/s | 17.3 t/s | 558 t/s |
| A100 PCIE 40 GB | 1314.5 GB/s | 39.8 t/s | 1297 t/s |
The 2.30x bandwidth ratio shows up almost exactly as 2.3x on both numbers. Vast's gpu_mem_bw on the offer is a benchmark, not the datasheet number, and it is the field worth sorting by.
This MoE model breaks that rule, which is the whole reason to run one. Only about 3B of its 35B parameters are active per token, so the card reads far less and finishes the token while the memory bus is idle: measured 239 GB/s of the A100's 1314, about 18%. Generation here is latency bound, not bandwidth bound.
Two things follow, both measured on the A100 with this model:
- A bigger quant is free, and slightly faster. Q6_K_P at 7.07 bits per weight beats Q4_K_M at 4.88: 140.9 against 134.6 tok/s at short context, and 112.1 against 106.2 at 104k. The extra bytes ride on bandwidth that was not being used anyway. So pick the largest quant that fits in VRAM, not the smallest that fits.
- The KV cache is faster at full precision, for the reason in the flag notes above.
Prompt processing behaves normally: it is compute bound, so a bigger batch helps and the KV type barely matters.
Two caveats. The first request after a boot pays roughly 36 s of CUDA warmup before it streams; every request after that is clean. And the prompt cache does work: re-sending the same 104k prompt drops the prompt token count from 104015 to 4 and prompt processing from 22637 ms to 99.6 ms. In a long Claude Code session only the new tokens get processed, so generation speed is the real limit, not prompt processing.
One flag that does nothing here: --cache-reuse is silently disabled because the vision mmproj is loaded. The server log says cache_reuse is not supported by multimodal, it will be disabled. It is left out of the launch command for that reason.
Put claude-qwen.zsh somewhere and source it from ~/.zshrc:
source /path/to/claude-qwen.zsh
The file defines one command plus one function per concern:
claude-qwen-instance- Vast.ai: find the instance by label, boot it if stopped, printhost portclaude-qwen-server- the box: make surellama-serveris serving (uploads the template and re-launches tmux if missing or down)claude-qwen-tunnel- the ssh tunnel11435 -> 8080, then wait until/healthanswersclaude-qwen- glue the three together and run Claude Code
claude-qwen runs the whole lifecycle, so after the first run you never touch the instance by hand:
- Find the instance by its
claude-qwenlabel. - Start it if stopped.
- Make sure
llama-serveris up (re-launch in tmux if/healthfails, re-upload the template if missing). - Open the tunnel
127.0.0.1:11435 -> 8080. - Run Claude Code against
http://127.0.0.1:11435.
Two paths are hardcoded and must match, or change them to taste:
- The local template is read from
~/.claude/qwen-chat-template.jinja. Keep the file there, or edit thescpline inclaude-qwen-server. - The remote template lives at
/root/qwen.jinjaand the launch script the function writes is/root/run-llama.sh.
Ports 11435 (local) and 8080 (remote) are also fixed; change them in both claude-qwen-tunnel and the --settings URL if they collide.
claude-qwen # interactive
claude-qwen -p "reply with exactly: pong" # one-shot smoke test
Pass: the one-shot prints pong. Pass: an interactive session makes a tool call (for example ask it to list the current directory) and it runs.
-
CLAUDE_CODE_MAX_CONTEXT_TOKENS=262144- must match the server's-c 262144. Claude Code assumes 200k for unknown models, sends too much, and the server rejects withContext size has been exceeded. -
ANTHROPIC_MODEL=qwen-uncensoredplusANTHROPIC_DEFAULT_OPUS_MODEL,ANTHROPIC_DEFAULT_SONNET_MODEL,ANTHROPIC_DEFAULT_HAIKU_MODELall set to the same name - without them Claude Code prompts for a real Anthropic model. -
ANTHROPIC_AUTH_TOKEN=dummy,ANTHROPIC_API_KEY=(empty) - the local server does no auth; Claude Code still wants a token field present. -
The
--settingsJSON forcesANTHROPIC_BASE_URL. Your~/.claude/settings.json`` envblock wins over shell env, so setting the URL only in the shell does not work. -
_CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL=0inside--settings- stops Claude Code treating the URL as a first-party endpoint. -
command claude- runs the real binary, skipping any shell alias or wrapper namedclaude. -
~$0.63/hr to run, ~$15/day if left on.
-
vastai stop instance <INSTANCE_ID>- stops billing, keeps the disk and downloaded model. -
vastai start instance <INSTANCE_ID>- resumes; the function re-launches the server for you. -
vastai destroy instance <INSTANCE_ID>- wipes it; you then re-rent and re-download.
Every error below was hit while building and running this. Symptom is the exact text; fix is what resolved it.
| Symptom | Cause | Fix |
|---|---|---|
System message must be at the beginning on a request |
The stock Qwen template requires the system message at position 0, but Claude Code sends a system -role message mid-array (roles[user, system, assistant, user] , usually around tool output) |
Use the patched qwen-chat-template.jinja from this gist and launch with--chat-template-file /root/qwen.jinja . It renders a mid-array system block inline instead of erroring. |
Every request 500s with Jinja Exception: Unexpected item type in content. |
Claude Code's ToolSearch returns a tool_result whosecontent is a list of{"type": "tool_reference", "tool_name": "Bash"} blocks. The stockrender_content`` else branch rejects anything that is not text, image or video, andtool_reference is none of those. |
Use the patched qwen-chat-template.jinja from this gist: it renderstool_reference as empty and any other unknown block as[unsupported block: <type>] instead of raising. To find which block it was, read the request bodies Claude Code saved under~/.claude/projects/<slug>/*.jsonl (records of typeapi-request-blob );llama-server never logs the payload. |
pkill -f llama-server kills the shell running it |
The pattern also matches the ssh command line, which contains that exact string | Use pkill -f "[l]lama-server" . The bracket stops the pattern matching itself. |
cache_reuse is not supported by multimodal, it will be disabled in the log |
The vision mmproj is loaded, so--cache-reuse does nothing at all |
Not a fault. Leave the flag out. The built-in prompt cache still works. |
Server responds Context size has been exceeded on long requests |
Server runs -c 262144 , but Claude Code assumes 200k for unknown models and sends 200k+ |
Set CLAUDE_CODE_MAX_CONTEXT_TOKENS=262144 on the client so it compacts before overflow. |
| Claude Code prompts to pick a real Anthropic model | Only ANTHROPIC_MODEL was set; the tier defaults were not |
Set all three ANTHROPIC_DEFAULT_*_MODEL vars to the same model name. |
ANTHROPIC_BASE_URL set as a shell env var is ignored |
~/.claude/settings.json`` env block wins over shell env |
Pass the base URL through --settings '{"env":{"ANTHROPIC_BASE_URL":"..."}}' , as the function does. |
| Server dies the moment the ssh session closes | A plain nohup ... & over ssh gets killed along with the session |
Run the server inside tmux (session llama ), as in step 3.setsid nohup ... < /dev/null & also survives, but tmux lets you attach and read the log. |
| Instance is up but nothing listens on 8080 | Vast's --ssh wrapper ignores the--args you passed at create time |
Launch llama-server yourself (step 3). The function does this automatically on each run. |
pgrep -f llama-server says it is running when it is not |
pgrep -f matches your own ssh command string |
Never decide "up" from a process list. Check curl -sf http://localhost:8080/health . The function checks only/health . |
Responses contain a thinking block |
Reasoning/thinking output is on by default | Add --reasoning-budget 0 --reasoning-format none . |
llama-server errors on-fa |
Flash attention now requires an explicit value | Use -fa on . |
If a start call in the function fails, read the remote log it points to: ssh -p <PORT> root@<HOST> 'tail -50 /root/llama.log'.