Run Claude Code on a self-hosted uncensored Qwen model (Vast.ai + llama-server)

A developer published a gist showing how to point the Claude Code CLI at a self-hosted, abliterated Qwen 35B MoE model instead of the Anthropic API, running on a rented Vast.ai A100 GPU and served by llama-server with --jinja, which speaks the Anthropic Messages API directly with no proxy in between. The setup preserves tool calling, multi-turn conversation and 262k context, and was measured at 141 tok/s generation at short context, 112 tok/s at 104k context, and 4505 tok/s prompt eval. The guide includes a patched chat template, zsh management functions, and a troubleshooting table, and is intended for authorized pentest work.

Point the Claude Code CLI at a self-hosted, abliterated Qwen model instead of the Anthropic API. The model runs on a rented Vast.ai GPU and is served by llama-server . There is no router or proxy in between: llama-server --jinja speaks the Anthropic Messages API /v1/messages directly, which is what Claude Code talks. Result: full Claude Code tool calling, multi-turn, 262k context on an uncensored 35B MoE model. Intended for authorized pentest work. Measured on an A100 PCIE 40 GB: 141 tok/s generation at short context, 112 tok/s at 104k context, 4505 tok/s prompt eval. Files in this gist: - README.md - this guide, setup steps, and a troubleshooting table for every error found while building this - qwen-chat-template.jinja - the patched chat template required, do not skip it - claude-qwen.zsh - the zsh functions that manage everything php Claude Code CLI ----HTTP /v1/messages---- ssh tunnel =====ssh===== llama-server Vast.ai box local machine 127.0.0.1:11435 11435 - 8080 :8080, model in RAM - llama-server listens on the box on port 8080. - A local ssh tunnel forwards 127.0.0.1:11435 to that port. - Claude Code talks to http://127.0.0.1:11435 as if it were the Anthropic API. - llama-server is started with --jinja , which makes it accept the Anthropic /v1/messages request and response format directly. No translation layer. Every placeholder in this guide is one of these, and each section says where to get it: | Placeholder | Meaning | How to get it | |---|---|---| | <MACHINE ID | The GPU host you rent on Vast | vastai search offers , or the machine card in the Vast web UI | | <INSTANCE ID | Your rented instance one machine can host many | The id Vast prints after vastai create instance succeeds | | <HOST , <PORT | ssh address of that instance | vastai ssh-url <INSTANCE ID prints ssh://root@<HOST :<PORT | Local machine needs these. Run each check and make sure it prints something. zsh --version the function file is zsh ssh -V for the tunnel and remote commands scp -V for the template upload note: -V prints usage, that is fine curl -V for the /health checks python3 -V the function parses JSON with python3 no jq needed vastai --version the Vast CLI vastai also needs a login: pip install vastai or pipx install vastai vastai set api-key <KEY key from the Vast web UI - Account - API Keys vastai show instances must print a possibly empty table, not an auth error If you do not have pip , any recent Python includes it via python3 -m pip . No HuggingFace token is needed: the model repo used below is ungated. The model is ~31 GB Q6 K P , so rent a 40 GB GPU or larger. An A100 PCIE 40 GB is the one this was measured on, and it loads at 37.1 of 40.9 GB, so 40 GB is the practical floor. vastai create instance <MACHINE ID \ --image ghcr.io/ggml-org/llama.cpp:server-cuda \ --disk 60 --ssh --direct --label claude-qwen - Find <MACHINE ID on the Vast web UI search "A100", pick a machine with 40 GB or more or with vastai search offers '.gpu ram = 40' see vastai search offers --help for filter syntax in your CLI version . Sort by gpu mem bw rather than by GPU name: Vast measures that field per machine, and it is a benchmark, not the datasheet number. - --label claude-qwen is required: the function finds the instance by this exact label. - --ssh and --direct give a direct ssh port, which the function needs. When it finishes, get the address: vastai ssh-url <INSTANCE ID prints something like: ssh://root@12.34.56.78:12345 That URL is <HOST = 12.34.56.78 , <PORT = 12345 . Known trap: Vast's --ssh wrapper ignores the --args you pass at create time. It never auto-launches the server, no matter what --args you gave. Plan to launch it yourself, which step 3 does. This file is required. The stock Qwen template breaks see troubleshooting row 1 . scp -P <PORT qwen-chat-template.jinja root@<HOST :/root/qwen.jinja The model has to keep serving after you disconnect. Use tmux, which is what the function does. A plain nohup ... & over ssh gets killed when the session closes. setsid nohup ... < /dev/null & does survive, but tmux is easier to attach to and read the log from. Write the launch command to a script on the box first the trailing heredoc goes to ssh's stdin , then start it in tmux: bash ssh -p <PORT root@<HOST 'cat /root/run-llama.sh' <<'EOF' /bin/sh cd /app || exit 1 export LLAMA CACHE=/root/models ./llama-server \ -hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6 K P \ --jinja --chat-template-file /root/qwen.jinja \ --reasoning-budget 0 --reasoning-format none \ -ngl 999 -c 262144 -fa on \ -ctk f16 -ctv f16 \ -b 8192 -ub 2048 -np 1 \ --host 0.0.0.0 --port 8080 2 &1 | tee /root/llama.log EOF ssh -p <PORT root@<HOST 'command -v tmux /dev/null || apt-get install -y -qq tmux; tmux has-session -t llama 2 /dev/null || tmux new-session -d -s llama sh /root/run-llama.sh' This is exactly what claude-qwen-server does, so once you source the function you can skip this by-hand step. Confirm it is up this also happens automatically when you run claude-qwen : ssh -p <PORT root@<HOST 'curl -sf http://localhost:8080/health' If this is empty or fails, watch the log: ssh -p <PORT root@<HOST 'tail -f /root/llama.log' . Notes on the flags: - -hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6 K P is ungated, so no HuggingFace token. Q6 K P is a "Perfect" quant: a profile tuned for that one model, 7.07 bits per weight against 4.88 for the plain Q4 K M . - The model downloads on first launch into /root/models LLAMA CACHE . 31 GB takes about 8 minutes. Wait for /health to return. - --reasoning-budget 0 --reasoning-format none disables the model's thinking output the thinking section . Claude Code does not read it. - -c 262144 is the context size. Step 5 sets the matching client variable. llama.cpp allocates the KV cache for the whole context at load, so this is a load-time VRAM cost, not something that grows while you chat. - -fa on is flash attention; note it now needs an explicit value, -fa alone errors. - -ngl 999 puts all layers on the GPU. - -ctk f16 -ctv f16 keeps the KV cache at full precision. This is the counter-intuitive one: the 8 bit cache is the SLOWER one. At 104k context f16 gives 106.0 tok/s, q8 0 gives 74.3, q4 0 gives 70.1. The attention kernel dequantizes the cache on every token, and that costs more than the bytes it saves. Weights plus f16 KV load at 37.1 of 40.9 GB, so it fits on a 40 GB card. - -b 8192 -ub 2048 raise the prompt-processing batch size. Prompt processing is compute bound, so bigger batches help: +11% at 104k context. It does nothing for generation. - -np 1 keeps one slot. The default is 4 slots, which splits -c 262144 into 4 x 65536 while Claude Code is still told 262144, so a long session overflows its slot. For a dense model, generation is memory-bandwidth bound, not compute bound. Each generated token reads every weight once plus the whole KV cache, so tokens per second is roughly bandwidth / weights + KV , and a card running at its rated bandwidth is already near its ceiling. Measured on a dense 27B Q6 K 23.46 GB : | GPU | measured bandwidth | generation | prompt processing | |---|---|---|---| | RTX A6000 48 GB | 572.5 GB/s | 17.3 t/s | 558 t/s | | A100 PCIE 40 GB | 1314.5 GB/s | 39.8 t/s | 1297 t/s | The 2.30x bandwidth ratio shows up almost exactly as 2.3x on both numbers. Vast's gpu mem bw on the offer is a benchmark, not the datasheet number, and it is the field worth sorting by. This MoE model breaks that rule, which is the whole reason to run one. Only about 3B of its 35B parameters are active per token, so the card reads far less and finishes the token while the memory bus is idle: measured 239 GB/s of the A100's 1314, about 18%. Generation here is latency bound, not bandwidth bound. Two things follow, both measured on the A100 with this model: - A bigger quant is free, and slightly faster. Q6 K P at 7.07 bits per weight beats Q4 K M at 4.88: 140.9 against 134.6 tok/s at short context, and 112.1 against 106.2 at 104k. The extra bytes ride on bandwidth that was not being used anyway. So pick the largest quant that fits in VRAM, not the smallest that fits. - The KV cache is faster at full precision, for the reason in the flag notes above. Prompt processing behaves normally: it is compute bound, so a bigger batch helps and the KV type barely matters. Two caveats. The first request after a boot pays roughly 36 s of CUDA warmup before it streams; every request after that is clean. And the prompt cache does work: re-sending the same 104k prompt drops the prompt token count from 104015 to 4 and prompt processing from 22637 ms to 99.6 ms. In a long Claude Code session only the new tokens get processed, so generation speed is the real limit, not prompt processing. One flag that does nothing here: --cache-reuse is silently disabled because the vision mmproj is loaded. The server log says cache reuse is not supported by multimodal, it will be disabled . It is left out of the launch command for that reason. Put claude-qwen.zsh somewhere and source it from ~/.zshrc : source /path/to/claude-qwen.zsh The file defines one command plus one function per concern: - claude-qwen-instance - Vast.ai: find the instance by label, boot it if stopped, print host port - claude-qwen-server - the box: make sure llama-server is serving uploads the template and re-launches tmux if missing or down - claude-qwen-tunnel - the ssh tunnel 11435 - 8080 , then wait until /health answers - claude-qwen - glue the three together and run Claude Code claude-qwen runs the whole lifecycle, so after the first run you never touch the instance by hand: 1. Find the instance by its claude-qwen label. 2. Start it if stopped. 3. Make sure llama-server is up re-launch in tmux if /health fails, re-upload the template if missing . 4. Open the tunnel 127.0.0.1:11435 - 8080 . 5. Run Claude Code against http://127.0.0.1:11435 . Two paths are hardcoded and must match, or change them to taste: - The local template is read from ~/.claude/qwen-chat-template.jinja . Keep the file there, or edit the scp line in claude-qwen-server . - The remote template lives at /root/qwen.jinja and the launch script the function writes is /root/run-llama.sh . Ports 11435 local and 8080 remote are also fixed; change them in both claude-qwen-tunnel and the --settings URL if they collide. claude-qwen interactive claude-qwen -p "reply with exactly: pong" one-shot smoke test Pass: the one-shot prints pong . Pass: an interactive session makes a tool call for example ask it to list the current directory and it runs. - CLAUDE CODE MAX CONTEXT TOKENS=262144 - must match the server's -c 262144 . Claude Code assumes 200k for unknown models, sends too much, and the server rejects with Context size has been exceeded . - ANTHROPIC MODEL=qwen-uncensored plus ANTHROPIC DEFAULT OPUS MODEL , ANTHROPIC DEFAULT SONNET MODEL , ANTHROPIC DEFAULT HAIKU MODEL all set to the same name - without them Claude Code prompts for a real Anthropic model. - ANTHROPIC AUTH TOKEN=dummy , ANTHROPIC API KEY= empty - the local server does no auth; Claude Code still wants a token field present. - The --settings JSON forces ANTHROPIC BASE URL . Your ~/.claude/settings.json env block wins over shell env, so setting the URL only in the shell does not work. - CLAUDE CODE ASSUME FIRST PARTY BASE URL=0 inside --settings - stops Claude Code treating the URL as a first-party endpoint. - command claude - runs the real binary, skipping any shell alias or wrapper named claude . - ~$0.63/hr to run, ~$15/day if left on. - vastai stop instance <INSTANCE ID - stops billing, keeps the disk and downloaded model. - vastai start instance <INSTANCE ID - resumes; the function re-launches the server for you. - vastai destroy instance <INSTANCE ID - wipes it; you then re-rent and re-download. Every error below was hit while building and running this. Symptom is the exact text; fix is what resolved it. | Symptom | Cause | Fix | |---|---|---| | System message must be at the beginning on a request | The stock Qwen template requires the system message at position 0, but Claude Code sends a system -role message mid-array roles user, system, assistant, user , usually around tool output | Use the patched qwen-chat-template.jinja from this gist and launch with --chat-template-file /root/qwen.jinja . It renders a mid-array system block inline instead of erroring. | | Every request 500s with Jinja Exception: Unexpected item type in content. | Claude Code's ToolSearch returns a tool result whose content is a list of {"type": "tool reference", "tool name": "Bash"} blocks. The stock render content else branch rejects anything that is not text, image or video, and tool reference is none of those. | Use the patched qwen-chat-template.jinja from this gist: it renders tool reference as empty and any other unknown block as unsupported block: <type instead of raising. To find which block it was, read the request bodies Claude Code saved under ~/.claude/projects/<slug / .jsonl records of type api-request-blob ; llama-server never logs the payload. | | pkill -f llama-server kills the shell running it | The pattern also matches the ssh command line, which contains that exact string | Use pkill -f " l lama-server" . The bracket stops the pattern matching itself. | | cache reuse is not supported by multimodal, it will be disabled in the log | The vision mmproj is loaded, so --cache-reuse does nothing at all | Not a fault. Leave the flag out. The built-in prompt cache still works. | | Server responds Context size has been exceeded on long requests | Server runs -c 262144 , but Claude Code assumes 200k for unknown models and sends 200k+ | Set CLAUDE CODE MAX CONTEXT TOKENS=262144 on the client so it compacts before overflow. | | Claude Code prompts to pick a real Anthropic model | Only ANTHROPIC MODEL was set; the tier defaults were not | Set all three ANTHROPIC DEFAULT MODEL vars to the same model name. | | ANTHROPIC BASE URL set as a shell env var is ignored | ~/.claude/settings.json env block wins over shell env | Pass the base URL through --settings '{"env":{"ANTHROPIC BASE URL":"..."}}' , as the function does. | | Server dies the moment the ssh session closes | A plain nohup ... & over ssh gets killed along with the session | Run the server inside tmux session llama , as in step 3. setsid nohup ... < /dev/null & also survives, but tmux lets you attach and read the log. | | Instance is up but nothing listens on 8080 | Vast's --ssh wrapper ignores the --args you passed at create time | Launch llama-server yourself step 3 . The function does this automatically on each run. | | pgrep -f llama-server says it is running when it is not | pgrep -f matches your own ssh command string | Never decide "up" from a process list. Check curl -sf http://localhost:8080/health . The function checks only /health . | | Responses contain a thinking block | Reasoning/thinking output is on by default | Add --reasoning-budget 0 --reasoning-format none . | | llama-server errors on -fa | Flash attention now requires an explicit value | Use -fa on . | If a start call in the function fails, read the remote log it points to: ssh -p <PORT root@<HOST 'tail -50 /root/llama.log' .