# Run Claude Code on a self-hosted uncensored Qwen model (Vast.ai + llama-server)

> Source: <https://gist.github.com/iklobato/236c25f4022154cd54a225b84aaa3df9>
> Published: 2026-11-01 01:53:05+00:00

Point the Claude Code CLI at a self-hosted, abliterated Qwen model instead of the Anthropic API. The model runs on a rented Vast.ai GPU and is served by `llama-server`. There is no router or proxy in between: `llama-server --jinja` speaks the Anthropic Messages API (`/v1/messages`) directly, which is what Claude Code talks.

Result: full Claude Code (tool calling, multi-turn, 262k context) on an uncensored 35B MoE model. Intended for authorized pentest work. Measured on an A100 PCIE 40 GB: 141 tok/s generation at short context, 112 tok/s at 104k context, 4505 tok/s prompt eval.

Files in this gist:

- `README.md` - this guide, setup steps, and a troubleshooting table for every error found while building this
- `qwen-chat-template.jinja` - the patched chat template (required, do not skip it)
- `claude-qwen.zsh` - the zsh functions that manage everything

``` php
Claude Code CLI  ----HTTP /v1/messages---->  ssh tunnel  =====ssh=====>  llama-server (Vast.ai box)
(local machine)     127.0.0.1:11435         11435 -> 8080                :8080, model in RAM
```

- `llama-server` listens on the box on port 8080.
- A local ssh tunnel forwards `127.0.0.1:11435` to that port.
- Claude Code talks to `http://127.0.0.1:11435` as if it were the Anthropic API.
- `llama-server` is started with`--jinja` , which makes it accept the Anthropic`/v1/messages` request and response format directly. No translation layer.

Every placeholder in this guide is one of these, and each section says where to get it:

| Placeholder | Meaning | How to get it | 
|---|---|---|
| `<MACHINE_ID>` | The GPU host you rent on Vast | `vastai search offers` , or the machine card in the Vast web UI | 
| `<INSTANCE_ID>` | Your rented instance (one machine can host many) | The id Vast prints after `vastai create instance` succeeds | 
| `<HOST>` ,`<PORT>` | ssh address of that instance | `vastai ssh-url <INSTANCE_ID>` prints`ssh://root@<HOST>:<PORT>` | 

Local machine needs these. Run each check and make sure it prints something.

```
zsh --version        # the function file is zsh
ssh -V               # for the tunnel and remote commands
scp -V               # for the template upload (note: -V prints usage, that is fine)
curl -V              # for the /health checks
python3 -V           # the function parses JSON with python3 (no jq needed)
vastai --version     # the Vast CLI
```

`vastai` also needs a login:

```
pip install vastai          # or pipx install vastai
vastai set api-key <KEY>    # key from the Vast web UI -> Account -> API Keys
vastai show instances       # must print a (possibly empty) table, not an auth error
```

If you do not have `pip`, any recent Python includes it via `python3 -m pip`.

No HuggingFace token is needed: the model repo used below is ungated.

The model is ~31 GB (Q6_K_P), so rent a 40 GB GPU or larger. An A100 PCIE 40 GB is the one this was measured on, and it loads at 37.1 of 40.9 GB, so 40 GB is the practical floor.

```
vastai create instance <MACHINE_ID> \
  --image ghcr.io/ggml-org/llama.cpp:server-cuda \
  --disk 60 --ssh --direct --label claude-qwen
```

- Find `<MACHINE_ID>` on the Vast web UI (search "A100", pick a machine with 40 GB or more) or with`vastai search offers '.gpu_ram >= 40'` (see`vastai search offers --help` for filter syntax in your CLI version). Sort by`gpu_mem_bw` rather than by GPU name: Vast measures that field per machine, and it is a benchmark, not the datasheet number.
- `--label claude-qwen` is required: the function finds the instance by this exact label.
- `--ssh` and`--direct` give a direct ssh port, which the function needs.

When it finishes, get the address:

```
vastai ssh-url <INSTANCE_ID>
# prints something like:  ssh://root@12.34.56.78:12345
```

That URL is `<HOST>` = `12.34.56.78`, `<PORT>` = `12345`.

Known trap: Vast's `--ssh` wrapper ignores the `--args` you pass at create time. It never auto-launches the server, no matter what `--args` you gave. Plan to launch it yourself, which step 3 does.

This file is required. The stock Qwen template breaks (see troubleshooting row 1).

```
scp -P <PORT> qwen-chat-template.jinja root@<HOST>:/root/qwen.jinja
```

The model has to keep serving after you disconnect. Use tmux, which is what the function does. A plain `nohup ... &` over ssh gets killed when the session closes. `setsid nohup ... < /dev/null &` does survive, but tmux is easier to attach to and read the log from.

Write the launch command to a script on the box first (the trailing heredoc goes to ssh's stdin), then start it in tmux:

``` bash
ssh -p <PORT> root@<HOST> 'cat > /root/run-llama.sh' <<'EOF'
#!/bin/sh
cd /app || exit 1
export LLAMA_CACHE=/root/models
./llama-server \
  -hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_P \
  --jinja --chat-template-file /root/qwen.jinja \
  --reasoning-budget 0 --reasoning-format none \
  -ngl 999 -c 262144 -fa on \
  -ctk f16 -ctv f16 \
  -b 8192 -ub 2048 -np 1 \
  --host 0.0.0.0 --port 8080 2>&1 | tee /root/llama.log
EOF

ssh -p <PORT> root@<HOST> 'command -v tmux >/dev/null || apt-get install -y -qq tmux; tmux has-session -t llama 2>/dev/null || tmux new-session -d -s llama sh /root/run-llama.sh'
```

This is exactly what `claude-qwen-server` does, so once you source the function you can skip this by-hand step.

Confirm it is up (this also happens automatically when you run `claude-qwen`):

```
ssh -p <PORT> root@<HOST> 'curl -sf http://localhost:8080/health'
```

If this is empty or fails, watch the log: `ssh -p <PORT> root@<HOST> 'tail -f /root/llama.log'`.

Notes on the flags:

- `-hf HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive:Q6_K_P` is ungated, so no HuggingFace token.`Q6_K_P` is a "Perfect" quant: a profile tuned for that one model, 7.07 bits per weight against 4.88 for the plain`Q4_K_M` .
- The model downloads on first launch into `/root/models` (`LLAMA_CACHE` ). 31 GB takes about 8 minutes. Wait for`/health` to return.
- `--reasoning-budget 0 --reasoning-format none` disables the model's thinking output (the `thinking` section). Claude Code does not read it.
- `-c 262144` is the context size. Step 5 sets the matching client variable. llama.cpp allocates the KV cache for the whole context at load, so this is a load-time VRAM cost, not something that grows while you chat.
- `-fa on` is flash attention; note it now needs an explicit value,`-fa` alone errors.
- `-ngl 999` puts all layers on the GPU.
- `-ctk f16 -ctv f16` keeps the KV cache at full precision. This is the counter-intuitive one: the 8 bit cache is the SLOWER one. At 104k context f16 gives 106.0 tok/s,`q8_0` gives 74.3,`q4_0` gives 70.1. The attention kernel dequantizes the cache on every token, and that costs more than the bytes it saves. Weights plus f16 KV load at 37.1 of 40.9 GB, so it fits on a 40 GB card.
- `-b 8192 -ub 2048` raise the prompt-processing batch size. Prompt processing is compute bound, so bigger batches help: +11% at 104k context. It does nothing for generation.
- `-np 1` keeps one slot. The default is 4 slots, which splits`-c 262144` into 4 x 65536 while Claude Code is still told 262144, so a long session overflows its slot.

For a dense model, generation is memory-bandwidth bound, not compute bound. Each generated token reads every weight once plus the whole KV cache, so tokens per second is roughly `bandwidth / (weights + KV)`, and a card running at its rated bandwidth is already near its ceiling. Measured on a dense 27B Q6_K (23.46 GB):

| GPU | measured bandwidth | generation | prompt processing | 
|---|---|---|---|
| RTX A6000 48 GB | 572.5 GB/s | 17.3 t/s | 558 t/s | 
| A100 PCIE 40 GB | 1314.5 GB/s | 39.8 t/s | 1297 t/s | 

The 2.30x bandwidth ratio shows up almost exactly as 2.3x on both numbers. Vast's `gpu_mem_bw` on the offer is a benchmark, not the datasheet number, and it is the field worth sorting by.

This MoE model breaks that rule, which is the whole reason to run one. Only about 3B of its 35B parameters are active per token, so the card reads far less and finishes the token while the memory bus is idle: measured 239 GB/s of the A100's 1314, about 18%. Generation here is latency bound, not bandwidth bound.

Two things follow, both measured on the A100 with this model:

- A bigger quant is free, and slightly faster. Q6_K_P at 7.07 bits per weight beats Q4_K_M at 4.88: 140.9 against 134.6 tok/s at short context, and 112.1 against 106.2 at 104k. The extra bytes ride on bandwidth that was not being used anyway. So pick the largest quant that fits in VRAM, not the smallest that fits.
- The KV cache is faster at full precision, for the reason in the flag notes above.

Prompt processing behaves normally: it is compute bound, so a bigger batch helps and the KV type barely matters.

Two caveats. The first request after a boot pays roughly 36 s of CUDA warmup before it streams; every request after that is clean. And the prompt cache does work: re-sending the same 104k prompt drops the prompt token count from 104015 to 4 and prompt processing from 22637 ms to 99.6 ms. In a long Claude Code session only the new tokens get processed, so generation speed is the real limit, not prompt processing.

One flag that does nothing here: `--cache-reuse` is silently disabled because the vision `mmproj` is loaded. The server log says `cache_reuse is not supported by multimodal, it will be disabled`. It is left out of the launch command for that reason.

Put `claude-qwen.zsh` somewhere and source it from `~/.zshrc`:

```
source /path/to/claude-qwen.zsh
```

The file defines one command plus one function per concern:

- `claude-qwen-instance` - Vast.ai: find the instance by label, boot it if stopped, print`host port`
- `claude-qwen-server` - the box: make sure`llama-server` is serving (uploads the template and re-launches tmux if missing or down)
- `claude-qwen-tunnel` - the ssh tunnel`11435 -> 8080` , then wait until`/health` answers
- `claude-qwen` - glue the three together and run Claude Code

`claude-qwen` runs the whole lifecycle, so after the first run you never touch the instance by hand:

1. Find the instance by its `claude-qwen` label.
2. Start it if stopped.
3. Make sure `llama-server` is up (re-launch in tmux if`/health` fails, re-upload the template if missing).
4. Open the tunnel `127.0.0.1:11435 -> 8080` .
5. Run Claude Code against `http://127.0.0.1:11435` .

Two paths are hardcoded and must match, or change them to taste:

- The local template is read from `~/.claude/qwen-chat-template.jinja` . Keep the file there, or edit the`scp` line in`claude-qwen-server` .
- The remote template lives at `/root/qwen.jinja` and the launch script the function writes is`/root/run-llama.sh` .

Ports 11435 (local) and 8080 (remote) are also fixed; change them in both `claude-qwen-tunnel` and the `--settings` URL if they collide.

```
claude-qwen                                   # interactive
claude-qwen -p "reply with exactly: pong"     # one-shot smoke test
```

Pass: the one-shot prints `pong`. Pass: an interactive session makes a tool call (for example ask it to list the current directory) and it runs.

- `CLAUDE_CODE_MAX_CONTEXT_TOKENS=262144` - must match the server's`-c 262144` . Claude Code assumes 200k for unknown models, sends too much, and the server rejects with`Context size has been exceeded` .
- `ANTHROPIC_MODEL=qwen-uncensored` plus`ANTHROPIC_DEFAULT_OPUS_MODEL` ,`ANTHROPIC_DEFAULT_SONNET_MODEL` ,`ANTHROPIC_DEFAULT_HAIKU_MODEL` all set to the same name - without them Claude Code prompts for a real Anthropic model.
- `ANTHROPIC_AUTH_TOKEN=dummy` ,`ANTHROPIC_API_KEY=` (empty) - the local server does no auth; Claude Code still wants a token field present.
- The `--settings` JSON forces`ANTHROPIC_BASE_URL` . Your`~/.claude/settings.json`` env` block wins over shell env, so setting the URL only in the shell does not work.
- `_CLAUDE_CODE_ASSUME_FIRST_PARTY_BASE_URL=0` inside`--settings` - stops Claude Code treating the URL as a first-party endpoint.
- `command claude` - runs the real binary, skipping any shell alias or wrapper named`claude` .

- ~$0.63/hr to run, ~$15/day if left on.
- `vastai stop instance <INSTANCE_ID>` - stops billing, keeps the disk and downloaded model.
- `vastai start instance <INSTANCE_ID>` - resumes; the function re-launches the server for you.
- `vastai destroy instance <INSTANCE_ID>` - wipes it; you then re-rent and re-download.

Every error below was hit while building and running this. Symptom is the exact text; fix is what resolved it.

| Symptom | Cause | Fix | 
|---|---|---|
| `System message must be at the beginning` on a request | The stock Qwen template requires the system message at position 0, but Claude Code sends a `system` -role message mid-array (roles`[user, system, assistant, user]` , usually around tool output) | Use the patched `qwen-chat-template.jinja` from this gist and launch with`--chat-template-file /root/qwen.jinja` . It renders a mid-array system block inline instead of erroring. | 
| Every request 500s with `Jinja Exception: Unexpected item type in content.` | Claude Code's ToolSearch returns a `tool_result` whose`content` is a list of`{"type": "tool_reference", "tool_name": "Bash"}` blocks. The stock`render_content`` else` branch rejects anything that is not text, image or video, and`tool_reference` is none of those. | Use the patched `qwen-chat-template.jinja` from this gist: it renders`tool_reference` as empty and any other unknown block as`[unsupported block: <type>]` instead of raising. To find which block it was, read the request bodies Claude Code saved under`~/.claude/projects/<slug>/*.jsonl` (records of type`api-request-blob` );`llama-server` never logs the payload. | 
| `pkill -f llama-server` kills the shell running it | The pattern also matches the ssh command line, which contains that exact string | Use `pkill -f "[l]lama-server"` . The bracket stops the pattern matching itself. | 
| `cache_reuse is not supported by multimodal, it will be disabled` in the log | The vision `mmproj` is loaded, so`--cache-reuse` does nothing at all | Not a fault. Leave the flag out. The built-in prompt cache still works. | 
| Server responds `Context size has been exceeded` on long requests | Server runs `-c 262144` , but Claude Code assumes 200k for unknown models and sends 200k+ | Set `CLAUDE_CODE_MAX_CONTEXT_TOKENS=262144` on the client so it compacts before overflow. | 
| Claude Code prompts to pick a real Anthropic model | Only `ANTHROPIC_MODEL` was set; the tier defaults were not | Set all three `ANTHROPIC_DEFAULT_*_MODEL` vars to the same model name. | 
| `ANTHROPIC_BASE_URL` set as a shell env var is ignored | `~/.claude/settings.json`` env` block wins over shell env | Pass the base URL through `--settings '{"env":{"ANTHROPIC_BASE_URL":"..."}}'` , as the function does. | 
| Server dies the moment the ssh session closes | A plain `nohup ... &` over ssh gets killed along with the session | Run the server inside tmux (session `llama` ), as in step 3.`setsid nohup ... < /dev/null &` also survives, but tmux lets you attach and read the log. | 
| Instance is up but nothing listens on 8080 | Vast's `--ssh` wrapper ignores the`--args` you passed at create time | Launch `llama-server` yourself (step 3). The function does this automatically on each run. | 
| `pgrep -f llama-server` says it is running when it is not | `pgrep -f` matches your own ssh command string | Never decide "up" from a process list. Check `curl -sf http://localhost:8080/health` . The function checks only`/health` . | 
| Responses contain a  `thinking` block | Reasoning/thinking output is on by default | Add `--reasoning-budget 0 --reasoning-format none` . | 
| `llama-server` errors on`-fa` | Flash attention now requires an explicit value | Use `-fa on` . | 

If a start call in the function fails, read the remote log it points to: `ssh -p <PORT> root@<HOST> 'tail -50 /root/llama.log'`.