# Humble Pi (Qwen 3.8 27B Edition) -- agentic coding on 24 GB of VRAM

> Source: <https://gist.github.com/joematthews/69e60b357470487618a0b14b0aabecb8>
> Published: 2026-08-19 14:55:04+00:00

This guide sets up [Qwen3.8 27B](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) with [pi](https://pi.dev) through [llama.cpp](https://github.com/ggml-org/llama.cpp) on a
32 GB machine: an Apple Silicon Mac or a Linux box with a 24 GB GPU. Thinking is kept in the
conversation on every turn, so the KV cache survives between messages. No API key, no cloud, works offline.

| download | 17.56 GB (`UD-Q4_K_XL` ) |
| resident | about 19.4 GB at 64k context |
| generation | about 8 tokens/second on an M-series with 32 GB, 9 with MTP |
| prompt | about 66 tokens/second |
| licence | Apache 2.0 |

Qwen3.8 needs a recent build. This gives you the `llama`

binary. Confirm with `llama version`

;
you want build 10470 or later.

**macOS** -- [Homebrew](https://formulae.brew.sh/formula/llama.cpp). Already installed? `brew upgrade llama.cpp`

instead.

```
brew install llama.cpp
```

**Linux** -- [installama.sh](https://github.com/angt/installama.sh) detects your CPU and GPU
(CUDA / ROCm / Vulkan) and drops `llama`

into `~/.local/bin`

. Run it again to upgrade.

```
curl -fsSL https://angt.github.io/installama.sh | sh
```

Unsloth's GGUF carries its own chat template and llama.cpp reads it from the model file. No download, no patch.

Put this in `~/.zshrc`

(`~/.bashrc`

on Linux):

```
alias qwen-27b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -c 65536 -fa 1 -ctk q8_0 -ctv q8_0 -ctxcp 8 --prio 2 --no-ui --jinja --parallel 1 --cache-ram 0 --no-mmproj --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --reasoning on --reasoning-preserve 2>&1 | tee ~/.llama-logs/qwen-27b-$(date +%Y%m%d-%H%M%S).log'
source ~/.zshrc   # source ~/.bashrc on Linux
qwen-27b
```

The first run downloads 17.56 GB to `~/.cache/huggingface/hub`

. The server is up when the log
prints `listening on http://127.0.0.1:8080`

. A `failed to set process priority`

warning is the OS
refusing `--prio 2`

without elevated rights and can be ignored.

| flag | what it does |
|---|---|
`-c 65536` |
Context. 64k fits 32 GB with room to spare; the model trains to 262144. |
`-fa 1` |
Flash attention. Required for a quantized KV cache. |
`-ctk q8_0 -ctv q8_0` |
8-bit KV cache, roughly half the size of f16. |
`-ctxcp 8` |
Cap context checkpoints. Each one snapshots the recurrent state (about 150 MiB). Inert at 64k, where only 8 can exist; keeps memory flat if you raise `-c` . |
`--prio 2` |
High thread priority where the OS allows it, so inference is interrupted less. |
`--parallel 1` |
One slot, so the whole context serves one conversation. |
`--cache-ram 0` |
No cross-chat KV cache in RAM. One chat at a time does not need it. |
`--no-mmproj` |
Skip the 930 MB vision projector. Text only. |
`--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0` |
Qwen's recommended sampling for thinking mode. |
`--reasoning on` |
Parse the model's `<think>` blocks into `reasoning_content` . |
`--reasoning-preserve` |
Keep every prior turn's thinking in the prompt, not just the last. The template already defaults to this; the flag makes it explicit. |

Only 16 of Qwen3.8's 64 layers use full attention; the rest are Gated DeltaNet. That is why 64k of context costs about 2.3 GiB rather than the ~9 GiB a conventional 27B would need.

```
curl -fsSL https://pi.dev/install.sh | sh
pi install npm:pi-llama-cpp
pi install npm:pi-smart-fetch
pi install npm:pi-smart-web-search
pi install npm:pi-plate
```

finds the local server and maps pi's thinking levels to a token budget.`pi-llama-cpp`

reads web pages.`pi-smart-fetch`

searches the web, no API key. Requires pi-smart-fetch.`pi-smart-web-search`

supplies the date, the machine and your git state, appended at the tail of the turn so the cached prefix stays intact.`pi-plate`

pi's thinking level sets two things on every request: the `reasoning_effort`

the template tells
the model, and the `thinking_budget_tokens`

that `pi-llama-cpp`

passes to the server. The default,
medium, means medium effort and an 8,192-token cap a turn.

| pi level | template effort | budget |
|---|---|---|
| off | thinking disabled | 0 |
| minimal | rejected | do not use |
| low | low | 2,048 |
| medium | medium | 8,192 |
| high | xhigh | 16,384 |
| xhigh | xhigh | 32,768 |
| max | rejected | do not use |

Qwen3.8's template accepts only `low`

, `medium`

and `xhigh`

(`high`

maps to `xhigh`

). Pick
`minimal`

or `max`

in pi and every request fails with `Unexpected reasoning effort`

; switch levels
and carry on.

```
# terminal 1
source ~/.zshrc && qwen-27b   # source ~/.bashrc on Linux
# terminal 2
mkdir -p ~/Code/demo && cd ~/Code/demo && pi
Build a todo app in ./todo with separate index.html, style.css and app.js. No dependencies, no build step.

Features: add, toggle complete, delete, filter all/active/done, clear completed, item count, persist to localStorage.

Use one delegated click handler. Keep ids as strings everywhere. When the files are written, re-read app.js and trace each feature once to confirm the logic holds.

Dark theme, readable at half-screen width. Then run npx serve.
```

KV cache costs 34 KiB per token with `-ctk q8_0 -ctv q8_0`

. Weights are 16.3 GiB. Both live in
GPU memory. Checkpoints are spaced 8192 tokens apart and live in ordinary process memory, so raise
`-ctxcp`

with `-c`

or old turns lose their rewind points and an edit or compaction re-reads from
token zero. Budget for one slot:

`-c` |
`-ctxcp` |
GPU (weights + KV) | checkpoints | total |
|---|---|---|---|---|
| 65536 | 8 | about 19.3 GiB | 1.2 GiB | about 20.5 GiB |
| 131072 | 16 | about 21.5 GiB | 2.4 GiB | about 24 GiB |
| 262144 | 32 | about 25.8 GiB | 4.7 GiB | about 30.5 GiB |

A 32 GB Mac lets Metal use about 25 GB, so 64k and 128k both fit:

```
-c 131072 -ctxcp 16
```

256k does not fit on 32 GB. A 24 GB Linux card holds 64k; 128k needs the 2.4 GiB of checkpoints in system RAM and about 21.5 GiB on the card, so it fits with nothing else running.

The Unsloth GGUF carries Qwen3.8's MTP head. llama.cpp can use it as a speculative draft with no second model file. Add to the alias:

```
--spec-type draft-mtp
```

Measured on an M-series with 32 GB: about 13% faster generation at short context (9.1 against 8.1 tokens/second), 75% draft acceptance over a 12-turn coding session, under 1 GiB of extra memory. Output is unchanged; speculative decoding verifies every token against the model's own distribution. Untested at long context and on Linux.

`--no-mmproj`

skips the 930 MB projector. Remove the flag and llama.cpp downloads it and lets pi
send images. Costs about 1 GiB of memory and a slower load. Text-only agentic coding does not
need it.