This guide sets up Qwen3.8 27B with pi through llama.cpp on a 32 GB machine: an Apple Silicon Mac or a Linux box with a 24 GB GPU. Thinking is kept in the conversation on every turn, so the KV cache survives between messages. No API key, no cloud, works offline.
| download | 17.56 GB (UD-Q4_K_XL ) |
| resident | about 19.4 GB at 64k context |
| generation | about 8 tokens/second on an M-series with 32 GB, 9 with MTP |
| prompt | about 66 tokens/second |
| licence | Apache 2.0 |
Qwen3.8 needs a recent build. This gives you the llama
binary. Confirm with llama version
; you want build 10470 or later.
macOS -- Homebrew. Already installed? brew upgrade llama.cpp
instead.
brew install llama.cpp
Linux -- installama.sh detects your CPU and GPU
(CUDA / ROCm / Vulkan) and drops llama
into ~/.local/bin
. Run it again to upgrade.
curl -fsSL https://angt.github.io/installama.sh | sh
Unsloth's GGUF carries its own chat template and llama.cpp reads it from the model file. No download, no patch.
Put this in ~/.zshrc
(~/.bashrc
on Linux):
alias qwen-27b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -c 65536 -fa 1 -ctk q8_0 -ctv q8_0 -ctxcp 8 --prio 2 --no-ui --jinja --parallel 1 --cache-ram 0 --no-mmproj --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --reasoning on --reasoning-preserve 2>&1 | tee ~/.llama-logs/qwen-27b-$(date +%Y%m%d-%H%M%S).log'
source ~/.zshrc # source ~/.bashrc on Linux
qwen-27b
The first run downloads 17.56 GB to ~/.cache/huggingface/hub
. The server is up when the log
prints listening on http://127.0.0.1:8080
. A failed to set process priority
warning is the OS
refusing --prio 2
without elevated rights and can be ignored.
| flag | what it does |
|---|---|
-c 65536 |
|
| Context. 64k fits 32 GB with room to spare; the model trains to 262144. | |
-fa 1 |
|
| Flash attention. Required for a quantized KV cache. | |
-ctk q8_0 -ctv q8_0 |
|
| 8-bit KV cache, roughly half the size of f16. | |
-ctxcp 8 |
|
Cap context checkpoints. Each one snapshots the recurrent state (about 150 MiB). Inert at 64k, where only 8 can exist; keeps memory flat if you raise -c . |
|
--prio 2 |
|
| High thread priority where the OS allows it, so inference is interrupted less. | |
--parallel 1 |
|
| One slot, so the whole context serves one conversation. | |
--cache-ram 0 |
|
| No cross-chat KV cache in RAM. One chat at a time does not need it. | |
--no-mmproj |
|
| Skip the 930 MB vision projector. Text only. | |
--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 |
|
| Qwen's recommended sampling for thinking mode. | |
--reasoning on |
|
Parse the model's <think> blocks into reasoning_content . |
|
--reasoning-preserve |
|
| Keep every prior turn's thinking in the prompt, not just the last. The template already defaults to this; the flag makes it explicit. |
Only 16 of Qwen3.8's 64 layers use full attention; the rest are Gated DeltaNet. That is why 64k of context costs about 2.3 GiB rather than the ~9 GiB a conventional 27B would need.
curl -fsSL https://pi.dev/install.sh | sh
pi install npm:pi-llama-cpp
pi install npm:pi-smart-fetch
pi install npm:pi-smart-web-search
pi install npm:pi-plate
finds the local server and maps pi's thinking levels to a token budget.pi-llama-cpp
reads web pages.pi-smart-fetch
searches the web, no API key. Requires pi-smart-fetch.pi-smart-web-search
supplies the date, the machine and your git state, appended at the tail of the turn so the cached prefix stays intact.pi-plate
pi's thinking level sets two things on every request: the reasoning_effort
the template tells
the model, and the thinking_budget_tokens
that pi-llama-cpp
passes to the server. The default, medium, means medium effort and an 8,192-token cap a turn.
| pi level | template effort | budget |
|---|---|---|
| off | thinking disabled | 0 |
| minimal | rejected | do not use |
| low | low | 2,048 |
| medium | medium | 8,192 |
| high | xhigh | 16,384 |
| xhigh | xhigh | 32,768 |
| max | rejected | do not use |
Qwen3.8's template accepts only low
, medium
and xhigh
(high
maps to xhigh
). Pick
minimal
or max
in pi and every request fails with Unexpected reasoning effort
; switch levels and carry on.
source ~/.zshrc && qwen-27b # source ~/.bashrc on Linux
mkdir -p ~/Code/demo && cd ~/Code/demo && pi
Build a todo app in ./todo with separate index.html, style.css and app.js. No dependencies, no build step.
Features: add, toggle complete, delete, filter all/active/done, clear completed, item count, persist to localStorage.
Use one delegated click handler. Keep ids as strings everywhere. When the files are written, re-read app.js and trace each feature once to confirm the logic holds.
Dark theme, readable at half-screen width. Then run npx serve.
KV cache costs 34 KiB per token with -ctk q8_0 -ctv q8_0
. Weights are 16.3 GiB. Both live in
GPU memory. Checkpoints are spaced 8192 tokens apart and live in ordinary process memory, so raise
-ctxcp
with -c
or old turns lose their rewind points and an edit or compaction re-reads from token zero. Budget for one slot:
-c |
-ctxcp |
GPU (weights + KV) | checkpoints | total |
|---|---|---|---|---|
| 65536 | 8 | about 19.3 GiB | 1.2 GiB | about 20.5 GiB |
| 131072 | 16 | about 21.5 GiB | 2.4 GiB | about 24 GiB |
| 262144 | 32 | about 25.8 GiB | 4.7 GiB | about 30.5 GiB |
A 32 GB Mac lets Metal use about 25 GB, so 64k and 128k both fit:
-c 131072 -ctxcp 16
256k does not fit on 32 GB. A 24 GB Linux card holds 64k; 128k needs the 2.4 GiB of checkpoints in system RAM and about 21.5 GiB on the card, so it fits with nothing else running.
The Unsloth GGUF carries Qwen3.8's MTP head. llama.cpp can use it as a speculative draft with no second model file. Add to the alias:
--spec-type draft-mtp
Measured on an M-series with 32 GB: about 13% faster generation at short context (9.1 against 8.1 tokens/second), 75% draft acceptance over a 12-turn coding session, under 1 GiB of extra memory. Output is unchanged; speculative decoding verifies every token against the model's own distribution. Untested at long context and on Linux.
--no-mmproj
skips the 930 MB projector. Remove the flag and llama.cpp downloads it and lets pi send images. Costs about 1 GiB of memory and a slower load. Text-only agentic coding does not need it.