Humble Pi (Qwen 3.8 27B Edition) -- agentic coding on 24 GB of VRAM

A developer detailed a setup for running Qwen3.8 27B, an agentic coding model, locally on a 24 GB VRAM machine using llama.cpp and the pi coding agent. The configuration achieves about 8 tokens per second generation on Apple Silicon and works offline without an API key. The guide covers installation, server flags, and pi integration for a fully local agentic coding experience.

This guide sets up Qwen3.8 27B https://huggingface.co/unsloth/Qwen3.8-27B-GGUF with pi https://pi.dev through llama.cpp https://github.com/ggml-org/llama.cpp on a 32 GB machine: an Apple Silicon Mac or a Linux box with a 24 GB GPU. Thinking is kept in the conversation on every turn, so the KV cache survives between messages. No API key, no cloud, works offline. | download | 17.56 GB UD-Q4 K XL | | resident | about 19.4 GB at 64k context | | generation | about 8 tokens/second on an M-series with 32 GB, 9 with MTP | | prompt | about 66 tokens/second | | licence | Apache 2.0 | Qwen3.8 needs a recent build. This gives you the llama binary. Confirm with llama version ; you want build 10470 or later. macOS -- Homebrew https://formulae.brew.sh/formula/llama.cpp . Already installed? brew upgrade llama.cpp instead. brew install llama.cpp Linux -- installama.sh https://github.com/angt/installama.sh detects your CPU and GPU CUDA / ROCm / Vulkan and drops llama into ~/.local/bin . Run it again to upgrade. curl -fsSL https://angt.github.io/installama.sh | sh Unsloth's GGUF carries its own chat template and llama.cpp reads it from the model file. No download, no patch. Put this in ~/.zshrc ~/.bashrc on Linux : alias qwen-27b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4 K XL -c 65536 -fa 1 -ctk q8 0 -ctv q8 0 -ctxcp 8 --prio 2 --no-ui --jinja --parallel 1 --cache-ram 0 --no-mmproj --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --reasoning on --reasoning-preserve 2 &1 | tee ~/.llama-logs/qwen-27b-$ date +%Y%m%d-%H%M%S .log' source ~/.zshrc source ~/.bashrc on Linux qwen-27b The first run downloads 17.56 GB to ~/.cache/huggingface/hub . The server is up when the log prints listening on http://127.0.0.1:8080 . A failed to set process priority warning is the OS refusing --prio 2 without elevated rights and can be ignored. | flag | what it does | |---|---| -c 65536 | Context. 64k fits 32 GB with room to spare; the model trains to 262144. | -fa 1 | Flash attention. Required for a quantized KV cache. | -ctk q8 0 -ctv q8 0 | 8-bit KV cache, roughly half the size of f16. | -ctxcp 8 | Cap context checkpoints. Each one snapshots the recurrent state about 150 MiB . Inert at 64k, where only 8 can exist; keeps memory flat if you raise -c . | --prio 2 | High thread priority where the OS allows it, so inference is interrupted less. | --parallel 1 | One slot, so the whole context serves one conversation. | --cache-ram 0 | No cross-chat KV cache in RAM. One chat at a time does not need it. | --no-mmproj | Skip the 930 MB vision projector. Text only. | --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 | Qwen's recommended sampling for thinking mode. | --reasoning on | Parse the model's <think blocks into reasoning content . | --reasoning-preserve | Keep every prior turn's thinking in the prompt, not just the last. The template already defaults to this; the flag makes it explicit. | Only 16 of Qwen3.8's 64 layers use full attention; the rest are Gated DeltaNet. That is why 64k of context costs about 2.3 GiB rather than the ~9 GiB a conventional 27B would need. curl -fsSL https://pi.dev/install.sh | sh pi install npm:pi-llama-cpp pi install npm:pi-smart-fetch pi install npm:pi-smart-web-search pi install npm:pi-plate finds the local server and maps pi's thinking levels to a token budget. pi-llama-cpp reads web pages. pi-smart-fetch searches the web, no API key. Requires pi-smart-fetch. pi-smart-web-search supplies the date, the machine and your git state, appended at the tail of the turn so the cached prefix stays intact. pi-plate pi's thinking level sets two things on every request: the reasoning effort the template tells the model, and the thinking budget tokens that pi-llama-cpp passes to the server. The default, medium, means medium effort and an 8,192-token cap a turn. | pi level | template effort | budget | |---|---|---| | off | thinking disabled | 0 | | minimal | rejected | do not use | | low | low | 2,048 | | medium | medium | 8,192 | | high | xhigh | 16,384 | | xhigh | xhigh | 32,768 | | max | rejected | do not use | Qwen3.8's template accepts only low , medium and xhigh high maps to xhigh . Pick minimal or max in pi and every request fails with Unexpected reasoning effort ; switch levels and carry on. terminal 1 source ~/.zshrc && qwen-27b source ~/.bashrc on Linux terminal 2 mkdir -p ~/Code/demo && cd ~/Code/demo && pi Build a todo app in ./todo with separate index.html, style.css and app.js. No dependencies, no build step. Features: add, toggle complete, delete, filter all/active/done, clear completed, item count, persist to localStorage. Use one delegated click handler. Keep ids as strings everywhere. When the files are written, re-read app.js and trace each feature once to confirm the logic holds. Dark theme, readable at half-screen width. Then run npx serve. KV cache costs 34 KiB per token with -ctk q8 0 -ctv q8 0 . Weights are 16.3 GiB. Both live in GPU memory. Checkpoints are spaced 8192 tokens apart and live in ordinary process memory, so raise -ctxcp with -c or old turns lose their rewind points and an edit or compaction re-reads from token zero. Budget for one slot: -c | -ctxcp | GPU weights + KV | checkpoints | total | |---|---|---|---|---| | 65536 | 8 | about 19.3 GiB | 1.2 GiB | about 20.5 GiB | | 131072 | 16 | about 21.5 GiB | 2.4 GiB | about 24 GiB | | 262144 | 32 | about 25.8 GiB | 4.7 GiB | about 30.5 GiB | A 32 GB Mac lets Metal use about 25 GB, so 64k and 128k both fit: -c 131072 -ctxcp 16 256k does not fit on 32 GB. A 24 GB Linux card holds 64k; 128k needs the 2.4 GiB of checkpoints in system RAM and about 21.5 GiB on the card, so it fits with nothing else running. The Unsloth GGUF carries Qwen3.8's MTP head. llama.cpp can use it as a speculative draft with no second model file. Add to the alias: --spec-type draft-mtp Measured on an M-series with 32 GB: about 13% faster generation at short context 9.1 against 8.1 tokens/second , 75% draft acceptance over a 12-turn coding session, under 1 GiB of extra memory. Output is unchanged; speculative decoding verifies every token against the model's own distribution. Untested at long context and on Linux. --no-mmproj skips the 930 MB projector. Remove the flag and llama.cpp downloads it and lets pi send images. Costs about 1 GiB of memory and a slower load. Text-only agentic coding does not need it.