{"slug": "humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram", "title": "Humble Pi (Qwen 3.8 27B Edition) -- agentic coding on 24 GB of VRAM", "summary": "A developer detailed a setup for running Qwen3.8 27B, an agentic coding model, locally on a 24 GB VRAM machine using llama.cpp and the pi coding agent. The configuration achieves about 8 tokens per second generation on Apple Silicon and works offline without an API key. The guide covers installation, server flags, and pi integration for a fully local agentic coding experience.", "body_md": "This guide sets up [Qwen3.8 27B](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) with [pi](https://pi.dev) through [llama.cpp](https://github.com/ggml-org/llama.cpp) on a\n32 GB machine: an Apple Silicon Mac or a Linux box with a 24 GB GPU. Thinking is kept in the\nconversation on every turn, so the KV cache survives between messages. No API key, no cloud, works offline.\n\n| download | 17.56 GB (`UD-Q4_K_XL` ) |\n| resident | about 19.4 GB at 64k context |\n| generation | about 8 tokens/second on an M-series with 32 GB, 9 with MTP |\n| prompt | about 66 tokens/second |\n| licence | Apache 2.0 |\n\nQwen3.8 needs a recent build. This gives you the `llama`\n\nbinary. Confirm with `llama version`\n\n;\nyou want build 10470 or later.\n\n**macOS** -- [Homebrew](https://formulae.brew.sh/formula/llama.cpp). Already installed? `brew upgrade llama.cpp`\n\ninstead.\n\n```\nbrew install llama.cpp\n```\n\n**Linux** -- [installama.sh](https://github.com/angt/installama.sh) detects your CPU and GPU\n(CUDA / ROCm / Vulkan) and drops `llama`\n\ninto `~/.local/bin`\n\n. Run it again to upgrade.\n\n```\ncurl -fsSL https://angt.github.io/installama.sh | sh\n```\n\nUnsloth's GGUF carries its own chat template and llama.cpp reads it from the model file. No download, no patch.\n\nPut this in `~/.zshrc`\n\n(`~/.bashrc`\n\non Linux):\n\n```\nalias qwen-27b='mkdir -p ~/.llama-logs && llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL -c 65536 -fa 1 -ctk q8_0 -ctv q8_0 -ctxcp 8 --prio 2 --no-ui --jinja --parallel 1 --cache-ram 0 --no-mmproj --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0 --reasoning on --reasoning-preserve 2>&1 | tee ~/.llama-logs/qwen-27b-$(date +%Y%m%d-%H%M%S).log'\nsource ~/.zshrc   # source ~/.bashrc on Linux\nqwen-27b\n```\n\nThe first run downloads 17.56 GB to `~/.cache/huggingface/hub`\n\n. The server is up when the log\nprints `listening on http://127.0.0.1:8080`\n\n. A `failed to set process priority`\n\nwarning is the OS\nrefusing `--prio 2`\n\nwithout elevated rights and can be ignored.\n\n| flag | what it does |\n|---|---|\n`-c 65536` |\nContext. 64k fits 32 GB with room to spare; the model trains to 262144. |\n`-fa 1` |\nFlash attention. Required for a quantized KV cache. |\n`-ctk q8_0 -ctv q8_0` |\n8-bit KV cache, roughly half the size of f16. |\n`-ctxcp 8` |\nCap context checkpoints. Each one snapshots the recurrent state (about 150 MiB). Inert at 64k, where only 8 can exist; keeps memory flat if you raise `-c` . |\n`--prio 2` |\nHigh thread priority where the OS allows it, so inference is interrupted less. |\n`--parallel 1` |\nOne slot, so the whole context serves one conversation. |\n`--cache-ram 0` |\nNo cross-chat KV cache in RAM. One chat at a time does not need it. |\n`--no-mmproj` |\nSkip the 930 MB vision projector. Text only. |\n`--temp 1.0 --top-p 0.95 --top-k 20 --min-p 0` |\nQwen's recommended sampling for thinking mode. |\n`--reasoning on` |\nParse the model's `<think>` blocks into `reasoning_content` . |\n`--reasoning-preserve` |\nKeep every prior turn's thinking in the prompt, not just the last. The template already defaults to this; the flag makes it explicit. |\n\nOnly 16 of Qwen3.8's 64 layers use full attention; the rest are Gated DeltaNet. That is why 64k of context costs about 2.3 GiB rather than the ~9 GiB a conventional 27B would need.\n\n```\ncurl -fsSL https://pi.dev/install.sh | sh\npi install npm:pi-llama-cpp\npi install npm:pi-smart-fetch\npi install npm:pi-smart-web-search\npi install npm:pi-plate\n```\n\nfinds the local server and maps pi's thinking levels to a token budget.`pi-llama-cpp`\n\nreads web pages.`pi-smart-fetch`\n\nsearches the web, no API key. Requires pi-smart-fetch.`pi-smart-web-search`\n\nsupplies the date, the machine and your git state, appended at the tail of the turn so the cached prefix stays intact.`pi-plate`\n\npi's thinking level sets two things on every request: the `reasoning_effort`\n\nthe template tells\nthe model, and the `thinking_budget_tokens`\n\nthat `pi-llama-cpp`\n\npasses to the server. The default,\nmedium, means medium effort and an 8,192-token cap a turn.\n\n| pi level | template effort | budget |\n|---|---|---|\n| off | thinking disabled | 0 |\n| minimal | rejected | do not use |\n| low | low | 2,048 |\n| medium | medium | 8,192 |\n| high | xhigh | 16,384 |\n| xhigh | xhigh | 32,768 |\n| max | rejected | do not use |\n\nQwen3.8's template accepts only `low`\n\n, `medium`\n\nand `xhigh`\n\n(`high`\n\nmaps to `xhigh`\n\n). Pick\n`minimal`\n\nor `max`\n\nin pi and every request fails with `Unexpected reasoning effort`\n\n; switch levels\nand carry on.\n\n```\n# terminal 1\nsource ~/.zshrc && qwen-27b   # source ~/.bashrc on Linux\n# terminal 2\nmkdir -p ~/Code/demo && cd ~/Code/demo && pi\nBuild a todo app in ./todo with separate index.html, style.css and app.js. No dependencies, no build step.\n\nFeatures: add, toggle complete, delete, filter all/active/done, clear completed, item count, persist to localStorage.\n\nUse one delegated click handler. Keep ids as strings everywhere. When the files are written, re-read app.js and trace each feature once to confirm the logic holds.\n\nDark theme, readable at half-screen width. Then run npx serve.\n```\n\nKV cache costs 34 KiB per token with `-ctk q8_0 -ctv q8_0`\n\n. Weights are 16.3 GiB. Both live in\nGPU memory. Checkpoints are spaced 8192 tokens apart and live in ordinary process memory, so raise\n`-ctxcp`\n\nwith `-c`\n\nor old turns lose their rewind points and an edit or compaction re-reads from\ntoken zero. Budget for one slot:\n\n`-c` |\n`-ctxcp` |\nGPU (weights + KV) | checkpoints | total |\n|---|---|---|---|---|\n| 65536 | 8 | about 19.3 GiB | 1.2 GiB | about 20.5 GiB |\n| 131072 | 16 | about 21.5 GiB | 2.4 GiB | about 24 GiB |\n| 262144 | 32 | about 25.8 GiB | 4.7 GiB | about 30.5 GiB |\n\nA 32 GB Mac lets Metal use about 25 GB, so 64k and 128k both fit:\n\n```\n-c 131072 -ctxcp 16\n```\n\n256k does not fit on 32 GB. A 24 GB Linux card holds 64k; 128k needs the 2.4 GiB of checkpoints in system RAM and about 21.5 GiB on the card, so it fits with nothing else running.\n\nThe Unsloth GGUF carries Qwen3.8's MTP head. llama.cpp can use it as a speculative draft with no second model file. Add to the alias:\n\n```\n--spec-type draft-mtp\n```\n\nMeasured on an M-series with 32 GB: about 13% faster generation at short context (9.1 against 8.1 tokens/second), 75% draft acceptance over a 12-turn coding session, under 1 GiB of extra memory. Output is unchanged; speculative decoding verifies every token against the model's own distribution. Untested at long context and on Linux.\n\n`--no-mmproj`\n\nskips the 930 MB projector. Remove the flag and llama.cpp downloads it and lets pi\nsend images. Costs about 1 GiB of memory and a slower load. Text-only agentic coding does not\nneed it.", "url": "https://wpnews.pro/news/humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram", "canonical_source": "https://gist.github.com/joematthews/69e60b357470487618a0b14b0aabecb8", "published_at": "2026-08-19 14:55:04+00:00", "updated_at": "2026-08-20 11:14:17.465432+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Qwen", "llama.cpp", "pi", "Unsloth", "Hugging Face", "Homebrew", "Apple Silicon"], "alternates": {"html": "https://wpnews.pro/news/humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram", "markdown": "https://wpnews.pro/news/humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram.md", "text": "https://wpnews.pro/news/humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram.txt", "jsonld": "https://wpnews.pro/news/humble-pi-qwen-3-8-27b-edition-agentic-coding-on-24-gb-of-vram.jsonld"}}