# POCKET-27B-Pi: Running a 27B Coding Agent on a Single 24GB GPU with a 16.5GB GGUF File

> Source: <https://dev.to/ai_openfree_b23025ef075cf/pocket-27b-pi-running-a-27b-coding-agent-on-a-single-24gb-gpu-with-a-165gb-gguf-file-2ce7>
> Published: 2026-11-22 23:01:40+00:00

**Key takeaway:** Merging the pi agent adapter into Qwen3.8-27B weights and shipping a 16.5GB Q4_K_M GGUF lets the model run on a single 24GB GPU at 64K context, completing 28 of 48 Terminal-Bench 2.1 tasks versus 19 for the base model.

**TL;DR:** VIDRAFT's POCKET-27B-Pi merges a Qwen3-based 27B coding agent adapter directly into model weights and ships it as a single Q4_K_M GGUF file weighing just 16.5GB — small enough to fit on a single 24GB GPU with a full 64K context window. On Terminal-Bench 2.1's 48 shared tasks, it completes 28 (58.3%), versus 19 (39.6%) for the base Qwen3.8-27B — a 47% relative improvement — at 3.3× smaller file size than the full-precision pi build.

POCKET-27B-Pi is a local-deployment coding agent released by VIDRAFT. It is built on a Qwen3.8-27B backbone and ships with the pi coding agent adapter already **merged into the model weights** — no separate adapter loading required at inference time.

Two quantization variants are published to the Hugging Face repository `FINAL-Bench/POCKET-27B-Pi-GGUF` under an **Apache-2.0 license**:

The key engineering proposition here isn't just the parameter count. It's a **verifiable memory budget**: this model, with a 64K context window open, fits inside a single 24GB GPU. That targets workstations, small-form-factor desktops, and on-premise developer machines — deployments that cannot or should not depend on a cloud API.

The memory efficiency comes from the base model's **hybrid attention architecture**. According to the model's public `config.json`:

This distinction matters enormously for KV cache sizing. Only the 16 full-attention layers accumulate per-token K and V tensors. With `num_key_value_heads = 4` and `head_dim = 256`, the per-token KV cost in f16 works out to a fixed, calculable figure. At 64K context, the f16 KV cache is approximately **4.0 GiB**. llama.cpp's quantized cache options bring this down further:

`q8_0` cache: ~2.13 GiB`q4_0` cache: ~1.13 GiB
Combining the Q4_K_M weights (~15.4 GiB) with a 64K f16 KV cache lands at roughly **19.55 GiB** before compute buffers — leaving headroom inside a 24GB card. Switching to `q8_0` cache reclaims about 1.9 GiB, which matters if you're running concurrent agent sessions.

There's also a decode-speed angle. On consumer hardware, tokens-per-second is largely a **memory bandwidth problem**: throughput ≈ bandwidth ÷ bytes-read-per-token. A file that's ~3.3× smaller than the full-precision build has a proportionally higher theoretical token throughput ceiling on the same GPU. For a time-budgeted coding agent (each task gets one hour), faster decoding directly translates to more reasoning turns completed before the clock runs out.

All numbers below are from VIDRAFT's own Terminal-Bench 2.1 evaluation. Thinking was disabled; each task received one attempt and a one-hour wall-clock budget; eight agent sessions ran in parallel on a single GPU.

**On the 48 tasks completed by all evaluated builds:**

| Model | Tasks Completed | Completion Rate | 
|---|---|---|
| POCKET-27B-Pi Q4_K_M | 28 | 58.3% | 
| Base Qwen3.8-27B | 19 | 39.6% | 

**On 50 shared tasks (direct comparison):**

**Q4_K_M vs. Q6_K on 62 overlapping tasks:** Both completed 39 tasks — the smaller file matched the larger one exactly, supporting Q4_K_M as the recommended delivery format.

**Agent efficiency (50-task set):**

The turn-count delta is significant: the pi agent reaches solutions in roughly half the turns, freeing up time budget for more tasks within the hour window.

The model files are publicly available on Hugging Face. You can download and serve locally using llama.cpp's `llama-server`:

```
# Download the Q4_K_M variant
huggingface-cli download FINAL-Bench/POCKET-27B-Pi-GGUF \
  --include "*.Q4_K_M.gguf" --local-dir ./pocket-27b-pi
# Serve with 64K context, thinking disabled, quantized KV cache for headroom
llama-server \
  -m ./pocket-27b-pi/<filename>.Q4_K_M.gguf \
  -c 65536 \
  --chat-template-kwargs '{"enable_thinking":false}' \
  --cache-type-k q8_0 \
  --cache-type-v q8_0
```

The server exposes an **OpenAI-compatible API** at `http://localhost:8080/v1`. The pi agent can be pointed at this endpoint via its `~/.pi/agent/models.json` config by setting `baseUrl` accordingly. Once the GGUF is downloaded, the full inference loop runs **entirely offline** — source code stays on your machine.

**Before switching team workflows:** Re-run your own task set to validate, and record both the llama.cpp version and GGUF file hash. That gives you a reproducible baseline for future model swaps.

**Q: Does the adapter need to be loaded separately at inference time?**

A: No. The pi coding agent adapter is already merged into the POCKET-27B-Pi weights. You load a single GGUF file — no extra adapter mounting step needed on the server.

**Q: Why does Q4_K_M match Q6_K on benchmark tasks if it's a heavier quantization?**

A: On the 62 directly comparable tasks, both variants completed 39 tasks. This suggests that for coding agent tasks under a time budget, the decode-speed advantage of the smaller Q4_K_M file may compensate for any precision loss from heavier quantization — though developers should run their own task sets to validate this for their specific workloads.

**Q: What is the license?**

A: The Hugging Face repository is marked Apache-2.0, which permits commercial use. Always verify the repository's license file directly before production deployment.

**Q: What evaluation harness were the benchmarks run against?**

A: VIDRAFT's own Terminal-Bench 2.1. These are self-reported numbers. VIDRAFT recommends re-running against your own task sets before drawing production conclusions.

*Originally reported by 点点资讯 디안디안 (중국) (2026-11-22) — [source article](https://www.dian8dian.com/27b-bian-ma-zhi-neng-ti-ben-di-hua-16-5gb-wen-jian-ru-he).*
