This article provides a step by step guide to serving Gemma 4 12B, Google's quantization-aware trained (QAT) build, on an 8 GB NVIDIA RTX 3050 with llama.cpp, with every layer on the GPU and Google's multi-token prediction (MTP) drafter for speculative decoding. A suite of Python MCP tools is built to simplify management of the llama.cpp hosted deployment.
This is the third part of a short series. Part 1, Gemma 4's MTP Drafter, Explained, covers how the drafter works and how the exact Q4_0 repack is built. Part 2, Do Drafters Work on Small Models?, measures E2B and E4B on this card and a GTX 1650 Ti. This part moves up to the 12B.
https://github.com/xbill9/gemma4-dev
Can a QAT 12B fit an 8 GB GPU? Yes, with every layer on the card and room for a working context:
| RTX 3050 OEM, 8 GB | Google's 12B Q4_0 GGUF | exact Q4_0 rebuild |
|---|---|---|
| File size | 6.98 GB | 6.72 GB |
| Most context that loads | 28,672 tokens | 40,960 tokens |
| Generation | 27.7 tok/s | 28.6 tok/s |
| With the 12B drafter | — | 56.4 tok/s |
| Most context with the drafter, 4-bit cache | — | 81,920 tokens |
The rebuild stores every weight on the 4-bit grid QAT trained, including the embedding table that Google's file keeps at 6 bits. That one table frees 247.5 MiB of the card, which buys 12,288 more tokens of context. Google's 12B drafter, rebuilt the same way, doubles generation, and a 4-bit context cache gives it 81,920 tokens of context with no change in answers.
Gemma 4 12B is a dense model with 48 layers and 11.9 billion parameters. Google publishes it already trained for 4-bit weights, as safetensors (google/gemma-4-12B-it-qat-q4_0-unquantized) and as a llama.cpp GGUF ( google/gemma-4-12B-it-qat-q4_0-gguf). At 4 bits its weights take about 6.7 GB, which leaves an 8 GB card very little for anything else.
The questions are whether the whole model fits on the card, how much conversation fits beside it, how fast it generates, and what Google's 12B drafter adds. The card is the same RTX 3050 OEM that Part 2 measured with E2B and E4B.
cmake
--spec-type draft-mtp — every number here is from 7182ef71e
python3, no virtualenv, with numpy
local-llamacpp-3050oem-12b-q4_0/
| E4B | 12B | |
|---|---|---|
| Layers | 42 | 48 |
| Hidden size | 2,560 | 3,840 |
| Per-layer embedding table | yes, read from disk | none |
| Layers that own a KV cache | 24 of 42 | all 48 |
| Sliding window | 512 | 1,024 |
| Generation on the 3050, no drafter | 61.2 tok/s | 28.6 tok/s |
E2B and E4B carry a large per-layer embedding table that llama.cpp reads a few rows at a time from the memory-mapped file, so much of their file never reaches the card. The 12B has no such table: almost the whole file goes to the GPU. Its embedding table doubles as the output layer, so it sits on the card as well.
The 12B generates at 0.47x E4B's speed on the same card and llama.cpp build.
The context cache (KV cache) decides how much conversation fits. Gemma 4 12B mixes two kinds of attention layers:
So 40,960 tokens of context cost 640 + 480 MiB. Those are the figures llama.cpp's allocation log prints:
I load_tensors: CUDA0 model buffer size = 6390.19 MiB
I llama_kv_cache: CUDA0 KV buffer size = 640.00 MiB
I llama_kv_cache: CUDA0 KV buffer size = 480.00 MiB
I sched_reserve: CUDA0 compute buffer size = 161.52 MiB
With 6.4 GB of weights on an 8 GB card, every 247 MiB saved on the weights is about 15,800 tokens of context (arithmetic, at 16 KiB per token).
Google's 12B GGUF is 6.131 GB of Q4_0 and 0.826 GB of Q6_K. The Q6_K is a single tensor, token_embd, the 262,144 × 3,840 embedding table, at 787.5 MiB.
QAT trained that table on a 4-bit grid like everything else: in each block of 32 values, every value is a block step times an integer from −8 to 7. Q4_0 stores one fp16 step and a 4-bit level per value, the same shape. gguf_exact.py reads each block's trained step off its own values (the largest magnitude divided by 8, 7, … 1, whichever puts every value on an integer level, refined by least squares) and writes every weight matrix as Q4_0. It copies Google's metadata, tokenizer and chat template byte for byte.
hf download google/gemma-4-12B-it-qat-q4_0-unquantized --revision b6ed862 --local-dir src
hf download google/gemma-4-12B-it-qat-q4_0-gguf gemma-4-12b-it-qat-q4_0.gguf --revision 29d0977 --local-dir gguf
python3 gguf_exact.py src gguf/gemma-4-12b-it-qat-q4_0.gguf gemma-4-12B-it-q4_0-exact.gguf
It is the same script that built the E2B and E4B files, unchanged: the 12B's tensor names already match. It rebuilt 329 tensors in 17 minutes, and every block landed on the grid. Per tensor kind, 96.77–96.88% of values match Google's bf16 weights bit for bit once rounded back to bf16; the rest differ by the fp16 rounding of the step.
| Google's GGUF | exact rebuild | |
|---|---|---|
| File | 6,975,879,296 B | 6,716,356,736 B |
token_embd |
Q6_K, 787.5 MiB | Q4_0, 540.0 MiB |
| Weights on the card | 6,637.69 MiB | 6,390.19 MiB |
The vision encoder ships as a separate file, mmproj-gemma-4-12b-it-qat-q4_0.gguf (167 MiB). Leave it out and the model is text only, as the rebuild is. On an 8 GB card those 167 MiB are another 10,700 tokens of context (arithmetic).
Both rebuilds are on Hugging Face:
gemma-4-12B-it-q4_0-exact.gguf, SHA-256 038337ee0375f602…
gemma-4-12B-it-qat-assistant-q4_0-exact.gguf, SHA-256 f22f6366ca76102a…
The rig's make download and make draft fetch them and check the hashes in tpu.env:
make download
make draft
/home/xbill/models/gemma-4-12B-it-qat-assistant/gemma-4-12B-it-qat-assistant-q4_0-exact.gguf: OK
fit.sh starts llama-server -ngl 99 -fa 1 at rising context sizes and records the largest that loads, with the card otherwise idle:
| RTX 3050, 8,192 MiB | 16-bit cache | 8-bit cache ( q8_0 ) |
4-bit cache ( q4_0 ) |
|---|---|---|---|
| Google's GGUF | 28,672 tokens | not tested | not tested |
| exact rebuild | 40,960 tokens | 65,536 tokens | 131,072 tokens |
| exact rebuild + drafter | 24,576 tokens | 49,152 tokens | 81,920 tokens |
Google's file fails at 32,768 tokens; the rebuild loads there with 556 MiB to spare and fails at 49,152. The 8-bit cache halves every cache figure, sliding layers included, and the 4-bit cache halves it again. Step 10 measures what that costs.
The model's own limit is 131,072 tokens. Without the drafter, the 4-bit cache reaches it on this card.
bench.sh runs llama-bench, five repetitions, in four passes that alternate which file goes first, with the card cooled to 40 °C before each:
| pass | file | prompt, 512 tokens | generation, 128 tokens |
|---|---|---|---|
| 1 | 981.57 tok/s | 27.68 tok/s | |
| 2 | exact | 987.49 tok/s | 28.64 tok/s |
| 3 | exact | 986.97 tok/s | 28.64 tok/s |
| 4 | 977.83 tok/s | 27.68 tok/s |
The rebuild generates 1.035x faster, the same in both passes of each file, and reads prompts at the same speed. Every generated token reads the output layer in full, and the rebuild's output layer is 247.5 MiB smaller. The gain is smaller than E4B's 1.06x and E2B's 1.11x because the output layer is a smaller share of what a 12B reads per token (inferred, not measured).
Generation slows as the conversation grows:
| tokens already in context | 0 | 8,192 | 16,384 |
|---|---|---|---|
| prompt, 512 tokens | 981.09 tok/s | 801.58 tok/s | 692.32 tok/s |
| generation | 28.63 tok/s | 26.81 tok/s | 26.33 tok/s |
run_kld.sh scores the same text with Google's bf16 weights and with each 4-bit file, and compares the predicted next-token distributions (llama-perplexity --kl-divergence, wikitext-2, 16 chunks of 512 tokens, on the CPU):
| against bf16 | Google's GGUF | exact rebuild |
|---|---|---|
| Mean KL divergence | 0.543 | 0.125 |
| 99th percentile KL divergence | 6.217 | 1.915 |
| Same top token | 71.77% | 89.14% |
| Perplexity, relative to bf16 | 1.386 | 1.257 |
The rebuild is 4.4x closer to bf16 by mean divergence, and picks the same top token 89% of the time against 72%.
Compare the two columns with each other only. The 12B instruction-tuned model reads raw Wikipedia text poorly, at a bf16 perplexity of 312 against E4B's 40 on the same test, and every divergence is larger in that regime. E4B's rebuild scores 0.0010 on this test.
Google's drafter for this model is google/gemma-4-12B-it-qat-q4_0-unquantized-assistant: 4 layers of width 1,024 with queries only, reading the 12B's own KV cache, and a tied 262,144-token embedding. It ships as bf16 on a 4-bit grid.
llama.cpp's converter writes a Q8_0 file, which the drafter version of gguf_exact.py uses for metadata:
python3 convert_hf_to_gguf.py hf --outtype q8_0 --outfile gemma-4-12B-it-qat-assistant-q8_0.gguf
python3 gguf_exact.py hf gemma-4-12B-it-qat-assistant-q8_0.gguf gemma-4-12B-it-qat-assistant-q4_0-exact.gguf
Every block landed on the grid. The rebuild is 253,708,960 bytes against 465,128,096 for Q8_0, and its weights take 226.90 MiB on the card against 428.53 MiB. This drafter is 4.7x the size of the E2B and E4B drafters.
The rig serves with four extra flags for the drafter, and a 4-bit context cache (Step 10):
llama-server -m gemma-4-12B-it-q4_0-exact.gguf -ngl 99 -c 81920 -ctk q4_0 -ctv q4_0 -fa 1 --reasoning off \
-md gemma-4-12B-it-qat-assistant-q4_0-exact.gguf --spec-type draft-mtp -ngld all --spec-draft-n-max 2
spec_ab.py runs six prompts (code, advice, a story, an explanation, a list, a summary), 256 tokens each, three greedy repeats plus one sampled pass, one server per setting at -c 8192, the card cooled before each. Serving without a drafter ran first and last, at 28.1 tok/s both times.
| greedy, geometric mean of 6 prompts | tok/s | gain | drafted tokens kept | memory |
|---|---|---|---|---|
| no drafter | 28.1 | — | — | 7,255 MiB |
| Q8_0 drafter, 2 tokens | 53.4 | 1.90x | 0.661 | 7,771 MiB |
| exact Q4_0 drafter, 2 tokens | 56.4 | 2.01x | 0.658 | 7,569 MiB |
| exact Q4_0 drafter, 1 token | 45.6 | 1.62x | 0.759 | 7,569 MiB |
| exact Q4_0 drafter, 3 tokens | 56.7 | 2.02x | 0.561 | 7,569 MiB |
Per prompt at 2 tokens: code 2.40x, a list of facts 2.30x, advice 1.94x, an explanation 1.93x, a summary 1.82x, a story 1.75x. Sampled at temperature 0.8: 1.90x.
The drafter doubles the 12B's generation speed. The gain is larger than E4B's 1.72x on this card in Part 2: the 12B keeps 66% of drafted tokens against E4B's 48%.
For E2B and E4B the Q4_0 and Q8_0 drafters ran level. For the 12B the Q4_0 drafter is 1.06x faster (1.04–1.07x on each prompt), at the same share of tokens kept, and uses 202 MiB less memory. This drafter is 4.7x larger, so its own weight reads are likely a larger share of each drafted step (inferred, not measured). On an 8 GB card the 202 MiB is also about 12,900 tokens of context (arithmetic).
With the drafter on, greedy replies repeat exactly from run to run, but differ from the replies without a drafter in all 18 at 2 drafted tokens and 9 of 18 at 1. Checking several tokens in one pass runs different GPU kernels, which can change a near-tie word. Step 9 checks whether the answers change.
The rig's own settings, one request at a time: GSM8K (1,319 grade-school math questions, up to 2,048 tokens per reply) and BFCL simple (400 single function calls), with the drafter and without. E4B's scores are from Part 2's run of the same tests on the same card.
| one request at a time | 12B, drafter | 12B, no drafter | E4B, drafter | E4B, no drafter |
|---|---|---|---|---|
| GSM8K | 0.960 | 0.963 | 0.930 | 0.931 |
| BFCL simple | 0.950 | 0.948 | 0.913 | 0.913 |
The 12B answers more questions than E4B on both tests. Paired question by question, it is 3.0 points ahead on GSM8K (95% range 1.8 to 4.3) and 3.8 points ahead on BFCL (1.0 to 6.5), both with the drafter.
The drafter changes no score. On GSM8K the runs with and without it differ by 0.3 points (95% range −0.8 to +0.2), on BFCL by one question of 400. The greedy text differs, as in Step 8: 332 of 1,319 GSM8K replies match word for word. Without the drafter GSM8K took 2.34x as long, 4 h 16 min against 1 h 49 min.
The rig serves one long conversation at a time, so the context it holds and its speed with that context matter most. kv_sweep.py loads each cache type at its largest context, with and without the drafter, puts 0 to 98,304 tokens of a long document ahead of three questions, and measures generation (256 greedy tokens, geometric mean of the three):
| tokens already in context | 0 | 16,384 | 32,768 | 65,536 | 98,304 |
|---|---|---|---|---|---|
| 16-bit cache, drafter (24,576) | 57.5 tok/s | 49.8 | — | — | — |
| 8-bit cache, drafter (49,152) | 55.1 | 43.3 | 39.3 | — | — |
| 4-bit cache, drafter (81,920) | 55.1 | 42.4 | 37.8 | 30.8 | — |
| 4-bit cache, no drafter (131,072) | 27.5 | 24.1 | 22.2 | 19.2 | 16.8 |
The drafter keeps paying with a long conversation held: 2.00x at 0 tokens, 1.70x at 32,768, 1.60x at 65,536. A smaller cache costs the drafter more than it costs plain generation: at 16,384 tokens the 4-bit cache runs at 0.85x the 16-bit one with the drafter and 0.94x without. Prompt reading is level across cache types, about 720 tok/s at 16,384 tokens and 408 at 65,536.
The same GSM8K and BFCL runs as Step 9, at each cache's largest context, paired question by question against the 16-bit cache:
| one request at a time | GSM8K | BFCL simple |
|---|---|---|
| 16-bit cache, drafter | 0.960 | 0.950 |
| 8-bit cache, drafter | 0.960 | 0.948 |
| 4-bit cache, drafter | 0.961 | 0.950 |
| 4-bit cache, no drafter | 0.962 | 0.950 |
The cache type changes no score: every paired range includes zero. A recall test plants eight six-digit codes through the document and asks for all of them: the 16-bit, 8-bit and 4-bit caches return the same share, about 85% at 16,384 tokens and 70% at 32,768, and the drafter changes neither.
3 drafted tokens ran level with 2 at every depth and 1 ran well behind, so 2 stays. Larger prompt batches (-ub 1024) do not load beside the drafter.
| RTX 3050 OEM, 8 GB | 🥇 Gemma 4 12B | 🥈 Gemma 4 E4B |
|---|---|---|
| File | 6.72 GB | 4.22 GB |
| Generation, no drafter | 28.6 tok/s | 61.2 tok/s |
| Generation, 2-token drafter | 56.4 tok/s | 100.0 tok/s (Part 2) |
| Drafter gain | 2.01x | 1.72x |
| Context with the drafter, 16-bit cache | 24,576 tokens | 131,072 tokens |
| Context with the drafter, 4-bit cache | 81,920 tokens | not measured |
| GSM8K, with the drafter | 0.960 | 0.930 |
| BFCL simple, with the drafter | 0.950 | 0.913 |
E4B's numbers are from llama.cpp 77c4540ce (Part 2) except the no-drafter generation, measured beside the 12B on 7182ef71e.
For answers, the 12B. It scores 3 to 4 points higher than E4B on math and on function calls, and with the drafter it generates at 56.4 tok/s, near E4B's 61.2 without one.
For one long conversation, the 12B with a 4-bit cache and the drafter: 81,920 tokens at 55 tok/s with nothing held and 31 tok/s at 65,536, with the same answers as the 16-bit cache. Past 81,920 tokens, drop the drafter for the full 131,072 at about half the speed.
For speed, E4B. It generates at 100 tok/s with its drafter and holds the full 131,072-token context with a 16-bit cache.
Either way, serve with the drafter: it doubles the 12B's speed and costs no answers.
Google publishes the same QAT drafters for 26B-A4B and 31B, both too large for this card. The next measurements are the 12B under several requests at once, which share the 81,920 tokens, and the 12B on a 16 GB card with its full context and the drafter.
The goal of this article was to find out whether Gemma 4 12B, in Google's QAT build, runs on an 8 GB RTX 3050 with every layer on the GPU. It does. The key to the solution was rebuilding Google's GGUF with every weight on the 4-bit grid QAT trained, which frees the memory a working context needs, and rebuilding the 12B drafter the same way. The results were:
Scope: Gemma 4 12B-it QAT only, text only, one RTX 3050 OEM (8 GB) in an i7-5820K desktop, llama.cpp 7182ef71e, Google's 12B Q4_0 GGUF against the exact Q4_0 rebuild, Google's 12B QAT drafter as exact Q4_0 and Q8_0, six short prompts for speed, divergence on 8,192 tokens of wikitext-2 on the CPU, GSM8K and BFCL simple one request at a time, 16-bit, 8-bit and 4-bit context caches with 0 to 98,304 tokens of one document in context, one request at a time throughout. E4B comparisons from Part 2 used llama.cpp 77c4540ce.
The strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.