{"slug": "a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4", "title": "A 4 GB Laptop GPU Beats a 12-Core CPU by 4.3x on Gemma 4", "summary": "A developer benchmarked llama.cpp serving Google's Gemma 4 E2B Q4_0 GGUF on a single laptop, comparing CPU-only inference against the machine's 4 GB GTX 1650 Ti Max-Q, with the two arms differing only by the -ngl flag. The GPU arm delivered a median 4.27x faster decode (4.04x–4.34x across all eight prompt/output cells) and 3.63x faster prefill, with worst-case run-to-run spread of 6.29% on CPU versus 0.84% on GPU. The writeup attributes the gain to decode being bandwidth-bound while prefill is compute-bound, and notes the card holds only 1598 MiB because the 1.93 GB embedding tensor is served lazily from the mmap rather than loaded onto the GPU.", "body_md": "This article compares two ways of serving the same small language model on the same laptop: CPU-only, and on the 4 GB GTX 1650 Ti sitting in the same chassis. The payload is byte-identical on both arms and the command lines differ by a single flag. The card takes decode by **4.3x**.\n\nThe repository is at [https://github.com/xbill9/gemma4-dev](https://github.com/xbill9/gemma4-dev)\n\nOne machine, a 13th Gen Intel Core i7-1360P laptop with a GTX 1650 Ti (Max-Q) in it. Two arms:\n\n|  | CPU arm | GPU arm | \n|---|---|---|\n| Device | i7-1360P, 12 cores / 16 threads | GTX 1650 Ti Max-Q, 4096 MiB | \n| Topology | 4 SMT P-cores (0-7) + 8 E-cores (8-15) | TU117, compute capability 7.5, **no tensor cores** | \n| SIMD / math | `avx2` ,`avx_vnni` ,**no AVX-512** | CUDA | \n| llama.cpp | `c6824a9` ,`GGML_CUDA=OFF` build | `c6824a9` , CUDA build | \n| Flag that differs | `-ngl 0` | `-ngl 99` | \n\nEverything else matches, and that is the whole exercise:\n\n```\n-m gemma-4-E2B_q4_0-it.gguf --host 127.0.0.1 --port 8080 \\\n  -ngl {0|99} -c 8192 -ctk f16 -ctv f16 -fa 1 -t 4 -tb 8 --parallel 1 --metrics\n```\n\nThe model is `google/gemma-4-E2B-it-qat-q4_0-gguf`, a 3.35 GB quantization-aware GGUF. The two arms were run alternately against the same endpoint, CPU first, with a fixed 120 second cooldown between them.\n\nEight cells: four prompt lengths by two output lengths, three repeats each, concurrency 1. Decode is client-side inter-token rate measured off the SSE stream, which is the only decode statistic both arms can produce.\n\n| in tok | out tok | CPU decode | GPU decode | 🥇 | CPU TTFT ms | GPU TTFT ms | 🥇 | \n|---|---|---|---|---|---|---|---|\n| 94 | 32 | 17.61 | 71.22 | **4.04x** | 1143 | 385 | **2.97x** | \n| 94 | 128 | 17.22 | 71.20 | **4.13x** | 1219 | 385 | **3.17x** | \n| 516 | 32 | 16.16 | 69.93 | **4.33x** | 5978 | 1610 | **3.71x** | \n| 516 | 128 | 16.39 | 69.49 | **4.24x** | 5995 | 1612 | **3.72x** | \n| 998 | 32 | 15.81 | 68.60 | **4.34x** | 11748 | 3244 | **3.62x** | \n| 998 | 128 | 16.10 | 68.52 | **4.26x** | 11532 | 3242 | **3.56x** | \n| 1959 | 32 | 15.66 | 67.61 | **4.32x** | 23926 | 6522 | **3.67x** | \n| 1959 | 128 | 15.80 | 67.64 | **4.28x** | 23773 | 6522 | **3.65x** | \n\nMedians over the eight cells: decode **4.27x**, prefill **3.63x**, end-to-end **3.81x**.\n\nWithin-cell spread over three repeats was **6.29%** worst case on the CPU arm and **0.84%** on the GPU arm. The decode ratio spans 4.04x to 4.34x across every cell. The effect is an order of magnitude clear of the noise, which is the only reason it is worth reporting at all.\n\nLook down the decode columns rather than across them.\n\nCPU decode moves from 17.61 to 15.66 tok/s. GPU decode moves from 71.22 to 67.61. Across a **21x range of prompt length**, on both arms, decode barely moves. TTFT over the same range goes from 1143 ms to 23926 ms on the CPU and 385 ms to 6522 ms on the GPU — linear in the prompt.\n\nThat is the expected shape stated twice. Decode reads the whole model per token and is bandwidth-bound; prefill multiplies through the prompt and is compute-bound. An accelerator helps both, but it helps them for different reasons, and a benchmark that quotes one number hides that.\n\nThe decode *ratio* also climbs with context — 4.04x at 94 tokens, 4.34x at 998. The CPU arm loses ground as the KV cache grows and the GPU arm barely does.\n\nIt does not have to. With the model loaded and serving, the card holds **1598 MiB**.\n\nThe artifact is only 32% Q4_0. Both embedding tensors are Q6_K and account for 2.257 GB of the 3.334 GB of tensor bytes, and the largest of them — `per_layer_token_embd`, at 1.93 GB, 58% of the file — is created with `TENSOR_READ_LAZY` in llama.cpp's `src/models/gemma4.cpp` and served by `GGML_OP_GET_ROWS` straight out of the mmap. A few rows are touched per token. It never goes to the card.\n\nWhat is resident is the ~1.08 GB Q4_0 transformer body, which is what decode reads every token, plus about 60 MiB of KV cache and the compute buffers.\n\nThe same mechanism costs something on the CPU arm instead of saving something. `RssAnon` there is 1,202,152 kB, because llama.cpp repacks Q4_0 weights into an interleaved layout for its AVX2 kernels, copying the body out of the mmap into anonymous memory.\n\nSo the headline works because of the checkpoint's shape, not in spite of the card's size. A 4 GB GPU from 2021 is not too small for this model. It is roughly 2.5x larger than it needs to be.\n\nThe two arms differ by one flag. Everything else was held equal deliberately, and checked rather than assumed:\n\n|  | How it was held | \n|---|---|\n| Engine | one llama.cpp commit, `c6824a9` , built twice — CPU build and CUDA build | \n| Binary identity | the SHA-256 of each running executable is recorded in its report | \n| Flags | identical apart from `-ngl` , including`-t 4 -tb 8` on both arms | \n| Prompts | one harness, byte-identical in both rigs, with device-neutral filler | \n| Prompt lengths | every paired cell matched exactly — 94, 516, 998 and 1959 tokens | \n| Endpoint | the same `127.0.0.1:8080` , one arm running at a time | \n\nThat last row is the one worth dwelling on. Both arms serve the same model on the same port, so an HTTP response says nothing about which device produced it. **The device is therefore read from the running process rather than taken from a label** — `/proc/<pid>/exe` for the binary, `/proc/<pid>/maps` for the ggml backends actually loaded, `/proc/<pid>/cmdline` for the real `-ngl` — and that attestation is stamped into every report beside the numbers.\n\n`maps` rather than `ldd`, because llama.cpp `dlopen` s its backends: a CUDA backend can be absent from `ldd` output and present in the running process. The verdict needs a GPU backend mapped in **and** layers assigned to it, since a CUDA build running `-ngl 0` computes on the CPU but is not a clean CPU arm — the device is initialised and large prefill batches can still land on it. The CPU arm hides the device from the process entirely rather than trusting `-ngl 0`.\n\nRead the 4x as real and anything under about 7% as nothing.\n\n`llama-bench` sweep on this\ndie found affinity worth `--parallel 1`.\nThe goal of this article was to measure what a 4 GB laptop GPU is worth against a modern 12-core CPU for serving a small quantized model. The key to the solution was a single-variable comparison: one llama.cpp commit, one prompt set, one flag apart, with the serving device read from `/proc` at run time and recorded beside every number. The results were:\n\nScope: one laptop, one GGUF, llama.cpp `c6824a9` built twice from the same commit, eight paired cells, three repeats per cell, concurrency 1, CPU arm first with a 120 second cooldown. CPU affinity was not set on either arm and the page cache was warm for both.\n\nThe strategy for using MCP for local accelerator comparison was validated with an incremental step by step approach.", "url": "https://wpnews.pro/news/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4", "canonical_source": "https://dev.to/gde/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-43x-on-gemma-4-4150", "published_at": "2026-09-16 20:03:02+00:00", "updated_at": "2026-09-16 20:23:46.505641+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-chips", "mlops", "developer-tools"], "entities": ["llama.cpp", "Gemma 4", "Google", "GTX 1650 Ti Max-Q", "Intel Core i7-1360P", "CUDA", "GGUF", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4", "markdown": "https://wpnews.pro/news/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4.md", "text": "https://wpnews.pro/news/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4.txt", "jsonld": "https://wpnews.pro/news/a-4-gb-laptop-gpu-beats-a-12-core-cpu-by-4-3x-on-gemma-4.jsonld"}}