{"slug": "how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu", "title": "How I Made Local LLMs Up to 2.24× Faster on a 4GB NVIDIA GPU", "summary": "Tuning llama.cpp configuration on a 4GB NVIDIA GeForce GTX 1650 Ti produced local LLM throughput gains of up to 2.24× with no hardware upgrades, retraining, or CUDA kernel changes, according to the author's two-round automated benchmark. The study tested three Q4_K_M GGUF models — Gemma 4 E2B IT (3.22 GiB), Qwen3.5-4B (2.55 GiB), and Qwen3-4B (2.33 GiB) — across 48 configurations combining GPU layer placement (auto, 12, 24, all), 4 or 6 CPU threads, and microbatch sizes of 128 or 256, with a 4096 context size and batch size of 512. The central finding was that llama.cpp's automatic GPU layer selection was not always the fastest: switching to --gpu-layers all dramatically sped up Qwen3 and Qwen3.5, while Gemma 4 saw almost no improvement, indicating GPU offloading should be tuned per model.", "body_md": "When people talk about running large language models locally, the conversation usually starts with expensive hardware: RTX 4090s, 24GB of VRAM, or dedicated AI workstations.\n\nI wanted to answer a different question:\n\n**How much performance can we extract from a laptop GPU with only 4GB of VRAM?**\n\nMy test machine uses an NVIDIA GeForce GTX 1650 Ti, a GPU released years before the current generative AI boom. With just 4GB of GDDR6 memory, it is hardly the first card anyone would recommend for running modern language models.\n\nYet after two rounds of systematic benchmarking with llama.cpp, the results were surprisingly encouraging.\n\nBy tuning GPU layer placement, CPU threads, microbatch sizes, and KV cache precision, I measured:\n\nNo hardware upgrades. No retraining. No modifications to CUDA kernels.\n\nJust better configuration.\n\nThe most important discovery was that **the automatic GPU configuration was not always the fastest**, even when llama.cpp was already running with CUDA acceleration.\n\nHere is how I tested it, what worked, and how to reproduce these\n\nI tested three approximately four-bit-quantized language models:\n\nTested models\n\n```\nGemma 4 E2B IT  Quantization: Q4_K_M  GGUF size:    3.22 GiBQwen3.5-4B  Quantization: Q4_K_M  GGUF size:    2.55 GiBQwen3-4B  Quantization: Q4_K_M  GGUF size:    2.33 GiB\n```\n\nThe results refer specifically to quantized GGUF inference, not full-precision inference.\n\nAn important lesson appeared early: a model’s file size does not tell us exactly how much GPU memory inference will require.\n\nSome tensors can remain in CPU-accessible memory, while others are placed on the GPU. Different model architectures can therefore behave quite differently on the same hardware.\n\nThat became especially relevant when comparing Gemma 4 with the Qwen models.\n\nllama.cpp allows us to control how many model layers are assigned to the GPU using the --gpu-layers option.\n\nThe configurations I tested included:\n\n```\n--gpu-layers auto--gpu-layers 12--gpu-layers 24--gpu-layers all\n```\n\nAutomatic selection is a sensible starting point on memory-constrained hardware.\n\nBut it is not necessarily the configuration with the highest throughput.\n\nInitially, I used auto for every model. All three models could generate responses, but the Qwen models were performing well below what the GPU could actually deliver.\n\nWhen I switched to --gpu-layers all, Qwen3 and Qwen3.5 became dramatically faster.\n\nGemma 4, however, saw almost no improvement.\n\nThis gave us the central finding of the study:\n\nGPU offloading should be tuned for each model rather than treated as one universal setting.\n\nI designed two rounds of experiments rather than relying on a few manually timed prompts.\n\nThe benchmark was automated with PowerShell scripts that launched llama-server, waited for readiness, issued API requests, collected timing information, recorded GPU memory usage, and saved results to CSV files.\n\nThe first experiment tested four GPU offloading strategies, two CPU thread counts, and two microbatch sizes.\n\n```\nGPU layers:   auto, 12, 24, allCPU threads:  4, 6Microbatch:   128, 256\n4 × 2 × 2 = 16 configs/model\n3 models × 16 = 48 configurations\n```\n\nThe test ran under a two-hour time budget, with up to three repetitions of each configuration.\n\nWe used five types of prompts: factual questions, humor, Serbian-to-English translation, English-to-Serbian translation, and prompt improvement.\n\nThe core inference settings remained consistent:\n\n```\nContext size:         4096Batch size:           512Concurrent servers:   1Warm-up:              enabledTemperature:          0GPU monitoring:       enabled\n```\n\nFor each request, the script captured generation throughput, total latency, completion status, GPU memory usage, and related performance information.\n\nOne methodological issue became apparent: the first benchmark limited output to 192 tokens, causing some longer responses to end before they were complete.\n\nThose samples still provided generation-speed observations, but they could not be treated as completed user tasks.\n\nToken generation throughput and task-completion time are different measurements.\n\nAfter identifying promising GPU/CPU parameters, I ran a second experiment focused on longer technical responses and KV cache optimization.\n\nThe second phase evaluated ten model configurations in two rounds, comparing FP16, Q8_0, and Q4_0 cache formats where appropriate.\n\nEach configuration used one warm-up request and two measured requests.\n\nThe output limit increased to 512 tokens.\n\nAll 40 measured responses completed normally, with finish_reason=stop.\n\nThis strengthened our confidence that the shortlisted configurations could handle more realistic requests without immediately exhausting the generation limit.\n\nThe most dramatic improvement came from GPU layer placement.\n\nFor an apples-to-apples comparison, I used results with the same CPU thread count and microbatch size, changing only GPU offloading between auto and all.\n\nGPU offloading comparison\n\n```\nQwen3.5-4B  GPU auto:    15.63 tok/s  GPU all:     34.96 tok/s  Speedup:      2.24×\nQwen3-4B  GPU auto:    20.46 tok/s  GPU all:     42.41 tok/s  Speedup:      2.07×\nGemma 4 E2B  GPU auto:    59.99 tok/s  GPU all:     59.78 tok/s  Speedup:     ~1.00×\n```\n\nFor Qwen3.5, that represents approximately 124% more generation throughput.\n\nFor Qwen3, approximately 107% more.\n\nThese are substantial gains from configuration changes alone.\n\nThe improvement did come with a GPU memory tradeoff.\n\nGPU memory use\n\n```\nAuto       AllQwen3.5-4B      2231 MiB   3089 MiBQwen3-4B        2281 MiB   3163 MiB\n```\n\nIn other words, the fastest configurations used more of the available GPU memory.\n\nThat is not necessarily a disadvantage when the GPU is dedicated to inference.\n\nUnused VRAM does not provide a performance benefit by itself. The question is whether using more VRAM reduces expensive CPU/GPU transfers and improves actual inference execution.\n\nIn these measurements, it clearly did.\n\nBut there is a limit. On a 4GB card, reserving enough GPU memory for runtime allocations, the operating system, and other applications remains important.\n\nGemma 4 E2B consistently achieved close to 60 generated tokens per second.\n\nInterestingly, changing its GPU setting from auto to all produced almost no difference.\n\nThe runtime reported 36 of 36 relevant layers as offloaded, while still retaining substantial tensor data in CPU-mapped memory.\n\nObserved model buffers\n\n```\nGemma 4 E2B\nGPU model buffer:   ~1450 MiBCPU-mapped buffer:  ~2153 MiBGPU layers:         36 / 36\n```\n\nThis does not mean that the entire model occupies only 1450 MiB. It means the runtime divides its tensors between different memory locations.\n\nIn a separate exploratory test, I forced additional Gemma tensors onto the GPU.\n\nThe GPU model buffer increased to roughly 3602 MiB, leaving very little VRAM headroom.\n\nThat did not improve practical responsiveness.\n\nThe lesson was clear:\n\nMore GPU-resident tensors do not guarantee faster inference if the resulting memory pressure harms execution.\n\nI also compared microbatch values of 256 and 128.\n\nGemma 4 microbatch comparison\n\n```\nub=256     ub=128\nSpeed           58.99      59.31 tok/sLatency          7.05       7.00 secGPU memory       1813       1685 MiBCompleted        4/4        4/4\n```\n\nThe smaller microbatch saved approximately 128 MiB with effectively unchanged performance.\n\nTherefore, I selected -ub 128 for Gemma.\n\nOne useful implication is that microbatch size should not simply be maximized because more GPU memory is available. On a single-user workload, a smaller value may be equally fast while providing more memory headroom.\n\nOnce the strongest GPU offload configurations were identified, I wanted to reduce VRAM usage without losing too much throughput.\n\nThat led to the second major part of the study: KV cache quantization.\n\nAutoregressive transformer inference stores information from earlier tokens in a key-value cache.\n\nAs context length grows, this cache can use substantial memory.\n\nllama.cpp allows the cache formats to be configured with:\n\n```\n--cache-type-k--cache-type-v\n```\n\nThe experiment compared FP16, Q8_0, and Q4_0.\n\nThe Qwen3.5 results were:\n\n```\nQwen3.5-4B\nKV cache   Speed       GPU memory---------------------------------FP16       34.58 tok/s   3090 MiBQ8_0       34.21 tok/s   3041 MiBQ4_0       34.14 tok/s   3009 MiB\n```\n\nQ4_0 reduced observed GPU memory use by approximately 81 MiB, while generation speed decreased by about 1.3%.\n\nThis is a reasonable tradeoff, but not an especially compelling one for this configuration.\n\nThe server logs also showed that Qwen3.5 uses additional recurrent state, alongside its KV cache.\n\nMy conclusion was to retain FP16 for Qwen3.5, unless a particular application requires additional VRAM headroom.\n\nQwen3 showed a much larger benefit.\n\n```\nQwen3-4B\nKV cache   Speed       GPU memory---------------------------------FP16       41.59 tok/s   3163 MiBQ8_0       39.92 tok/s   2903 MiBQ4_0       39.80 tok/s   2759 MiB\n```\n\nThe tradeoffs were striking.\n\nQ8_0 saved 260 MiB, with a generation-speed reduction of approximately 4%.\n\nQ4_0 saved 404 MiB, with a roughly 4.3% reduction.\n\nOn a GPU with only 4GB of VRAM, hundreds of megabytes are meaningful.\n\nQ8_0 became my preferred balance: nearly all the measured performance of FP16, but with substantially more free GPU memory.\n\nThere is a caveat: this experiment measured timing, memory, and successful completion. It did not formally evaluate whether KV quantization changed semantic accuracy or reasoning quality.\n\nThat would require a separate task-quality benchmark.\n\nWhile benchmarking, I encountered a practical issue familiar to anyone using reasoning-capable LLMs.\n\nSome models can spend a large number of tokens on internal reasoning before producing visible output.\n\nFor complex mathematical reasoning or difficult planning tasks, this can be useful.\n\nBut for basic coding questions, translations, summaries, and direct explanations, the additional delay may be undesirable.\n\nThe tested llama.cpp build supports these options:\n\n```\n--reasoning off --reasoning-budget 0\n```\n\nI used them to configure the models for direct responses.\n\nIn practice, this made short interactive exchanges more usable and helped avoid cases where the output-token allowance was exhausted before the visible answer was complete.\n\nHowever, there is an important distinction.\n\nDisabling thinking does not necessarily make the GPU generate individual tokens faster. Instead, it can reduce the total number of generated tokens required before the user receives an answer.\n\nThe 2.24× and 2.07× GPU-offloading gains reported earlier should not be attributed to reasoning settings.\n\nI also would not recommend disabling reasoning for every possible task. For problems where deliberate multi-step reasoning matters, a reasoning-enabled configuration may be the better choice.\n\nThe best setting depends on the purpose of the model.\n\nCombining the results of both experiments, I selected the following practical configurations.\n\nRecommended llama.cpp profiles\n\n```\nGEMMA 4 E2B  GPU layers:   auto  CPU threads:  4  Context:      4096  Batch:        512  Microbatch:   128  KV cache:     f16 / f16  Reasoning:    off\nQWEN3.5-4B  GPU layers:   all  CPU threads:  6  Context:      4096  Batch:        512  Microbatch:   256  KV cache:     f16 / f16  Reasoning:    off\nQWEN3-4B  GPU layers:   all  CPU threads:  6  Context:      4096  Batch:        512  Microbatch:   256  KV cache:     q8_0 / q8_0  Reasoning:    off\n```\n\nThese should be viewed as tested profiles for my hardware, not universal defaults.\n\nDifferent GPUs, driver versions, model quantizations, context sizes, and concurrency settings can produce different outcomes.\n\nHere is how to reproduce the configuration on Windows.\n\nI used the following directory structure:\n\n```\nC:\\llama.cpp\\  bin\\    llama-cli.exe    llama-server.exe  models\\    google_gemma-4-E2B-it-Q4_K_M.gguf    Qwen3.5-4B-Q4_K_M.gguf    Qwen3-4B-Q4_K_M.gguf\n```\n\nThe executable names and directory layout may differ depending on how you install llama.cpp.\n\nFirst, open PowerShell and verify that Windows detects the NVIDIA GPU:\n\n```\nnvidia-smi\n```\n\nThen check your llama.cpp installation:\n\n```\ncd C:\\llama.cpp.\\bin\\llama-cli.exe --help\n```\n\nReview the available flags before copying any configuration.\n\nOne practical challenge I encountered was CUDA toolchain compatibility. A newer prebuilt llama.cpp binary failed with an unsupported PTX toolchain error on my machine, while build b8729 worked correctly.\n\nThe important point is not that everyone should use b8729. It is that a binary and its packaged CUDA components must be compatible with the installed NVIDIA driver.\n\n```\n.\\bin\\llama-cli.exe `  -m \".\\models\\google_gemma-4-E2B-it-Q4_K_M.gguf\" `  --gpu-layers auto `  -t 4 `  -c 4096 `  -b 512 `  -ub 128 `  --cache-type-k f16 `  --cache-type-v f16 `  --reasoning off `  --reasoning-budget 0\n```\n\nThis profile produced approximately 59 tokens per second in the validation experiment.\n\n```\n.\\bin\\llama-cli.exe `  -m \".\\models\\Qwen3.5-4B-Q4_K_M.gguf\" `  --gpu-layers all `  -t 6 `  -c 4096 `  -b 512 `  -ub 256 `  --cache-type-k f16 `  --cache-type-v f16 `  --reasoning off `  --reasoning-budget 0\n```\n\nWith maximum GPU layer offloading requested, Qwen3.5 achieved about 34.6 generated tokens per second in the longer-response validation experiment.\n\n```\n.\\bin\\llama-cli.exe `  -m \".\\models\\Qwen3-4B-Q4_K_M.gguf\" `  --gpu-layers all `  -t 6 `  -c 4096 `  -b 512 `  -ub 256 `  --cache-type-k q8_0 `  --cache-type-v q8_0 `  --reasoning off `  --reasoning-budget 0\n```\n\nThis configuration achieved approximately 39.9 generated tokens per second while using roughly 2,903 MiB of GPU memory.\n\nFor the highest measured Qwen3 throughput, FP16 KV cache remains an alternative, at the cost of more VRAM.\n\nFor terminal interactions, llama-cli is convenient.\n\nBut if you want to integrate local inference with your own software, llama-server offers an OpenAI-compatible API.\n\nFor example, here is the optimized Qwen3 server configuration:\n\n```\n.\\bin\\llama-server.exe `  -m \".\\models\\Qwen3-4B-Q4_K_M.gguf\" `  --host 127.0.0.1 `  --port 8080 `  --gpu-layers all `  -t 6 `  -c 4096 `  -b 512 `  -ub 256 `  --cache-type-k q8_0 `  --cache-type-v q8_0 `  --reasoning off `  --reasoning-budget 0 `  --no-webui\n```\n\nOnce started, the API base URL is:\n\n```\nhttp://127.0.0.1:8080/v1\n```\n\nTo check that the server is responding:\n\n```\nInvoke-RestMethod `  http://127.0.0.1:8080/v1/models\n```\n\nTo send a chat completion request:\n\n``` php\n$body = @{    model = \"Qwen3-4B-Q4_K_M.gguf\"    messages = @(        @{            role = \"user\"            content = \"Explain AES-256 simply.\"        }    )    temperature = 0    max_tokens = 256} | ConvertTo-Json -Depth 5\nphp\n$result = Invoke-RestMethod `    -Uri \"http://127.0.0.1:8080/v1/chat/completions\" `    -Method Post `    -ContentType \"application/json\" `    -Body $body\n$result.choices[0].message.content\n```\n\nIf your server reports a different model identifier from /v1/models, use that identifier instead.\n\nTo use the built-in browser interface, omit --no-webui on builds that include the web interface.\n\nThis example binds the server to 127.0.0.1, restricting access to the local machine. Do not expose an unauthenticated inference server directly to the internet.\n\nIf you have a GTX 1650, GTX 1660, RTX 3050 4GB, or another memory-constrained NVIDIA GPU, I would recommend the following approach.\n\nStart with a correctness baseline. Verify that the model loads, completes prompts, and produces sensible answers before measuring performance.\n\nTest GPU offloading independently. Compare automatic layer selection with explicitly requested layer counts. Do not assume maximum offloading is always fastest or always feasible.\n\nTune CPU threads and microbatch size. Higher numbers may increase memory pressure without improving single-user token generation.\n\nInspect memory allocation. Separate model weights, KV cache, compute buffers, and other runtime allocations wherever logs permit.\n\nEvaluate cache quantization. Especially on smaller GPUs, the savings from Q8_0 or Q4_0 can be worth considering.\n\nMeasure complete answers. Report token generation speed and total response latency separately. A truncated response should not be counted as a fully completed task.\n\nRepeat measurements. A configuration should be selected based on reproducible observations, not one unusually fast run.\n\nMost importantly, record the exact model file, quantization, inference engine version, driver, and GPU for every benchmark.\n\nWithout that context, performance comparisons become difficult to reproduce.\n\nThere are limits to what can be concluded from this study.\n\nAll measurements came from one Windows 10 laptop with an NVIDIA GTX 1650 Ti and a single llama.cpp build. Other GPUs and software versions may behave differently.\n\nThe three tested models also have different architectures. A higher tokens-per-second figure does not establish superior model quality.\n\nThe experiment focused on one inference request at a time, not concurrent multi-user serving.\n\nThe validation workload contained two technical prompt types and four measured responses per configuration, providing useful repeatability checks but not a large-scale statistical characterization.\n\nAnd although the quantized KV configurations completed requests successfully, we did not formally benchmark answer quality under different cache precisions.\n\nFinally, disabling thinking may improve responsiveness for simple tasks while weakening performance on difficult reasoning problems.\n\nThese results should therefore be interpreted as measurements of local inference performance, not general model-quality rankings.\n\nBefore running these experiments, I expected a 4GB laptop GPU to impose severe limits on local LLM inference.\n\nThose limits are real. But the performance available within them turned out to be more useful than I initially expected.\n\nWith carefully chosen configurations, I measured:\n\n```\nGemma 4 E2B  ~59 tok/s  GPU auto, FP16 KV\nQwen3.5-4B  ~35 tok/s  GPU all, FP16 KV\nQwen3-4B  ~42 tok/s  GPU all, FP16 KV\nQwen3-4B (memory-balanced)  ~40 tok/s  GPU all, Q8 KV  260 MiB less GPU memory than FP16\n```\n\nThe biggest performance improvement came from requesting more GPU layer offloading for models that could benefit from it.\n\nKV cache quantization offered a second optimization opportunity: recovering GPU memory with a relatively small reduction in throughput.\n\nAnd disabling unnecessary reasoning helped keep interactive conversations focused on direct answers.\n\nNone of these adjustments required purchasing additional hardware.\n\nThe broader takeaway is straightforward:\n\nBefore buying a faster GPU, measure whether your inference engine is making effective use of the one you already own.\n\nOn hardware with only 4GB of VRAM, the right settings can make the difference between a sluggish local model and a genuinely usable one.\n\nExperimental setup: NVIDIA GeForce GTX 1650 Ti Laptop GPU (4GB), Intel Core i5–10300H, Windows 10, llama.cpp build b8729, October 2026. Performance measurements are rounded medians from the two experimental phases.\n\n[How I Made Local LLMs Up to 2.24× Faster on a 4GB NVIDIA GPU](https://pub.towardsai.net/how-i-made-local-llms-up-to-2-24-faster-on-a-4gb-nvidia-gpu-e0c02f2b1821) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu", "canonical_source": "https://pub.towardsai.net/how-i-made-local-llms-up-to-2-24-faster-on-a-4gb-nvidia-gpu-e0c02f2b1821?source=rss----98111c9905da---4", "published_at": "2026-10-20 06:55:50+00:00", "updated_at": "2026-10-20 07:16:23.940399+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "machine-learning"], "entities": ["llama.cpp", "NVIDIA", "GeForce GTX 1650 Ti", "Gemma 4 E2B IT", "Qwen3.5-4B", "Qwen3-4B", "GGUF", "CUDA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu", "markdown": "https://wpnews.pro/news/how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu.md", "text": "https://wpnews.pro/news/how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu.txt", "jsonld": "https://wpnews.pro/news/how-i-made-local-llms-up-to-2-24x-faster-on-a-4gb-nvidia-gpu.jsonld"}}