# How I Made Local LLMs Up to 2.24× Faster on a 4GB NVIDIA GPU > Source: > Published: 2026-10-20 06:55:50+00:00 When people talk about running large language models locally, the conversation usually starts with expensive hardware: RTX 4090s, 24GB of VRAM, or dedicated AI workstations. I wanted to answer a different question: **How much performance can we extract from a laptop GPU with only 4GB of VRAM?** My test machine uses an NVIDIA GeForce GTX 1650 Ti, a GPU released years before the current generative AI boom. With just 4GB of GDDR6 memory, it is hardly the first card anyone would recommend for running modern language models. Yet after two rounds of systematic benchmarking with llama.cpp, the results were surprisingly encouraging. By tuning GPU layer placement, CPU threads, microbatch sizes, and KV cache precision, I measured: No hardware upgrades. No retraining. No modifications to CUDA kernels. Just better configuration. The most important discovery was that **the automatic GPU configuration was not always the fastest**, even when llama.cpp was already running with CUDA acceleration. Here is how I tested it, what worked, and how to reproduce these I tested three approximately four-bit-quantized language models: Tested models ``` Gemma 4 E2B IT Quantization: Q4_K_M GGUF size: 3.22 GiBQwen3.5-4B Quantization: Q4_K_M GGUF size: 2.55 GiBQwen3-4B Quantization: Q4_K_M GGUF size: 2.33 GiB ``` The results refer specifically to quantized GGUF inference, not full-precision inference. An important lesson appeared early: a model’s file size does not tell us exactly how much GPU memory inference will require. Some tensors can remain in CPU-accessible memory, while others are placed on the GPU. Different model architectures can therefore behave quite differently on the same hardware. That became especially relevant when comparing Gemma 4 with the Qwen models. llama.cpp allows us to control how many model layers are assigned to the GPU using the --gpu-layers option. The configurations I tested included: ``` --gpu-layers auto--gpu-layers 12--gpu-layers 24--gpu-layers all ``` Automatic selection is a sensible starting point on memory-constrained hardware. But it is not necessarily the configuration with the highest throughput. Initially, I used auto for every model. All three models could generate responses, but the Qwen models were performing well below what the GPU could actually deliver. When I switched to --gpu-layers all, Qwen3 and Qwen3.5 became dramatically faster. Gemma 4, however, saw almost no improvement. This gave us the central finding of the study: GPU offloading should be tuned for each model rather than treated as one universal setting. I designed two rounds of experiments rather than relying on a few manually timed prompts. The benchmark was automated with PowerShell scripts that launched llama-server, waited for readiness, issued API requests, collected timing information, recorded GPU memory usage, and saved results to CSV files. The first experiment tested four GPU offloading strategies, two CPU thread counts, and two microbatch sizes. ``` GPU layers: auto, 12, 24, allCPU threads: 4, 6Microbatch: 128, 256 4 × 2 × 2 = 16 configs/model 3 models × 16 = 48 configurations ``` The test ran under a two-hour time budget, with up to three repetitions of each configuration. We used five types of prompts: factual questions, humor, Serbian-to-English translation, English-to-Serbian translation, and prompt improvement. The core inference settings remained consistent: ``` Context size: 4096Batch size: 512Concurrent servers: 1Warm-up: enabledTemperature: 0GPU monitoring: enabled ``` For each request, the script captured generation throughput, total latency, completion status, GPU memory usage, and related performance information. One methodological issue became apparent: the first benchmark limited output to 192 tokens, causing some longer responses to end before they were complete. Those samples still provided generation-speed observations, but they could not be treated as completed user tasks. Token generation throughput and task-completion time are different measurements. After identifying promising GPU/CPU parameters, I ran a second experiment focused on longer technical responses and KV cache optimization. The second phase evaluated ten model configurations in two rounds, comparing FP16, Q8_0, and Q4_0 cache formats where appropriate. Each configuration used one warm-up request and two measured requests. The output limit increased to 512 tokens. All 40 measured responses completed normally, with finish_reason=stop. This strengthened our confidence that the shortlisted configurations could handle more realistic requests without immediately exhausting the generation limit. The most dramatic improvement came from GPU layer placement. For an apples-to-apples comparison, I used results with the same CPU thread count and microbatch size, changing only GPU offloading between auto and all. GPU offloading comparison ``` Qwen3.5-4B GPU auto: 15.63 tok/s GPU all: 34.96 tok/s Speedup: 2.24× Qwen3-4B GPU auto: 20.46 tok/s GPU all: 42.41 tok/s Speedup: 2.07× Gemma 4 E2B GPU auto: 59.99 tok/s GPU all: 59.78 tok/s Speedup: ~1.00× ``` For Qwen3.5, that represents approximately 124% more generation throughput. For Qwen3, approximately 107% more. These are substantial gains from configuration changes alone. The improvement did come with a GPU memory tradeoff. GPU memory use ``` Auto AllQwen3.5-4B 2231 MiB 3089 MiBQwen3-4B 2281 MiB 3163 MiB ``` In other words, the fastest configurations used more of the available GPU memory. That is not necessarily a disadvantage when the GPU is dedicated to inference. Unused VRAM does not provide a performance benefit by itself. The question is whether using more VRAM reduces expensive CPU/GPU transfers and improves actual inference execution. In these measurements, it clearly did. But there is a limit. On a 4GB card, reserving enough GPU memory for runtime allocations, the operating system, and other applications remains important. Gemma 4 E2B consistently achieved close to 60 generated tokens per second. Interestingly, changing its GPU setting from auto to all produced almost no difference. The runtime reported 36 of 36 relevant layers as offloaded, while still retaining substantial tensor data in CPU-mapped memory. Observed model buffers ``` Gemma 4 E2B GPU model buffer: ~1450 MiBCPU-mapped buffer: ~2153 MiBGPU layers: 36 / 36 ``` This does not mean that the entire model occupies only 1450 MiB. It means the runtime divides its tensors between different memory locations. In a separate exploratory test, I forced additional Gemma tensors onto the GPU. The GPU model buffer increased to roughly 3602 MiB, leaving very little VRAM headroom. That did not improve practical responsiveness. The lesson was clear: More GPU-resident tensors do not guarantee faster inference if the resulting memory pressure harms execution. I also compared microbatch values of 256 and 128. Gemma 4 microbatch comparison ``` ub=256 ub=128 Speed 58.99 59.31 tok/sLatency 7.05 7.00 secGPU memory 1813 1685 MiBCompleted 4/4 4/4 ``` The smaller microbatch saved approximately 128 MiB with effectively unchanged performance. Therefore, I selected -ub 128 for Gemma. One useful implication is that microbatch size should not simply be maximized because more GPU memory is available. On a single-user workload, a smaller value may be equally fast while providing more memory headroom. Once the strongest GPU offload configurations were identified, I wanted to reduce VRAM usage without losing too much throughput. That led to the second major part of the study: KV cache quantization. Autoregressive transformer inference stores information from earlier tokens in a key-value cache. As context length grows, this cache can use substantial memory. llama.cpp allows the cache formats to be configured with: ``` --cache-type-k--cache-type-v ``` The experiment compared FP16, Q8_0, and Q4_0. The Qwen3.5 results were: ``` Qwen3.5-4B KV cache Speed GPU memory---------------------------------FP16 34.58 tok/s 3090 MiBQ8_0 34.21 tok/s 3041 MiBQ4_0 34.14 tok/s 3009 MiB ``` Q4_0 reduced observed GPU memory use by approximately 81 MiB, while generation speed decreased by about 1.3%. This is a reasonable tradeoff, but not an especially compelling one for this configuration. The server logs also showed that Qwen3.5 uses additional recurrent state, alongside its KV cache. My conclusion was to retain FP16 for Qwen3.5, unless a particular application requires additional VRAM headroom. Qwen3 showed a much larger benefit. ``` Qwen3-4B KV cache Speed GPU memory---------------------------------FP16 41.59 tok/s 3163 MiBQ8_0 39.92 tok/s 2903 MiBQ4_0 39.80 tok/s 2759 MiB ``` The tradeoffs were striking. Q8_0 saved 260 MiB, with a generation-speed reduction of approximately 4%. Q4_0 saved 404 MiB, with a roughly 4.3% reduction. On a GPU with only 4GB of VRAM, hundreds of megabytes are meaningful. Q8_0 became my preferred balance: nearly all the measured performance of FP16, but with substantially more free GPU memory. There is a caveat: this experiment measured timing, memory, and successful completion. It did not formally evaluate whether KV quantization changed semantic accuracy or reasoning quality. That would require a separate task-quality benchmark. While benchmarking, I encountered a practical issue familiar to anyone using reasoning-capable LLMs. Some models can spend a large number of tokens on internal reasoning before producing visible output. For complex mathematical reasoning or difficult planning tasks, this can be useful. But for basic coding questions, translations, summaries, and direct explanations, the additional delay may be undesirable. The tested llama.cpp build supports these options: ``` --reasoning off --reasoning-budget 0 ``` I used them to configure the models for direct responses. In practice, this made short interactive exchanges more usable and helped avoid cases where the output-token allowance was exhausted before the visible answer was complete. However, there is an important distinction. Disabling thinking does not necessarily make the GPU generate individual tokens faster. Instead, it can reduce the total number of generated tokens required before the user receives an answer. The 2.24× and 2.07× GPU-offloading gains reported earlier should not be attributed to reasoning settings. I also would not recommend disabling reasoning for every possible task. For problems where deliberate multi-step reasoning matters, a reasoning-enabled configuration may be the better choice. The best setting depends on the purpose of the model. Combining the results of both experiments, I selected the following practical configurations. Recommended llama.cpp profiles ``` GEMMA 4 E2B GPU layers: auto CPU threads: 4 Context: 4096 Batch: 512 Microbatch: 128 KV cache: f16 / f16 Reasoning: off QWEN3.5-4B GPU layers: all CPU threads: 6 Context: 4096 Batch: 512 Microbatch: 256 KV cache: f16 / f16 Reasoning: off QWEN3-4B GPU layers: all CPU threads: 6 Context: 4096 Batch: 512 Microbatch: 256 KV cache: q8_0 / q8_0 Reasoning: off ``` These should be viewed as tested profiles for my hardware, not universal defaults. Different GPUs, driver versions, model quantizations, context sizes, and concurrency settings can produce different outcomes. Here is how to reproduce the configuration on Windows. I used the following directory structure: ``` C:\llama.cpp\ bin\ llama-cli.exe llama-server.exe models\ google_gemma-4-E2B-it-Q4_K_M.gguf Qwen3.5-4B-Q4_K_M.gguf Qwen3-4B-Q4_K_M.gguf ``` The executable names and directory layout may differ depending on how you install llama.cpp. First, open PowerShell and verify that Windows detects the NVIDIA GPU: ``` nvidia-smi ``` Then check your llama.cpp installation: ``` cd C:\llama.cpp.\bin\llama-cli.exe --help ``` Review the available flags before copying any configuration. One practical challenge I encountered was CUDA toolchain compatibility. A newer prebuilt llama.cpp binary failed with an unsupported PTX toolchain error on my machine, while build b8729 worked correctly. The important point is not that everyone should use b8729. It is that a binary and its packaged CUDA components must be compatible with the installed NVIDIA driver. ``` .\bin\llama-cli.exe ` -m ".\models\google_gemma-4-E2B-it-Q4_K_M.gguf" ` --gpu-layers auto ` -t 4 ` -c 4096 ` -b 512 ` -ub 128 ` --cache-type-k f16 ` --cache-type-v f16 ` --reasoning off ` --reasoning-budget 0 ``` This profile produced approximately 59 tokens per second in the validation experiment. ``` .\bin\llama-cli.exe ` -m ".\models\Qwen3.5-4B-Q4_K_M.gguf" ` --gpu-layers all ` -t 6 ` -c 4096 ` -b 512 ` -ub 256 ` --cache-type-k f16 ` --cache-type-v f16 ` --reasoning off ` --reasoning-budget 0 ``` With maximum GPU layer offloading requested, Qwen3.5 achieved about 34.6 generated tokens per second in the longer-response validation experiment. ``` .\bin\llama-cli.exe ` -m ".\models\Qwen3-4B-Q4_K_M.gguf" ` --gpu-layers all ` -t 6 ` -c 4096 ` -b 512 ` -ub 256 ` --cache-type-k q8_0 ` --cache-type-v q8_0 ` --reasoning off ` --reasoning-budget 0 ``` This configuration achieved approximately 39.9 generated tokens per second while using roughly 2,903 MiB of GPU memory. For the highest measured Qwen3 throughput, FP16 KV cache remains an alternative, at the cost of more VRAM. For terminal interactions, llama-cli is convenient. But if you want to integrate local inference with your own software, llama-server offers an OpenAI-compatible API. For example, here is the optimized Qwen3 server configuration: ``` .\bin\llama-server.exe ` -m ".\models\Qwen3-4B-Q4_K_M.gguf" ` --host 127.0.0.1 ` --port 8080 ` --gpu-layers all ` -t 6 ` -c 4096 ` -b 512 ` -ub 256 ` --cache-type-k q8_0 ` --cache-type-v q8_0 ` --reasoning off ` --reasoning-budget 0 ` --no-webui ``` Once started, the API base URL is: ``` http://127.0.0.1:8080/v1 ``` To check that the server is responding: ``` Invoke-RestMethod ` http://127.0.0.1:8080/v1/models ``` To send a chat completion request: ``` php $body = @{ model = "Qwen3-4B-Q4_K_M.gguf" messages = @( @{ role = "user" content = "Explain AES-256 simply." } ) temperature = 0 max_tokens = 256} | ConvertTo-Json -Depth 5 php $result = Invoke-RestMethod ` -Uri "http://127.0.0.1:8080/v1/chat/completions" ` -Method Post ` -ContentType "application/json" ` -Body $body $result.choices[0].message.content ``` If your server reports a different model identifier from /v1/models, use that identifier instead. To use the built-in browser interface, omit --no-webui on builds that include the web interface. This example binds the server to 127.0.0.1, restricting access to the local machine. Do not expose an unauthenticated inference server directly to the internet. If you have a GTX 1650, GTX 1660, RTX 3050 4GB, or another memory-constrained NVIDIA GPU, I would recommend the following approach. Start with a correctness baseline. Verify that the model loads, completes prompts, and produces sensible answers before measuring performance. Test GPU offloading independently. Compare automatic layer selection with explicitly requested layer counts. Do not assume maximum offloading is always fastest or always feasible. Tune CPU threads and microbatch size. Higher numbers may increase memory pressure without improving single-user token generation. Inspect memory allocation. Separate model weights, KV cache, compute buffers, and other runtime allocations wherever logs permit. Evaluate cache quantization. Especially on smaller GPUs, the savings from Q8_0 or Q4_0 can be worth considering. Measure complete answers. Report token generation speed and total response latency separately. A truncated response should not be counted as a fully completed task. Repeat measurements. A configuration should be selected based on reproducible observations, not one unusually fast run. Most importantly, record the exact model file, quantization, inference engine version, driver, and GPU for every benchmark. Without that context, performance comparisons become difficult to reproduce. There are limits to what can be concluded from this study. All measurements came from one Windows 10 laptop with an NVIDIA GTX 1650 Ti and a single llama.cpp build. Other GPUs and software versions may behave differently. The three tested models also have different architectures. A higher tokens-per-second figure does not establish superior model quality. The experiment focused on one inference request at a time, not concurrent multi-user serving. The validation workload contained two technical prompt types and four measured responses per configuration, providing useful repeatability checks but not a large-scale statistical characterization. And although the quantized KV configurations completed requests successfully, we did not formally benchmark answer quality under different cache precisions. Finally, disabling thinking may improve responsiveness for simple tasks while weakening performance on difficult reasoning problems. These results should therefore be interpreted as measurements of local inference performance, not general model-quality rankings. Before running these experiments, I expected a 4GB laptop GPU to impose severe limits on local LLM inference. Those limits are real. But the performance available within them turned out to be more useful than I initially expected. With carefully chosen configurations, I measured: ``` Gemma 4 E2B ~59 tok/s GPU auto, FP16 KV Qwen3.5-4B ~35 tok/s GPU all, FP16 KV Qwen3-4B ~42 tok/s GPU all, FP16 KV Qwen3-4B (memory-balanced) ~40 tok/s GPU all, Q8 KV 260 MiB less GPU memory than FP16 ``` The biggest performance improvement came from requesting more GPU layer offloading for models that could benefit from it. KV cache quantization offered a second optimization opportunity: recovering GPU memory with a relatively small reduction in throughput. And disabling unnecessary reasoning helped keep interactive conversations focused on direct answers. None of these adjustments required purchasing additional hardware. The broader takeaway is straightforward: Before buying a faster GPU, measure whether your inference engine is making effective use of the one you already own. On hardware with only 4GB of VRAM, the right settings can make the difference between a sluggish local model and a genuinely usable one. Experimental setup: NVIDIA GeForce GTX 1650 Ti Laptop GPU (4GB), Intel Core i5–10300H, Windows 10, llama.cpp build b8729, October 2026. Performance measurements are rounded medians from the two experimental phases. [How I Made Local LLMs Up to 2.24× Faster on a 4GB NVIDIA GPU](https://pub.towardsai.net/how-i-made-local-llms-up-to-2-24-faster-on-a-4gb-nvidia-gpu-e0c02f2b1821) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.