By Nokka | September 11, 2026
This article was written by AI (deepseek-v4.1-flash) through Hermes Agent, reviewed and edited by Nokka.
People install Ollama and hit the same wall: responses are slower than expected even on good hardware.
Most of the time the hardware is not the problem. The defaults are simply not tuned for sustained heavy use [1].
Flash Attention speeds up the attention computation inside the model by keeping data in GPU cache instead of main memory [2].
A common misconception is that this is off by default. In practice Ollama enables it automatically when the system supports it [3].
The environment variable exists to force it on, for cases where auto-detection misses or you want a guarantee [2][3].
The payoff is both speed and quality on long-context work, because the technique manages memory more efficiently.
KV cache type controls how the model stores the key/value state of the computation [1].
The important condition: compression only works when Flash Attention is already on. Without it, this option does nothing [3].
Compressing to one of the quantized formats can roughly double the context length you can hold before memory runs out, because the data takes less space.
The trade is a small accuracy loss. On ordinary tasks you will not feel it. On precision work, benchmark before committing.
The default context length is usually low to save memory. If you work with long documents or big code files, it will not be enough [4].
Raising it lets the model remember longer conversations, but memory use climbs sharply.
A safe practice is to start at the size your actual work needs. Do not max it out. An oversized context makes every single response slower.
By default the system decides which model layers run on the GPU [1].
If VRAM is limited but you still want a larger model, setting the layer count explicitly lets you use the hardware more fully. This is split inference: some layers on GPU, some on CPU. Slower, but it runs.
Every first call to Ollama has to load the model into memory, which takes anywhere from a few seconds to tens of seconds [3].
Pre with an empty prompt keeps the model resident, so your first request is as fast as the rest.
This suits machines with plenty of spare memory. If memory is tight, keeping multiple models resident slows everything down.
| Setting | Default | Set it to | Trade-off |
|---|---|---|---|
| Flash Attention | Auto when supported | Force on | Almost none |
| KV cache type | f16 | q8_0 (needs FA on) | Minor accuracy |
| num_ctx | Low | Match your workload | Memory |
| num_gpu | Automatic | Explicit when needed | Configuration complexity |
| Preload | No | As needed | Reserved memory |
One Flash Attention behaves differently across GPUs. Some cards reportedly get slower with it on [1]. Measure before and after on your own workload.
Two KV cache compression reduces accuracy on work that needs precision, such as numerical analysis or reading legal documents.
Three The best values differ per machine. Copying someone else's settings without measuring your own gives unpredictable results.
Four Advertised speed numbers are usually measured on an idle machine. If you have other software running, expect less.
I run Ollama locally and the clearest lesson is that measurement has to be systematic.
Early on I changed several settings at once, saw an improvement, and had no idea which one helped. Changing one at a time with notes gave answers I could actually reuse.
The other thing I found is that Flash Attention is the least risky win for anyone on a recent NVIDIA card, because the technique was designed to manage memory directly.
My advice is to start with the two safest changes, Flash Attention on and context length matched to your work, then measure. Only reach for the settings that trade accuracy if that is still not enough.
[1] Khatik, K., "Optimizing Ollama Performance on Windows: Hardware, Quantization, Parallelism & More", Medium (2026), https://medium.com/@kapildevkhatik2/optimizing-ollama-performance-on-windows-hardware-quantization-parallelism-more-fac04802288e
[2] Broadcom TechDocs, "Understanding the Ollama provider" (2026), https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-0/ai/explanation-understanding-ollama-configuration.html
[3] Ollama, "FAQ" (accessed Sep 11, 2026), https://docs.ollama.com/faq [4] Easton Dev, "Ollama Performance Tuning: Batching, KV Cache, and OOM" (2026), https://eastondev.com/blog/en/posts/ai/20260410-ollama-performance-optimization/