Running a 27B model with a 128k context window on a single 24GB card sounds like a recipe for an OOM (Out of Memory) crash, but I'm seeing reports that it's actually working. The key seems to be a specific combo of Ollama environment variables that drastically cuts down the memory footprint of the KV cache.
It's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags.
If you're trying to squeeze long-context LLM agents into limited VRAM, you need to look at these two flags:
OLLAMA_FLASH_ATTENTION=1
OLLAMA_KV_CACHE_TYPE=q4_0
The q4_0
KV cache quantization is the real heavy lifter here. Instead of storing the cache in FP16, it compresses it, which is why someone is reporting 83% VRAM usage at 128k context on a 24GB card. Without this, the KV cache for a 27B model at that context length would easily blow past 24GB on its own, regardless of the model quantization.
The performance hit is surprisingly minimal too—hitting around 30 tok/s. If it spills over into system RAM, you're looking at a drop to maybe 4 tok/s, which makes the local deployment basically unusable for real-time work.
A few observations from this setup:
Q4 Quantization: 128k context fits comfortably.Q5 Quantization: Hits a wall around 96k context.
It's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags.
Next DeepSeek-R1 Local Deployment: My Hardware Struggles →
All Replies （4） #
Q
Flash attention 2 makes a huge difference here, though it's a pain to compile sometimes.
0
N
KV cache quantization helps, but I still hit a wall around 64k on my 3090.
0
T
Does the perplexity tank as you get closer to that limit, or does it hold up?
0
J
That's the real question. I've noticed some "lost in the middle" issues with other models, wonder if Qwen does the same.
0