Qwen 27B with 128k Context on 24GB VRAM?

Users report running Qwen 27B with a 128k context window on a single 24GB VRAM card by setting Ollama environment variables OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q4_0, which quantizes the KV cache to q4_0 and reduces memory usage to 83% at 128k context. Without these flags, the KV cache alone would exceed 24GB. Performance reaches about 30 tok/s, dropping to 4 tok/s if spilling to system RAM.

Qwen 27B with 128k Context on 24GB VRAM? Running a 27B model with a 128k context window on a single 24GB card sounds like a recipe for an OOM Out of Memory crash, but I'm seeing reports that it's actually working. The key seems to be a specific combo of Ollama environment variables that drastically cuts down the memory footprint of the KV cache. It's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags. If you're trying to squeeze long-context LLM agents into limited VRAM, you need to look at these two flags: OLLAMA FLASH ATTENTION=1 OLLAMA KV CACHE TYPE=q4 0 The q4 0 KV cache quantization is the real heavy lifter here. Instead of storing the cache in FP16, it compresses it, which is why someone is reporting 83% VRAM usage at 128k context on a 24GB card. Without this, the KV cache for a 27B model at that context length would easily blow past 24GB on its own, regardless of the model quantization. The performance hit is surprisingly minimal too—hitting around 30 tok/s. If it spills over into system RAM, you're looking at a drop to maybe 4 tok/s, which makes the local deployment basically unusable for real-time work. A few observations from this setup: Q4 Quantization: 128k context fits comfortably. Q5 Quantization: Hits a wall around 96k context. It's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags. Next DeepSeek-R1 Local Deployment: My Hardware Struggles → /en/threads/3346/ All Replies （4） Q Flash attention 2 makes a huge difference here, though it's a pain to compile sometimes. 0 N KV cache quantization helps, but I still hit a wall around 64k on my 3090. 0 T Does the perplexity tank as you get closer to that limit, or does it hold up? 0 J That's the real question. I've noticed some "lost in the middle" issues with other models, wonder if Qwen does the same. 0