{"slug": "qwen-27b-with-128k-context-on-24gb-vram", "title": "Qwen 27B with 128k Context on 24GB VRAM?", "summary": "Users report running Qwen 27B with a 128k context window on a single 24GB VRAM card by setting Ollama environment variables OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q4_0, which quantizes the KV cache to q4_0 and reduces memory usage to 83% at 128k context. Without these flags, the KV cache alone would exceed 24GB. Performance reaches about 30 tok/s, dropping to 4 tok/s if spilling to system RAM.", "body_md": "# Qwen 27B with 128k Context on 24GB VRAM?\n\nRunning a 27B model with a 128k context window on a single 24GB card sounds like a recipe for an OOM (Out of Memory) crash, but I'm seeing reports that it's actually working. The key seems to be a specific combo of Ollama environment variables that drastically cuts down the memory footprint of the KV cache.\n\nIt's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags.\n\nIf you're trying to squeeze long-context LLM agents into limited VRAM, you need to look at these two flags:\n\n```\nOLLAMA_FLASH_ATTENTION=1\nOLLAMA_KV_CACHE_TYPE=q4_0\n```\n\nThe `q4_0`\n\nKV cache quantization is the real heavy lifter here. Instead of storing the cache in FP16, it compresses it, which is why someone is reporting 83% VRAM usage at 128k context on a 24GB card. Without this, the KV cache for a 27B model at that context length would easily blow past 24GB on its own, regardless of the model quantization.\n\nThe performance hit is surprisingly minimal too—hitting around 30 tok/s. If it spills over into system RAM, you're looking at a drop to maybe 4 tok/s, which makes the local deployment basically unusable for real-time work.\n\nA few observations from this setup:\n\n**Q4 Quantization:** 128k context fits comfortably.**Q5 Quantization:** Hits a wall around 96k context.\n\nIt's a decent deep dive into how much we can optimize the AI workflow by tweaking the backend rather than just hoping for a smaller model. If you've been capping your context at 32k because of VRAM limits, definitely test these flags.\n\n[Next DeepSeek-R1 Local Deployment: My Hardware Struggles →](/en/threads/3346/)\n\n## All Replies （4）\n\nQ\n\nFlash attention 2 makes a huge difference here, though it's a pain to compile sometimes.\n\n0\n\nN\n\nKV cache quantization helps, but I still hit a wall around 64k on my 3090.\n\n0\n\nT\n\nDoes the perplexity tank as you get closer to that limit, or does it hold up?\n\n0\n\nJ\n\nThat's the real question. I've noticed some \"lost in the middle\" issues with other models, wonder if Qwen does the same.\n\n0", "url": "https://wpnews.pro/news/qwen-27b-with-128k-context-on-24gb-vram", "canonical_source": "https://promptcube3.com/en/threads/3354/", "published_at": "2026-07-25 20:01:48+00:00", "updated_at": "2026-07-25 20:35:56.892211+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure"], "entities": ["Qwen", "Ollama", "Qwen 27B"], "alternates": {"html": "https://wpnews.pro/news/qwen-27b-with-128k-context-on-24gb-vram", "markdown": "https://wpnews.pro/news/qwen-27b-with-128k-context-on-24gb-vram.md", "text": "https://wpnews.pro/news/qwen-27b-with-128k-context-on-24gb-vram.txt", "jsonld": "https://wpnews.pro/news/qwen-27b-with-128k-context-on-24gb-vram.jsonld"}}