{"slug": "the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact", "title": "The GPU VRAM Calculus: Why your 70B model crashes on 48GB VRAM (and the exact production sizing formula)", "summary": "A developer published a production VRAM sizing formula showing that a 4-bit quantized 70B model (~38.5–40GB of weights) overflows 48GB GPUs once KV cache, activations and CUDA runtime overhead are counted, with a batch of 8 requests at 8k context adding over 13.4GB for the KV cache alone. The writeup recommends capping vLLM's --gpu-memory-utilization at 0.90 and enabling --enable-prefix-caching to share system-prompt blocks across concurrent users, and accompanies an open-sourced repository of reference architectures including docker-compose files for vLLM and DeepSeek R1 and pgvector retrieval pipelines.", "body_md": "A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk.\n\nIf a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: *\"It will fit comfortably on dual RTX 3090s or an A6000 (48GB physical VRAM).\"*\n\nThen they spin up a vLLM or HuggingFace TGI container, fire a modest batch of 8 concurrent user requests with an 8k context window, and immediately hit the dreaded:\n\n```\ntorch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.84 GiB...\n```\n\nHere is the exact mathematical reality of where your physical GPU VRAM actually goes, and the production sizing formula to keep your servers rock-solid.\n\nIn production serving, your total required memory is never just the weights:\n\n$$\\text{VRAM}*{\\text{Total}} = M*{\\text{weights}} + M_{\\text{kv_cache}} + M_{\\text{activations}} + M_{\\text{cuda_runtime}}$$\n\nThe memory footprint of the model parameters loaded into memory:\n\nThis is the memory required to store the Key and Value attention tensors for past tokens across active requests.\n\n$$\\text{Memory}*{\\text{KV}} = 2 \\times L \\times H*{\\text{kv}} \\times D_{\\text{head}} \\times T_{\\text{ctx}} \\times B \\times P_{\\text{bytes}}$$\n\nWhere:\n\n**Result:** In naive PyTorch, a batch of 8 requests with 8k context requires **over 13.4 GB of additional VRAM just for the KV cache!**\n\nTransient memory required during the forward pass tensor operations. Scales directly with batch size and context length. For 70B models, budget **2GB to 4GB**.\n\nThe CUDA driver, PyTorch runtime context, and cuBLAS workspace grab **0.8GB to 1.5GB** the instant the process initializes.\n\n`--gpu-memory-utilization 0.90` (leaving 10% strictly for dynamic spikes).`--enable-prefix-caching` so repetitive system prompts (e.g., 2,000 token system instructions) share the exact same physical memory blocks across concurrent users.\nI’ve open-sourced our complete production reference architectures, including:\n\n`docker-compose.yml` for vLLM & DeepSeek R1`pgvector` knowledge retrieval pipelines\n⭐ **Check out the GitHub Repository:**\n\n[github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026](https://github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026)\n\nIf you are an engineering team or indie developer taking open-weight models and autonomous agents into production, check out our complete 21-page production engineering blueprint:\n\nHappy to answer multi-GPU tensor parallelism and quantization questions in the comments!", "url": "https://wpnews.pro/news/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact", "canonical_source": "https://dev.to/john_peter_69de840350463a/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact-production-sizing-9g9", "published_at": "2026-11-20 14:10:23+00:00", "updated_at": "2026-11-20 14:14:11.666073+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-tools"], "entities": ["vLLM", "HuggingFace TGI", "DeepSeek R1", "pgvector", "PyTorch", "NVIDIA RTX 3090", "NVIDIA A6000"], "also_reported_by": [], "takeaway": "Size serving VRAM as weights plus KV cache plus 2–4GB activations plus 0.8–1.5GB CUDA runtime; a 4-bit 70B model's ~40GB of weights leaves no room on 48GB cards for 8 concurrent 8k-context requests, which need over 13.4GB of KV cache alone.", "recommendation": {"advice": "Budget KV cache and runtime overhead on top of weight size before choosing GPUs, cap vLLM at --gpu-memory-utilization 0.90, and enable --enable-prefix-caching to share repeated system prompts across concurrent requests.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact", "markdown": "https://wpnews.pro/news/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact.md", "text": "https://wpnews.pro/news/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact.txt", "jsonld": "https://wpnews.pro/news/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact.jsonld"}}