cd /news/ai-infrastructure/the-gpu-vram-calculus-why-your-70b-m… · home › topics › ai-infrastructure › article
[ARTICLE · art-173263] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

The GPU VRAM Calculus: Why your 70B model crashes on 48GB VRAM (and the exact production sizing formula)

A developer published a production VRAM sizing formula showing that a 4-bit quantized 70B model (~38.5–40GB of weights) overflows 48GB GPUs once KV cache, activations and CUDA runtime overhead are counted, with a batch of 8 requests at 8k context adding over 13.4GB for the KV cache alone. The writeup recommends capping vLLM's --gpu-memory-utilization at 0.90 and enabling --enable-prefix-caching to share system-prompt blocks across concurrent users, and accompanies an open-sourced repository of reference architectures including docker-compose files for vLLM and DeepSeek R1 and pgvector retrieval pipelines.

recommendation

Budget KV cache and runtime overhead on top of weight size before choosing GPUs, cap vLLM at --gpu-memory-utilization 0.90, and enable --enable-prefix-caching to share repeated system prompts across concurrent requests.

more in playbook: Inference and self-hosting
by read2 min views6 publishedNov 20, 2026

A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk.

If a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: "It will fit comfortably on dual RTX 3090s or an A6000 (48GB physical VRAM)."

Then they spin up a vLLM or HuggingFace TGI container, fire a modest batch of 8 concurrent user requests with an 8k context window, and immediately hit the dreaded:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.84 GiB...

Here is the exact mathematical reality of where your physical GPU VRAM actually goes, and the production sizing formula to keep your servers rock-solid.

In production serving, your total required memory is never just the weights:

$$\text{VRAM}{\text{Total}} = M{\text{weights}} + M_{\text{kv_cache}} + M_{\text{activations}} + M_{\text{cuda_runtime}}$$

The memory footprint of the model parameters loaded into memory:

This is the memory required to store the Key and Value attention tensors for past tokens across active requests.

$$\text{Memory}{\text{KV}} = 2 \times L \times H{\text{kv}} \times D_{\text{head}} \times T_{\text{ctx}} \times B \times P_{\text{bytes}}$$

Where:

Result: In naive PyTorch, a batch of 8 requests with 8k context requires over 13.4 GB of additional VRAM just for the KV cache!

Transient memory required during the forward pass tensor operations. Scales directly with batch size and context length. For 70B models, budget 2GB to 4GB.

The CUDA driver, PyTorch runtime context, and cuBLAS workspace grab 0.8GB to 1.5GB the instant the process initializes.

--gpu-memory-utilization 0.90 (leaving 10% strictly for dynamic spikes).--enable-prefix-caching so repetitive system prompts (e.g., 2,000 token system instructions) share the exact same physical memory blocks across concurrent users. I’ve open-sourced our complete production reference architectures, including:

docker-compose.yml for vLLM & DeepSeek R1pgvector knowledge retrieval pipelines ⭐ Check out the GitHub Repository:

github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026

If you are an engineering team or indie developer taking open-weight models and autonomous agents into production, check out our complete 21-page production engineering blueprint:

Happy to answer multi-GPU tensor parallelism and quantization questions in the comments!

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @vllm 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/the-gpu-vram-calculu…] indexed:0 read:2min 2026-11-20 · —