# The GPU VRAM Calculus: Why your 70B model crashes on 48GB VRAM (and the exact production sizing formula)

> Source: <https://dev.to/john_peter_69de840350463a/the-gpu-vram-calculus-why-your-70b-model-crashes-on-48gb-vram-and-the-exact-production-sizing-9g9>
> Published: 2026-11-20 14:10:23+00:00

**Recommendation:** Budget KV cache and runtime overhead on top of weight size before choosing GPUs, cap vLLM at --gpu-memory-utilization 0.90, and enable --enable-prefix-caching to share repeated system prompts across concurrent requests.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk.

If a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: *"It will fit comfortably on dual RTX 3090s or an A6000 (48GB physical VRAM)."*

Then they spin up a vLLM or HuggingFace TGI container, fire a modest batch of 8 concurrent user requests with an 8k context window, and immediately hit the dreaded:

```
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.84 GiB...
```

Here is the exact mathematical reality of where your physical GPU VRAM actually goes, and the production sizing formula to keep your servers rock-solid.

In production serving, your total required memory is never just the weights:

$$\text{VRAM}*{\text{Total}} = M*{\text{weights}} + M_{\text{kv_cache}} + M_{\text{activations}} + M_{\text{cuda_runtime}}$$

The memory footprint of the model parameters loaded into memory:

This is the memory required to store the Key and Value attention tensors for past tokens across active requests.

$$\text{Memory}*{\text{KV}} = 2 \times L \times H*{\text{kv}} \times D_{\text{head}} \times T_{\text{ctx}} \times B \times P_{\text{bytes}}$$

Where:

**Result:** In naive PyTorch, a batch of 8 requests with 8k context requires **over 13.4 GB of additional VRAM just for the KV cache!**

Transient memory required during the forward pass tensor operations. Scales directly with batch size and context length. For 70B models, budget **2GB to 4GB**.

The CUDA driver, PyTorch runtime context, and cuBLAS workspace grab **0.8GB to 1.5GB** the instant the process initializes.

`--gpu-memory-utilization 0.90` (leaving 10% strictly for dynamic spikes).`--enable-prefix-caching` so repetitive system prompts (e.g., 2,000 token system instructions) share the exact same physical memory blocks across concurrent users.
I’ve open-sourced our complete production reference architectures, including:

`docker-compose.yml` for vLLM & DeepSeek R1`pgvector` knowledge retrieval pipelines
⭐ **Check out the GitHub Repository:**

[github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026](https://github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026)

If you are an engineering team or indie developer taking open-weight models and autonomous agents into production, check out our complete 21-page production engineering blueprint:

Happy to answer multi-GPU tensor parallelism and quantization questions in the comments!
