A lot of developers and machine learning teams calculate model memory by simply looking at the static weight size on disk.
If a 4-bit quantized 70B parameter model is ~38.5GB to 40GB, the natural assumption is: "It will fit comfortably on dual RTX 3090s or an A6000 (48GB physical VRAM)."
Then they spin up a vLLM or HuggingFace TGI container, fire a modest batch of 8 concurrent user requests with an 8k context window, and immediately hit the dreaded:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.84 GiB...
Here is the exact mathematical reality of where your physical GPU VRAM actually goes, and the production sizing formula to keep your servers rock-solid.
In production serving, your total required memory is never just the weights:
$$\text{VRAM}{\text{Total}} = M{\text{weights}} + M_{\text{kv_cache}} + M_{\text{activations}} + M_{\text{cuda_runtime}}$$
The memory footprint of the model parameters loaded into memory:
This is the memory required to store the Key and Value attention tensors for past tokens across active requests.
$$\text{Memory}{\text{KV}} = 2 \times L \times H{\text{kv}} \times D_{\text{head}} \times T_{\text{ctx}} \times B \times P_{\text{bytes}}$$
Where:
Result: In naive PyTorch, a batch of 8 requests with 8k context requires over 13.4 GB of additional VRAM just for the KV cache!
Transient memory required during the forward pass tensor operations. Scales directly with batch size and context length. For 70B models, budget 2GB to 4GB.
The CUDA driver, PyTorch runtime context, and cuBLAS workspace grab 0.8GB to 1.5GB the instant the process initializes.
--gpu-memory-utilization 0.90 (leaving 10% strictly for dynamic spikes).--enable-prefix-caching so repetitive system prompts (e.g., 2,000 token system instructions) share the exact same physical memory blocks across concurrent users.
I’ve open-sourced our complete production reference architectures, including:
docker-compose.yml for vLLM & DeepSeek R1pgvector knowledge retrieval pipelines
⭐ Check out the GitHub Repository:
github.com/jppeter621/awesome-local-ai-and-mcp-architectures-2026
If you are an engineering team or indie developer taking open-weight models and autonomous agents into production, check out our complete 21-page production engineering blueprint:
Happy to answer multi-GPU tensor parallelism and quantization questions in the comments!