{"slug": "self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory", "title": "Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory", "summary": "An engineer details the GPU memory requirements for self-hosting large language models, explaining that the KV cache, not model weights, is the primary bottleneck under real traffic. The post provides formulas and examples showing how context length and batch size can consume more memory than the model itself, and recommends using paged-attention serving frameworks like vLLM to manage memory efficiently.", "body_md": "Here is the short version: the model weights are the *smallest* GPU-memory surprise you'll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most \"run an LLM on your GPU\" tutorials load the weights, run one short prompt, and declare victory. Then you send a 6,000-token document at a batch of eight and it OOMs.\n\nI've set this up enough times to know the failure isn't random. It's arithmetic you can do before you rent the GPU. This post is that arithmetic.\n\nGPU memory for inference is four separate buckets, and tutorials only mention the first:\n\n`context_length × batch_size`\n\n. This is the one that bites.The weights number is the one everyone quotes because it's easy. Parameter count × bytes-per-parameter:\n\n| Precision | Bytes/param | 7B model | 13B model | 70B model |\n|---|---|---|---|---|\n| FP16/BF16 | 2 | ~14 GB | ~26 GB | ~140 GB |\n| INT8 | 1 | ~7 GB | ~13 GB | ~70 GB |\n| INT4 (GPTQ/AWQ/GGUF Q4) | ~0.5 | ~4 GB | ~7 GB | ~38 GB |\n\nSo a 7B model in 4-bit \"fits\" on a 24GB card with 20GB to spare, and the tutorial ends there. That leftover 20GB is not spare — it's your working budget for everything in buckets 2 through 4, and it disappears faster than you'd guess.\n\n**Takeaway:** Weights tell you if the model loads; they tell you nothing about whether it serves traffic.\n\nThe KV cache stores the key and value tensors for every token already in the context so the model doesn't recompute them each step. Its size, per request, is roughly:\n\n```\nkv_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element\n```\n\nThe `2`\n\nis for keys and values. Note `num_kv_heads`\n\n, not the full attention-head count — modern models use grouped-query attention (GQA), which is the single biggest reason KV cache is smaller than older formulas suggest. Worth checking your model's config, because it swings the number by 4-8×.\n\nWorked example, Llama-3-8B-class architecture (32 layers, 8 KV heads, head_dim 128, FP16):\n\n```\nper_token = 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB/token\n```\n\nThat's about 1 GB for a single 8,192-token request. Now the part tutorials skip: this is **per concurrent request**. Serve a batch of 16 at that context length and you've spent ~16 GB on KV cache alone — more than the weights. On a 24GB card holding a 4-bit 8B model (~5 GB with overhead), you have maybe 17-18GB left, and you just watched a modest batch eat all of it.\n\nTwo levers shrink this:\n\n`max_model_len`\n\n.**Takeaway:** Budget KV cache as `per_token_KB × max_context × expected_concurrency`\n\nbefore you pick a card — it often dwarfs the weights.\n\nEven before a single token, loading CUDA and your inference framework claims memory. The CUDA context alone is typically a few hundred MB to over a gigabyte depending on driver and GPU. PyTorch's caching allocator reserves more. Frameworks like vLLM deliberately grab a large fraction of remaining VRAM up front (controlled by `gpu_memory_utilization`\n\n, default 0.9) to manage the KV cache themselves — which is great for throughput but means \"nvidia-smi shows 90% used\" is expected, not a leak.\n\nThen there's **fragmentation**. Naive allocators hand out one contiguous block per request's KV cache. When requests of varying lengths come and go, you get Swiss-cheese memory: 4GB free, but no single 2GB hole. This is exactly the problem vLLM's PagedAttention solves — it pages the KV cache like an OS pages RAM, so non-contiguous free memory is usable. If you're comparing serving frameworks, this is the practical reason vLLM sustains higher concurrency than a plain `transformers`\n\nloop on the same card.\n\n**Takeaway:** Reserve 1-2GB for runtime overhead as a floor, and prefer a paged-attention serving stack the moment you have concurrent requests.\n\nThese are the four I actually reach for, with honest limitations:\n\n| Tool | Best for | Real drawback |\n|---|---|---|\nOllama |\nLocal dev, single user, \"just run it\" | Not built for high-concurrency serving; batching is limited |\nllama.cpp |\nCPU/GPU hybrid, low VRAM, edge boxes | GGUF quant setup is fiddly; peak throughput trails GPU-native servers |\nvLLM |\nProduction serving, high concurrency | Heavier setup; needs a proper CUDA GPU; startup VRAM grab surprises people |\nTGI (Text Generation Inference) |\nProduction serving with HF ecosystem | Tighter model-support window; also GPU-hungry at start |\n\nFor a first self-host on a single 24GB consumer card (RTX 3090/4090), a 7-8B model quantized to 4-bit via **Ollama** or **llama.cpp** is the reliable starting point. When you move past one user, switch to **vLLM** and set `max_model_len`\n\nand `gpu_memory_utilization`\n\ndeliberately rather than accepting defaults.\n\n**Takeaway:** Match the framework to concurrency, not to model size — the model fits on the card either way; only one of them survives real traffic.\n\nRather than guess, do this arithmetic with your actual model config (`config.json`\n\nhas `num_hidden_layers`\n\n, `num_key_value_heads`\n\n, `head_dim`\n\nor `hidden_size / num_attention_heads`\n\n):\n\n``` python\ndef vram_estimate_gb(params_b, bytes_per_param,\n                     num_layers, num_kv_heads, head_dim,\n                     max_ctx, concurrency,\n                     kv_bytes=2, overhead_gb=2.0):\n    weights = params_b * 1e9 * bytes_per_param / 1e9\n    per_tok = 2 * num_layers * num_kv_heads * head_dim * kv_bytes\n    kv = per_tok * max_ctx * concurrency / 1e9\n    return round(weights + kv + overhead_gb, 1)\n\n# Llama-3-8B, 4-bit weights, FP16 KV, 8k context, 8 concurrent\nprint(vram_estimate_gb(8, 0.5, 32, 8, 128, 8192, 8))  # ~14.5 GB\n```\n\nBump `concurrency`\n\nto 24 and that same setup crosses 30GB — past a 24GB card. This ten-line function has saved me more grief than any benchmark, because it turns \"will it work?\" into a number you check before spending money on a bigger GPU or a cloud instance. Treat it as an estimate with ±15% slack for allocator behavior, not a guarantee.\n\nIf you're self-hosting your first LLM, size for the KV cache and overhead, not just the weights — the weights are the part that always fits. For a single user on one consumer GPU, a 4-bit 7-8B model under Ollama or llama.cpp is the safe first step. The moment you have concurrent requests, move to vLLM or TGI, cap `max_model_len`\n\nto what you actually need, and consider FP8 KV cache. And before you upgrade to a pricier card because you hit OOM, run the arithmetic above — nine times out of ten the fix is a smaller context ceiling or KV quantization, not more VRAM.", "url": "https://wpnews.pro/news/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory", "canonical_source": "https://dev.to/libme/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory-50pc", "published_at": "2026-08-10 00:45:24+00:00", "updated_at": "2026-08-10 01:16:25.119963+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops"], "entities": ["vLLM", "PyTorch", "CUDA", "Llama-3-8B", "GPTQ", "AWQ", "GGUF"], "alternates": {"html": "https://wpnews.pro/news/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory", "markdown": "https://wpnews.pro/news/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory.md", "text": "https://wpnews.pro/news/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory.txt", "jsonld": "https://wpnews.pro/news/self-hosting-your-first-llm-what-the-tutorials-skip-about-gpu-memory.jsonld"}}