Self-Hosting Your First LLM: What the Tutorials Skip About GPU Memory

Here is the short version: the model weights are the smallest GPU-memory surprise you'll hit. A 7B model in FP16 needs about 14GB just for weights, but the KV cache — the per-request memory that grows with context length and batch size — is what actually decides whether your setup survives real traffic. Most "run an LLM on your GPU" tutorials load the weights, run one short prompt, and declare victory. Then you send a 6,000-token document at a batch of eight and it OOMs.

I've set this up enough times to know the failure isn't random. It's arithmetic you can do before you rent the GPU. This post is that arithmetic.

GPU memory for inference is four separate buckets, and tutorials only mention the first:

context_length × batch_size

. This is the one that bites.The weights number is the one everyone quotes because it's easy. Parameter count × bytes-per-parameter:

So a 7B model in 4-bit "fits" on a 24GB card with 20GB to spare, and the tutorial ends there. That leftover 20GB is not spare — it's your working budget for everything in buckets 2 through 4, and it disappears faster than you'd guess.

Takeaway: Weights tell you if the model loads; they tell you nothing about whether it serves traffic.

The KV cache stores the key and value tensors for every token already in the context so the model doesn't recompute them each step. Its size, per request, is roughly:

kv_bytes = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_element

The 2

is for keys and values. Note num_kv_heads

, not the full attention-head count — modern models use grouped-query attention (GQA), which is the single biggest reason KV cache is smaller than older formulas suggest. Worth checking your model's config, because it swings the number by 4-8×.

Worked example, Llama-3-8B-class architecture (32 layers, 8 KV heads, head_dim 128, FP16):

per_token = 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes ≈ 128 KB/token

That's about 1 GB for a single 8,192-token request. Now the part tutorials skip: this is per concurrent request. Serve a batch of 16 at that context length and you've spent ~16 GB on KV cache alone — more than the weights. On a 24GB card holding a 4-bit 8B model (~5 GB with overhead), you have maybe 17-18GB left, and you just watched a modest batch eat all of it.

Two levers shrink this:

max_model_len

.Takeaway: Budget KV cache as per_token_KB × max_context × expected_concurrency

before you pick a card — it often dwarfs the weights.

Even before a single token, CUDA and your inference framework claims memory. The CUDA context alone is typically a few hundred MB to over a gigabyte depending on driver and GPU. PyTorch's caching allocator reserves more. Frameworks like vLLM deliberately grab a large fraction of remaining VRAM up front (controlled by gpu_memory_utilization

, default 0.9) to manage the KV cache themselves — which is great for throughput but means "nvidia-smi shows 90% used" is expected, not a leak.

Then there's fragmentation. Naive allocators hand out one contiguous block per request's KV cache. When requests of varying lengths come and go, you get Swiss-cheese memory: 4GB free, but no single 2GB hole. This is exactly the problem vLLM's PagedAttention solves — it pages the KV cache like an OS pages RAM, so non-contiguous free memory is usable. If you're comparing serving frameworks, this is the practical reason vLLM sustains higher concurrency than a plain transformers

loop on the same card.

Takeaway: Reserve 1-2GB for runtime overhead as a floor, and prefer a paged-attention serving stack the moment you have concurrent requests.

These are the four I actually reach for, with honest limitations:

For a first self-host on a single 24GB consumer card (RTX 3090/4090), a 7-8B model quantized to 4-bit via Ollama or llama.cpp is the reliable starting point. When you move past one user, switch to vLLM and set max_model_len

and gpu_memory_utilization

deliberately rather than accepting defaults.

Takeaway: Match the framework to concurrency, not to model size — the model fits on the card either way; only one of them survives real traffic.

Rather than guess, do this arithmetic with your actual model config (config.json

has num_hidden_layers

, num_key_value_heads

, head_dim

or hidden_size / num_attention_heads

):

def vram_estimate_gb(params_b, bytes_per_param,
                     num_layers, num_kv_heads, head_dim,
                     max_ctx, concurrency,
                     kv_bytes=2, overhead_gb=2.0):
    weights = params_b * 1e9 * bytes_per_param / 1e9
    per_tok = 2 * num_layers * num_kv_heads * head_dim * kv_bytes
    kv = per_tok * max_ctx * concurrency / 1e9
    return round(weights + kv + overhead_gb, 1)

print(vram_estimate_gb(8, 0.5, 32, 8, 128, 8192, 8))  # ~14.5 GB

Bump concurrency

to 24 and that same setup crosses 30GB — past a 24GB card. This ten-line function has saved me more grief than any benchmark, because it turns "will it work?" into a number you check before spending money on a bigger GPU or a cloud instance. Treat it as an estimate with ±15% slack for allocator behavior, not a guarantee.

If you're self-hosting your first LLM, size for the KV cache and overhead, not just the weights — the weights are the part that always fits. For a single user on one consumer GPU, a 4-bit 7-8B model under Ollama or llama.cpp is the safe first step. The moment you have concurrent requests, move to vLLM or TGI, cap max_model_len

to what you actually need, and consider FP8 KV cache. And before you upgrade to a pricier card because you hit OOM, run the arithmetic above — nine times out of ten the fix is a smaller context ceiling or KV quantization, not more VRAM.