Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping. GPU infrastructure setup — coming soon.
This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement.
After reading this article you will have enough knowledge to pick a GPU family with confidence.
Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running.
AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [paper], but the model isn't the only thing consuming it.
Below is a short list of things that consume our precious VRAM:
The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic?
Guidance on which model to choose is outside the bounds of this article.
What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on your shortlist and eliminate the ones that don't fit your requirements.
For demonstration purposes I will use Hugging Face's Qwen2.5-7B-Instruct-AWQ model card, config card
| What | Name | Value Qwen2.5-7B-Instruct-AWQ |
|---|---|---|
| Parameter count | ||
quantization_config
num_hidden_layers
num_key_value_heads
hidden_size / num_attention_heads
The first two size the weights. The last three size the KV cache per token.
That's the whole shopping list.
Rule of thumb: weights ≈ parameter_count × bytes_per_parameter
and many models have different precision offerings
| Precision | Bytes/param | 7.6 B params |
|---|---|---|
| fp16 / bf16 | 2 | ~15.2 GB |
| int8 | 1 | ~7.6 GB |
| AWQ / 4-bit | ~0.5 | ~3.8 GB |
Note on Weights and quantization: the above table compares the same model, just with different quantization:
Quantization is storing each weight in fewer bits than it was trained in. Models train in 16-bit float, so every parameter costs 2 bytes (16 bits); quantizing re-encodes them as 8-bit or 4-bit integers.
So the above table shows same model, same parameter count. Only the memory footprint for your model changes.
Every GB you don't spend on weights is a GB left for KV cache, which allows serving more concurrent requests resulting in more happy customers.
Already this helps guide decisions:
The fp16 model variant (~15 GB) plus workspace would nearly fill a 24 GB GPU before serving a single request.
The AWQ variant (~5.6 GB) leaves the majority of VRAM free for the KV cache.
Same model, same GPU — wildly different serving capacity.^^
In truth, the AWQ actually consumes ~5.6 GB, not the advertise ~3.8 GB above. This can be seen by looking at the sum of the
.safetensors
file sizes on the repo's[Files and versions tab]For this model that's exactly two files:[model-00001-of-00002.safetensors](~4.0 GB) and[model-00002-of-00002.safetensors](~1.6 GB). These are the weight tensors themselves — the files vLLM downloads and loads into VRAM at startup — so their combined sizeisthe weights footprint (~5.6 GB).
So now you know your weights footprint, - its time to look choose a GPU that fits your requirements.
Below are links to some of the large cloud providers specification sheets.
NOTE — reserve room for the engine:vLLM pre-claims a fraction of total VRAM and is set using the--gpu-memory-utilization
flag.
The vLLM engine fits weights + activations + KV cache inside that claim, leaving the remainder as headroom for CUDA overhead and fragmentation.
The default is 0.92; experiments showed that was too aggressive on our GPU and[we run 0.85].
So for the first pass, we are going to use Azures Standard_NV36ads_A10_v5 processor. (Note VRAM is listed under the Accelerators Tab in the documents — the "Memory (GiB)" column on the Basics tab is the VM's system RAM, not the GPU's.)
| Name | Accelerators | VRAM (GB) |
|---|---|---|
Now the centerpiece.
On our A10 (24 GB): take 85% of it, subtract ~5.6 GB of weights and vLLM's measured activation/overhead reservation, and roughly 13.76 GiB remains for the KV cache.
How many tokens fit in that? Each token in flight stores a key and a value vector in every layer:
bytes_per_token* = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes
= 2 × 28 × 4 × 128 × 2 (fp16)
= 57,344 bytes ≈ 57 KB per token
[*Pope et al., 2022](https://arxiv.org/abs/2211.05102)
Derivation assumes standard attention (MHA/MQA/GQA); sliding-window, MLA and hybrid SSM models cache differently and need a different formula.
Divide the remaining KV cache by the bytes per token:
Token budget = 13.76 GiB / 57,344 bytes ≈ 257,584 tokens
And convert tokens into the unit you actually care about — concurrent requests (assuming ~1,000 tokens per request):
Concurrent_requests = 257,584 / 1,000 ≈ 258
One A10 can hold roughly 258 average requests in flight. That single number is what connects GPU shopping to capacity planning — it's the same Concurrent_requests
the flapping article builds its theoretical autoscaling threshold from.
In reality the requirement is the fixed thing (ie 100 concurrent requests at peak) and the hardware is what you get to choose.
So lets walk through how to solve for the GPU sizing from the given requirements
KV bytes needed = concurrent_requests × avg_tokens_per_request × bytes_per_token
VRAM target = (KV bytes + weights) / gpu_memory_utilization
Requirements/Assumptions — 100 concurrent requests at ~1,000 tokens each, same token size as above since we have already decided our model:
KV bytes needed = 100 × 1,000 × 57,344 bytes ≈ 5.7 GB
VRAM target = (5.7 + 5.6) / 0.85 ≈ 13.3 GB
13.3 GB is your shopping floor: any card below it can't hold this workload, and the fractional A10 sizes (4/8/12 GB) are eliminated on the spot.
The A10's 24 GB clears it with room to spare — which is the useful answer, because "fits with headroom" is what lets you absorb a traffic spike without a second replica.
Now you have baseline requirements and can choose the model that fits those requirements BEFORE you've spent a dollar on hardware.
OK, now the cluster and service is alive with the desired GPU (infrastructure setup is covered in a companion piece — coming soon).
The truth at startup is finally available becuase the truth comes from vLLM itself — at startup it profiles the hardware and prints exactly what it measured:.
INFO ... Available KV cache memory: 13.76 GiB
INFO ... GPU KV cache size: 257,584 tokens
kubectl logs <vllm-pod> | grep -i "kv cache"
and compare against your Step 4 numbers.
If they're close, your mental model of the card is correct.
If they're way off, something in your assumptions is wrong (usually the quantization variant or the gpu-memory-utilization
value) — better to find out now than after you've sized a node pool around it.
config.json
: parameter count, quantization, num_hidden_layers
, num_key_value_heads
, and hidden_size / num_attention_heads
..safetensors
file sizes (the rule of thumb runs low on quantized models).gpu-memory-utilization
, minus weights, ÷ bytes-per-token = tokens; ÷ tokens-per-request = Concurrent_requests
. Note 0.85 is a safer starting point than vLLM's 0.92 default.Next in the series: that Concurrent_requests
number is the foundation of a defensible autoscaling threshold — and why even a defensible threshold isn't enough: Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal.
Everything above sized one pod on one GPU. Fair question: what happens when the cluster has more than one? There are three scenarios here, and they're worth keeping distinct — because only one of them changes the math you just learned.
Scenario 1 — many single-GPU nodes (what this series runs). Each pod owns one GPU and holds a full copy of the model; traffic is load-balanced across replicas. This is data parallelism, and it's already a multi-GPU cluster — the flapping article scales exactly this fleet from 1 to 3 GPUs. Nothing in the sizing math changes: total capacity is simply C × replicas
.
Scenario 2 — multi-GPU nodes, still one GPU per pod. Some VM sizes pack multiple cards (e.g. Azure's NV72ads_A10_v5 has 2× A10). Keep nvidia.com/gpu: "1"
per pod and Kubernetes schedules two vLLM pods onto one node. The sizing math is still unchanged — each pod sees its own 24 GB card. This holds as long as GPU time-slicing is disabled; with it enabled, two pods can share one physical card with no memory isolation and none of the numbers above apply.
What does change is operational :
Scenario 3 — multiple GPUs per pod: tensor parallelism. This is the one that changes the math. When the model doesn't fit on any single card — Llama-70B at fp16 is ~140 GB of weights alone — vLLM can split every layer
--tensor-parallel-size N
, and the pod requests nvidia.com/gpu: N
. Three consequences for sizing:kv_heads
term in the bytes-per-token formula divides across GPUs too. (Note the constraint: with Qwen's 4 KV heads, tensor parallelism beyond 4 stops dividing cleanly.)The clean rule of thumb: more traffic → more replicas (Scenarios 1–2); bigger model → tensor parallelism (Scenario 3). Reach for Scenario 3 only when weights plus a workable KV budget exceed the biggest single card you can buy — otherwise replicas are simpler, cheaper, and fail more gracefully.
--gpu-memory-utilization 0.85
fixed it.params × 0.5 bytes
says 3.8 GB; the real artifact is ~5.6 GB (fp16 embeddings + quantization scales). Check the actual file sizes on the repo.