Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess A developer detailed a rule-of-thumb method for estimating GPU memory requirements for AI models, using Qwen2.5-7B-Instruct-AWQ as an example. The method sizes weights and KV cache from a model's spec sheet, then compares GPU options from cloud providers. The post emphasizes that quantization reduces weight footprint, leaving more VRAM for serving concurrent requests. Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping. GPU infrastructure setup — coming soon. This piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement. After reading this article you will have enough knowledge to pick a GPU family with confidence. Disclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running. AI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently paper https://arxiv.org/abs/2309.06180 , but the model isn't the only thing consuming it. Below is a short list of things that consume our precious VRAM: The sizing question is really: after weights and overhead, how much is left for the KV cache — and is that enough for your traffic? Guidance on which model to choose is outside the bounds of this article. What matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on your shortlist and eliminate the ones that don't fit your requirements. For demonstration purposes I will use Hugging Face's Qwen2.5-7B-Instruct-AWQ model card https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-AWQ , config card https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-AWQ/blob/main/config.json | What | Name | Value Qwen2.5-7B-Instruct-AWQ | |---|---|---| | Parameter count | | quantization config num hidden layers num key value heads hidden size / num attention heads The first two size the weights . The last three size the KV cache per token . That's the whole shopping list. Rule of thumb: weights ≈ parameter count × bytes per parameter and many models have different precision offerings | Precision | Bytes/param | 7.6 B params | |---|---|---| | fp16 / bf16 | 2 | ~15.2 GB | | int8 | 1 | ~7.6 GB | | AWQ / 4-bit | ~0.5 | ~3.8 GB | Note on Weights and quantization: the above table compares the same model, just with different quantization: Quantization is storing each weight in fewer bits than it was trained in. Models train in 16-bit float, so every parameter costs 2 bytes 16 bits ; quantizing re-encodes them as 8-bit or 4-bit integers. So the above table shows same model, same parameter count. Only the memory footprint for your model changes. Every GB you don't spend on weights is a GB left for KV cache, which allows serving more concurrent requests resulting in more happy customers. Already this helps guide decisions: The fp16 model variant ~15 GB plus workspace would nearly fill a 24 GB GPU before serving a single request. The AWQ variant ~5.6 GB leaves the majority of VRAM free for the KV cache. Same model, same GPU — wildly different serving capacity.^^ In truth, the AWQ actually consumes ~5.6 GB, not the advertise ~3.8 GB above. This can be seen by looking at the sum of the .safetensors file sizes on the repo's Files and versions tab For this model that's exactly two files: model-00001-of-00002.safetensors ~4.0 GB and model-00002-of-00002.safetensors ~1.6 GB . These are the weight tensors themselves — the files vLLM downloads and loads into VRAM at startup — so their combined sizeisthe weights footprint ~5.6 GB . So now you know your weights footprint, - its time to look choose a GPU that fits your requirements. Below are links to some of the large cloud providers specification sheets. GCP https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines , Azure https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview?tabs=breakdownseries%2Cgeneralsizelist%2Ccomputesizelist%2Cmemorysizelist%2Cstoragesizelist%2Cgpusizelist%2Cfpgasizelist%2Chpcsizelist , AWS https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html NOTE — reserve room for the engine:vLLM pre-claims a fraction of total VRAM and is set using the --gpu-memory-utilization flag. The vLLM engine fits weights + activations + KV cache inside that claim, leaving the remainder as headroom for CUDA overhead and fragmentation. The default is 0.92; experiments showed that was too aggressive on our GPU and we run 0.85 . So for the first pass, we are going to use Azures Standard NV36ads A10 v5 processor. Note VRAM is listed under the Accelerators Tab in the documents https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nvadsa10v5-series?tabs=sizeaccelerators — the "Memory GiB " column on the Basics tab is the VM's system RAM, not the GPU's. | Name | Accelerators | VRAM GB | |---|---|---| Now the centerpiece. On our A10 24 GB : take 85% of it, subtract ~5.6 GB of weights and vLLM's measured activation/overhead reservation, and roughly 13.76 GiB remains for the KV cache. How many tokens fit in that? Each token in flight stores a key and a value vector in every layer: bytes per token = 2 K and V × layers × kv heads × head dim × dtype bytes = 2 × 28 × 4 × 128 × 2 fp16 = 57,344 bytes ≈ 57 KB per token Pope et al., 2022 https://arxiv.org/abs/2211.05102 Derivation assumes standard attention MHA/MQA/GQA ; sliding-window, MLA and hybrid SSM models cache differently and need a different formula. Divide the remaining KV cache by the bytes per token: Token budget = 13.76 GiB / 57,344 bytes ≈ 257,584 tokens And convert tokens into the unit you actually care about — concurrent requests assuming ~1,000 tokens per request : Concurrent requests = 257,584 / 1,000 ≈ 258 One A10 can hold roughly 258 average requests in flight . That single number is what connects GPU shopping to capacity planning — it's the same Concurrent requests the flapping article https://dev.to/josef doornink 930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf builds its theoretical autoscaling threshold from. In reality the requirement is the fixed thing ie 100 concurrent requests at peak and the hardware is what you get to choose. So lets walk through how to solve for the GPU sizing from the given requirements KV bytes needed = concurrent requests × avg tokens per request × bytes per token VRAM target = KV bytes + weights / gpu memory utilization Requirements/Assumptions — 100 concurrent requests at ~1,000 tokens each, same token size as above since we have already decided our model: KV bytes needed = 100 × 1,000 × 57,344 bytes ≈ 5.7 GB VRAM target = 5.7 + 5.6 / 0.85 ≈ 13.3 GB 13.3 GB is your shopping floor: any card below it can't hold this workload, and the fractional A10 sizes 4/8/12 GB are eliminated on the spot. The A10's 24 GB clears it with room to spare — which is the useful answer, because "fits with headroom" is what lets you absorb a traffic spike without a second replica. Now you have baseline requirements and can choose the model that fits those requirements BEFORE you've spent a dollar on hardware. OK, now the cluster and service is alive with the desired GPU infrastructure setup is covered in a companion piece — coming soon . The truth at startup is finally available becuase the truth comes from vLLM itself — at startup it profiles the hardware and prints exactly what it measured:. INFO ... Available KV cache memory: 13.76 GiB INFO ... GPU KV cache size: 257,584 tokens kubectl logs