{"slug": "choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess", "title": "Choosing the Right GPU for Your Model — A Sizing Method, Not a Guess", "summary": "A developer detailed a rule-of-thumb method for estimating GPU memory requirements for AI models, using Qwen2.5-7B-Instruct-AWQ as an example. The method sizes weights and KV cache from a model's spec sheet, then compares GPU options from cloud providers. The post emphasizes that quantization reduces weight footprint, leaving more VRAM for serving concurrent requests.", "body_md": "*Part of a series on running vLLM on AKS. Companion piece: How to avoid flapping. GPU infrastructure setup — coming soon.*\n\nThis piece walks through estimating GPU memory requirements from both a model's parameter count or a concurrent requests requirement.\n\nAfter reading this article you will have enough knowledge to pick a GPU family with confidence.\n\nDisclaimer: this process is a rule-of-thumb filter, not a precise calculation — the last step covers how to get exact numbers once the model is actually running.\n\nAI models live in GPU memory — VRAM — and engines such as vLLM provide novel techniques for managing that memory efficiently [[paper](https://arxiv.org/abs/2309.06180)], but the model isn't the only thing consuming it.\n\nBelow is a short list of things that consume our precious VRAM:\n\nThe sizing question is really: **after weights and overhead, how much is left for the KV cache — and is that enough for your traffic?**\n\nGuidance on *which* model to choose is outside the bounds of this article.\n\nWhat matters here: once you have a candidate, everything below can be read off its spec sheet — you can then run this method on every model on your shortlist and eliminate the ones that don't fit your requirements.\n\nFor demonstration purposes I will use Hugging Face's Qwen2.5-7B-Instruct-AWQ [model card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-AWQ), [config card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-AWQ/blob/main/config.json)\n\n| What | Name | Value Qwen2.5-7B-Instruct-AWQ |\n|---|---|---|\n| Parameter count |\n|\n\n`quantization_config`\n\n`num_hidden_layers`\n\n`num_key_value_heads`\n\n`hidden_size / num_attention_heads`\n\nThe first two size the **weights**. The last three size the **KV cache per token**.\n\nThat's the whole shopping list.\n\nRule of thumb: `weights ≈ parameter_count × bytes_per_parameter`\n\nand many models have different precision offerings\n\n| Precision | Bytes/param | 7.6 B params |\n|---|---|---|\n| fp16 / bf16 | 2 | ~15.2 GB |\n| int8 | 1 | ~7.6 GB |\n| AWQ / 4-bit | ~0.5 | ~3.8 GB |\n\nNote on Weights and quantization: the above table compares the same model, just with different quantization:\n\n**Quantization** is storing each weight in fewer bits than it was trained in. Models train in 16-bit float, so every parameter costs 2 bytes (16 bits); quantizing re-encodes them as 8-bit or 4-bit integers.\n\nSo the above table shows same model, same parameter count. Only the memory footprint for your model changes.\n\nEvery GB you don't spend on weights is a GB left for KV cache, which allows serving more concurrent requests resulting in more happy customers.\n\nAlready this helps guide decisions:\n\nThe fp16 model variant (~15 GB) plus workspace would nearly fill a 24 GB GPU before serving a single request.\n\nThe AWQ variant (~5.6 GB) leaves the majority of VRAM free for the KV cache.\n\nSame model, same GPU — wildly different serving capacity.^^\n\nIn truth, the AWQ actually consumes ~5.6 GB, not the advertise ~3.8 GB above. This can be seen by looking at the sum of the\n\n`.safetensors`\n\nfile sizes on the repo's[Files and versions tab]For this model that's exactly two files:[model-00001-of-00002.safetensors](~4.0 GB) and[model-00002-of-00002.safetensors](~1.6 GB). These are the weight tensors themselves — the files vLLM downloads and loads into VRAM at startup — so their combined sizeisthe weights footprint (~5.6 GB).\n\nSo now you know your weights footprint, - its time to look choose a GPU that fits your requirements.\n\nBelow are links to some of the large cloud providers specification sheets.\n\n[GCP](https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines), [Azure](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/overview?tabs=breakdownseries%2Cgeneralsizelist%2Ccomputesizelist%2Cmemorysizelist%2Cstoragesizelist%2Cgpusizelist%2Cfpgasizelist%2Chpcsizelist), [AWS](https://docs.aws.amazon.com/ec2/latest/instancetypes/ac.html)\n\nNOTE — reserve room for the engine:vLLM pre-claims a fraction of total VRAM and is set using the`--gpu-memory-utilization`\n\nflag.\n\nThe vLLM engine fits weights + activations + KV cache inside that claim, leaving the remainder as headroom for CUDA overhead and fragmentation.\n\nThe default is 0.92; experiments showed that was too aggressive on our GPU and[we run 0.85].\n\nSo for the first pass, we are going to use Azures Standard_NV36ads_A10_v5 processor. (Note VRAM is listed under the Accelerators Tab in the [documents](https://learn.microsoft.com/en-us/azure/virtual-machines/sizes/gpu-accelerated/nvadsa10v5-series?tabs=sizeaccelerators) — the \"Memory (GiB)\" column on the Basics tab is the VM's system RAM, not the GPU's.)\n\n| Name | Accelerators | VRAM (GB) |\n\n|---|---|---|\n\nNow the centerpiece.\n\nOn our A10 (24 GB): take 85% of it, subtract ~5.6 GB of weights and vLLM's measured activation/overhead reservation, and roughly **13.76 GiB** remains for the KV cache.\n\nHow many tokens fit in that? Each token in flight stores a key and a value vector in every layer:\n\n```\nbytes_per_token* = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes\n                = 2 × 28 × 4 × 128 × 2 (fp16)\n                = 57,344 bytes ≈ 57 KB per token\n\n[*Pope et al., 2022](https://arxiv.org/abs/2211.05102) \nDerivation assumes standard attention (MHA/MQA/GQA); sliding-window, MLA and hybrid SSM models cache differently and need a different formula.\n```\n\nDivide the remaining KV cache by the bytes per token:\n\n```\nToken budget  = 13.76 GiB / 57,344 bytes ≈ 257,584 tokens\n```\n\nAnd convert tokens into the unit you actually care about — concurrent requests (assuming ~1,000 tokens per request):\n\n```\nConcurrent_requests = 257,584 / 1,000 ≈ 258\n```\n\nOne A10 can hold roughly **258 average requests in flight**. That single number is what connects GPU shopping to capacity planning — it's the same `Concurrent_requests`\n\nthe [flapping article](https://dev.to/josef_doornink_930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf) builds its theoretical autoscaling threshold from.\n\nIn reality the *requirement* is the fixed thing (ie 100 concurrent requests at peak) and the hardware is what you get to choose.\n\nSo lets walk through how to solve for the GPU sizing from the given requirements\n\n```\nKV bytes needed = concurrent_requests × avg_tokens_per_request × bytes_per_token\nVRAM target     = (KV bytes + weights) / gpu_memory_utilization\n```\n\nRequirements/Assumptions — 100 concurrent requests at ~1,000 tokens each, same token size as above since we have already decided our model:\n\n```\nKV bytes needed = 100 × 1,000 × 57,344 bytes ≈ 5.7 GB\nVRAM target = (5.7 + 5.6) / 0.85         ≈ 13.3 GB\n```\n\n13.3 GB is your shopping floor: any card below it can't hold this workload, and the fractional A10 sizes (4/8/12 GB) are eliminated on the spot.\n\nThe A10's 24 GB clears it with room to spare — which is the useful answer, because \"fits with headroom\" is what lets you absorb a traffic spike without a second replica.\n\nNow you have baseline requirements and can choose the model that fits those requirements BEFORE you've spent a dollar on hardware.\n\nOK, now the cluster and service is alive with the desired GPU (infrastructure setup is covered in a companion piece — coming soon).\n\nThe truth at startup is finally available becuase the truth comes from vLLM itself — at startup it profiles the hardware and prints exactly what it measured:.\n\n```\nINFO ... Available KV cache memory: 13.76 GiB\nINFO ... GPU KV cache size: 257,584 tokens\n```\n\n`kubectl logs <vllm-pod> | grep -i \"kv cache\"`\n\nand compare against your Step 4 numbers.\n\nIf they're close, your mental model of the card is correct.\n\nIf they're way off, something in your assumptions is wrong (usually the quantization variant or the `gpu-memory-utilization`\n\nvalue) — better to find out now than after you've sized a node pool around it.\n\n`config.json`\n\n: parameter count, quantization, `num_hidden_layers`\n\n, `num_key_value_heads`\n\n, and `hidden_size / num_attention_heads`\n\n.`.safetensors`\n\nfile sizes (the rule of thumb runs low on quantized models).`gpu-memory-utilization`\n\n, minus weights, ÷ bytes-per-token = tokens; ÷ tokens-per-request = `Concurrent_requests`\n\n. Note 0.85 is a safer starting point than vLLM's 0.92 default.Next in the series: that `Concurrent_requests`\n\nnumber is the foundation of a defensible autoscaling threshold — and why even a defensible threshold isn't enough: [Your vLLM Autoscaler Is Flapping Because You Picked the Wrong Signal](https://dev.to/josef_doornink_930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf).\n\nEverything above sized *one* pod on *one* GPU. Fair question: what happens when the cluster has more than one? There are three scenarios here, and they're worth keeping distinct — because only one of them changes the math you just learned.\n\n**Scenario 1 — many single-GPU nodes (what this series runs).** Each pod owns one GPU and holds a full copy of the model; traffic is load-balanced across replicas. This is *data parallelism*, and it's already a multi-GPU cluster — the [flapping article](https://dev.to/josef_doornink_930b2caf1c/your-vllm-autoscaler-is-flapping-because-you-picked-the-wrong-signal-not-the-wrong-number-24lf) scales exactly this fleet from 1 to 3 GPUs. Nothing in the sizing math changes: total capacity is simply `C × replicas`\n\n.\n\n**Scenario 2 — multi-GPU nodes, still one GPU per pod.** Some VM sizes pack multiple cards (e.g. Azure's NV72ads_A10_v5 has 2× A10). Keep `nvidia.com/gpu: \"1\"`\n\nper pod and Kubernetes schedules two vLLM pods onto one node. The sizing math is *still* unchanged — each pod sees its own 24 GB card. This holds as long as [GPU time-slicing](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) is disabled; with it enabled, two pods can share one physical card with no memory isolation and none of the numbers above apply.\n\nWhat does change is operational :\n\n**Scenario 3 — multiple GPUs per pod: tensor parallelism. This is the one that changes the math.** When the model doesn't fit on any single card — Llama-70B at fp16 is ~140 GB of weights alone — vLLM can split every layer\n\n`--tensor-parallel-size N`\n\n, and the pod requests `nvidia.com/gpu: N`\n\n. Three consequences for sizing:`kv_heads`\n\nterm in the bytes-per-token formula divides across GPUs too. (Note the constraint: with Qwen's 4 KV heads, tensor parallelism beyond 4 stops dividing cleanly.)The clean rule of thumb: **more traffic → more replicas (Scenarios 1–2); bigger model → tensor parallelism (Scenario 3).** Reach for Scenario 3 only when weights plus a workable KV budget exceed the biggest single card you can buy — otherwise replicas are simpler, cheaper, and fail more gracefully.\n\n`--gpu-memory-utilization 0.85`\n\nfixed it.`params × 0.5 bytes`\n\nsays 3.8 GB; the real artifact is ~5.6 GB (fp16 embeddings + quantization scales). Check the actual file sizes on the repo.", "url": "https://wpnews.pro/news/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess", "canonical_source": "https://dev.to/josef_doornink_930b2caf1c/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess-4fe5", "published_at": "2026-08-19 00:24:15+00:00", "updated_at": "2026-08-19 00:42:12.370921+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "ai-tools"], "entities": ["vLLM", "Hugging Face", "Qwen2.5-7B-Instruct-AWQ", "GCP", "Azure", "AWS"], "alternates": {"html": "https://wpnews.pro/news/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess", "markdown": "https://wpnews.pro/news/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess.md", "text": "https://wpnews.pro/news/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess.txt", "jsonld": "https://wpnews.pro/news/choosing-the-right-gpu-for-your-model-a-sizing-method-not-a-guess.jsonld"}}