Why a 24 GB GPU Does Not Give Your Local LLM 24 GB

An engineer has published a practical guide to estimating GPU memory requirements for running large language models locally, warning that a 24 GB GPU does not provide a full 24 GB budget for model weights. The guide provides formulas for calculating weight memory, KV cache size, and usable VRAM, and includes a browser-based calculator tool. The author emphasizes that factors like context length, concurrency, and quantization overhead can significantly impact memory usage.

I keep seeing the same local LLM sizing mistake: "The model file is smaller than my GPU, so it should fit." That is only the first check. A 24 GB GPU does not give your model a clean 24 GB memory budget. The display stack, runtime, temporary buffers, model weights, and KV cache all compete for the same space. Here is the worksheet I use before I download a model or rent a GPU. The simplest weight estimate is: weight memory gib = parameters bits per parameter / 8 / 1024^3 For a simple 4-bit estimate: | Model size | Weight floor | |---|---| | 7B | 3.3 GiB | | 13B | 6.1 GiB | | 70B | 32.6 GiB | These are floors, not promises. Real quantized files can also contain scales, metadata, and layers stored at higher precision. If you know the exact checkpoint size, use that instead of the simple bits-per-parameter estimate. Also use total parameters for a sparse mixture-of-experts model unless your runtime really offloads inactive experts. Active parameters describe compute per token. They do not automatically describe how many weights must be stored. I normally start with 90 percent usable VRAM for planning: usable vram = physical vram usable fraction For a 24 GB card: 24 0.90 = 21.6 GiB usable The exact reserve depends on the OS, display use, driver, runtime, graph capture, allocator behavior, and other processes. The important part is to stop treating the number on the box as fully available. The KV cache is where context length and concurrency become expensive. A useful planning formula is: kv cache bytes = 2 layers kv heads head dimension context tokens concurrent sequences bytes per kv value The factor of two stores keys and values. Take a model with: The KV cache is about 1 GiB. Raise the context to 32,768 tokens and it becomes about 4 GiB. Keep that context and run four concurrent sequences, and it becomes about 16 GiB. This is why a model can work in a short local chat, then fail when the server uses a larger context window or handles several requests. Grouped-query attention matters here. Use the number of KV heads, not the total attention head count. I use this planning target: planning target = weight memory + kv cache 1 + headroom rate A 20 percent headroom rate is a reasonable first estimate when no runtime measurement exists. Replace it with measured data as soon as you can. Here is an example for a hypothetical 32B model: A 24 GB GPU with a 21.6 GiB usable budget is short by about 5.9 GiB. The 4-bit model file looked small enough, but the deployment did not. The architecture values in this example are only a worksheet. Read the actual model configuration before making a hardware decision. Two 24 GB cards do not always behave like one clean 48 GB pool. Tensor parallelism, pipeline parallelism, layer placement, replicated buffers, interconnect speed, and runtime support all matter. A capacity estimate tells you whether the plan is plausible. It does not prove latency or throughput. Before calling a local model deployable, I write down: If any one of those is missing, I call the answer a floor, not a deployment plan. I put these formulas into a browser-only LLM GPU memory calculator https://tools.researchaudio.io/llm-gpu-memory-calculator/?utm source=devto&utm medium=community article&utm campaign=local llm vram gate&utm content=24gb article . It does not upload the values you enter. The two references I use most often are the Hugging Face model memory estimator guide https://huggingface.co/docs/accelerate/en/usage guides/model size estimator and the Transformers KV cache guide https://huggingface.co/docs/transformers/main/en/kv cache . What runtime-specific memory cost has surprised you most: context, concurrency, quantization overhead, or something else? Disclosure: I used an AI assistant to help edit the structure and wording. I checked the numerical examples against the formulas above.