I keep seeing the same local LLM sizing mistake:
"The model file is smaller than my GPU, so it should fit."
That is only the first check. A 24 GB GPU does not give your model a clean 24 GB memory budget. The display stack, runtime, temporary buffers, model weights, and KV cache all compete for the same space.
Here is the worksheet I use before I download a model or rent a GPU.
The simplest weight estimate is:
weight_memory_gib = parameters * bits_per_parameter / 8 / 1024^3
For a simple 4-bit estimate:
| Model size | Weight floor |
|---|---|
| 7B | 3.3 GiB |
| 13B | 6.1 GiB |
| 70B | 32.6 GiB |
These are floors, not promises. Real quantized files can also contain scales, metadata, and layers stored at higher precision. If you know the exact checkpoint size, use that instead of the simple bits-per-parameter estimate.
Also use total parameters for a sparse mixture-of-experts model unless your runtime really offloads inactive experts. Active parameters describe compute per token. They do not automatically describe how many weights must be stored.
I normally start with 90 percent usable VRAM for planning:
usable_vram = physical_vram * usable_fraction
For a 24 GB card:
24 * 0.90 = 21.6 GiB usable
The exact reserve depends on the OS, display use, driver, runtime, graph capture, allocator behavior, and other processes. The important part is to stop treating the number on the box as fully available.
The KV cache is where context length and concurrency become expensive.
A useful planning formula is:
kv_cache_bytes =
2
* layers
* kv_heads
* head_dimension
* context_tokens
* concurrent_sequences
* bytes_per_kv_value
The factor of two stores keys and values.
Take a model with:
The KV cache is about 1 GiB.
Raise the context to 32,768 tokens and it becomes about 4 GiB. Keep that context and run four concurrent sequences, and it becomes about 16 GiB.
This is why a model can work in a short local chat, then fail when the server uses a larger context window or handles several requests.
Grouped-query attention matters here. Use the number of KV heads, not the total attention head count.
I use this planning target:
planning_target = (weight_memory + kv_cache) * (1 + headroom_rate)
A 20 percent headroom rate is a reasonable first estimate when no runtime measurement exists. Replace it with measured data as soon as you can.
Here is an example for a hypothetical 32B model:
A 24 GB GPU with a 21.6 GiB usable budget is short by about 5.9 GiB. The 4-bit model file looked small enough, but the deployment did not.
The architecture values in this example are only a worksheet. Read the actual model configuration before making a hardware decision.
Two 24 GB cards do not always behave like one clean 48 GB pool.
Tensor parallelism, pipeline parallelism, layer placement, replicated buffers, interconnect speed, and runtime support all matter. A capacity estimate tells you whether the plan is plausible. It does not prove latency or throughput.
Before calling a local model deployable, I write down:
If any one of those is missing, I call the answer a floor, not a deployment plan.
I put these formulas into a browser-only LLM GPU memory calculator. It does not upload the values you enter.
The two references I use most often are the Hugging Face model memory estimator guide and the Transformers KV cache guide.
What runtime-specific memory cost has surprised you most: context, concurrency, quantization overhead, or something else?
Disclosure: I used an AI assistant to help edit the structure and wording. I checked the numerical examples against the formulas above.