Running a large language model locally sounds simple until you start looking at GPU memory.
A model has 70 billion parameters. Your GPU has 24 GB, 32 GB, 48 GB, or maybe 80 GB of VRAM.
So will it fit?
Unfortunately, parameter count alone does not answer that question.
To estimate how much VRAM an LLM actually needs, you need to consider at least four things:
And if the model does not fit entirely in VRAM, you also need to think about CPU off and multi-GPU inference.
Let's break it down.
At the most basic level, the memory required for model weights can be estimated with:
Model memory ≈ parameters × bits per parameter ÷ 8
For a 70-billion-parameter model:
| Precision | Approx. raw weight memory |
|---|---|
| FP16 / BF16 | 140 GB |
| INT8 | 70 GB |
| 6-bit | 52.5 GB |
| 5-bit | 43.75 GB |
| 4-bit | 35 GB |
| 3-bit | 26.25 GB |
These numbers are only a starting point.
A "4-bit" model does not necessarily occupy exactly 35 GB in VRAM.
Quantization formats often include scales, metadata, higher-precision tensors, and other overhead. Different quantization methods can therefore produce noticeably different memory requirements even when both are described as "4-bit."
This is why looking only at the advertised quantization level can be misleading.
Without quantization, a 70B model is far beyond the memory capacity of normal consumer GPUs.
At FP16:
70 billion × 2 bytes ≈ 140 GB
That means even an 80 GB accelerator cannot hold the raw weights entirely in memory.
At INT8:
70 billion × 1 byte ≈ 70 GB
Now the model becomes feasible on very large accelerator cards, although there still needs to be room for runtime overhead and KV cache.
At approximately 4-bit:
70 billion × 0.5 bytes ≈ 35 GB
Suddenly the model becomes practical on configurations with around 48 GB of GPU memory.
This is why quantization has been so important for local LLM inference.
Instead of needing several enterprise accelerators, heavily quantized models can sometimes run on workstation GPUs or multiple consumer cards.
The tradeoff is that increasingly aggressive quantization can affect model quality.
So the goal should not simply be:
Use the smallest model possible.
A better goal is:
Use the highest-quality quantization that fits comfortably within the hardware you have.
Cards with 24 GB of VRAM are extremely useful for local AI.
But 24 GB is still substantially below the roughly 35 GB theoretical weight requirement of a 70B model at exactly 4 bits per parameter.
That means something has to change.
You can:
CPU off is particularly interesting because it allows models much larger than GPU memory to run.
But capacity and performance are two different problems.
Suppose you have:
GPU VRAM: 24 GB
System RAM: 64 GB
You potentially have enough total memory to store a quantized 70B model.
The runtime can keep some layers in GPU memory while storing the remaining layers in normal system RAM.
That works.
But system RAM bandwidth is dramatically lower than modern GPU VRAM bandwidth.
For autoregressive LLM inference, weights may need to be accessed repeatedly as each token is generated.
If part of those weights must travel between CPU memory and the GPU, token generation can slow considerably.
So when evaluating hardware for local AI, I separate two questions:
and
Those are not the same thing.
A 32 GB GPU gets much closer.
The theoretical size of a 4-bit 70B model is still around:
35 GB
So a straightforward 4-bit model will generally still exceed 32 GB before accounting for additional memory requirements.
However, more aggressive quantizations can bring some 70B-class models within range.
That makes 32 GB cards interesting for users willing to trade some model fidelity for the ability to stay mostly—or entirely—on the GPU.
But fitting the weights is only part of the problem.
There is another large consumer of GPU memory:
context.
When an LLM processes a conversation, it stores information associated with previous tokens in a structure called the KV cache.
The longer the conversation becomes, the larger that cache becomes.
So a model that fits comfortably at:
4,096 tokens
may consume considerably more memory at:
32,768 tokens
or:
131,072 tokens
This creates one of the most common mistakes when estimating GPU requirements.
Someone downloads a model that appears to require 22 GB of memory and assumes it will fit comfortably on a 24 GB GPU.
Then the runtime loads:
and suddenly there isn't enough memory.
Consider two people running the exact same model.
Runs:
4K context
1 concurrent request
128K context
4 concurrent requests
They may have dramatically different memory requirements even though they're using the same model.
This is especially important for:
If you're buying hardware for AI, model size should never be considered separately from intended context length.
Even after accounting for weights and KV cache, you should avoid planning a system that uses exactly 100% of available VRAM.
Inference frameworks need working memory.
Depending on the runtime, that can include:
A machine that technically fits a model with 200 MB of VRAM remaining may be much less useful than one with several gigabytes of headroom.
For that reason, I generally think of GPU memory as a budget, not a hard model-size limit.
Around 48 GB of VRAM is where 70B-class local inference becomes significantly easier.
A 4-bit 70B model with a theoretical weight size around 35 GB leaves substantially more room for:
That does not mean every 70B model and every context configuration will fit.
But compared with 24 GB or 32 GB, you have far more flexibility.
This is one reason older professional GPUs with large VRAM capacities can remain interesting for AI even when newer gaming GPUs have considerably more raw compute.
For LLM inference, sometimes:
the slower GPU that fits the whole model is more useful than the faster GPU that doesn't.
This is another common question.
At first glance:
24 GB + 24 GB = 48 GB
So two 24 GB GPUs should behave exactly like one 48 GB GPU.
Not quite.
A runtime can distribute model layers or tensors across both GPUs, allowing the combined memory capacity to hold a larger model.
But the GPUs still have physically separate memory pools.
Communication must occur over:
The topology and inference framework therefore matter.
Two GPUs can dramatically expand the models you are able to run, but they do not magically become a single GPU.
Still, for local AI enthusiasts, used high-VRAM consumer GPUs can sometimes create very interesting price-to-memory configurations.
Once the model fits, another specification becomes increasingly important:
memory bandwidth.
LLM token generation frequently involves moving large amounts of model data through memory.
That means two GPUs with similar compute capability can behave very differently depending on:
Other important factors include:
This is why comparing AI GPUs purely by TFLOPS is often misleading.
Instead of asking:
What is the fastest GPU?
I recommend asking these questions in order.
8B?
32B?
70B?
Mixture-of-Experts model?
FP16?
FP8?
INT8?
Q6?
Q5?
Q4?
4K?
32K?
128K?
More?
If not:
There is a huge difference between:
2 tokens/sec
and:
50 tokens/sec
Both configurations technically "run" the model.
Only one may be pleasant to use interactively.
Here is a deliberately simplified way to think about it.
A normal 4-bit 70B model will not fit entirely in VRAM.
Expect aggressive quantization, CPU off, or multiple GPUs.
Closer, but still below the theoretical size of a standard 4-bit 70B model.
Aggressive quantization may make some configurations possible.
A much more comfortable target for 4-bit 70B-class inference.
Context length and runtime overhead still matter.
Enough for approximately 8-bit weights in theory, although KV cache and runtime overhead must still be accounted for.
FP16 70B remains far above the capacity of a single 80 GB GPU.
The most important lesson is that:
parameters ≠ VRAM requirement
The real calculation is closer to:
Model weights
+ KV cache
+ runtime overhead
+ safety margin
= required GPU memory
And even after answering that question, you still need to consider memory bandwidth and software support to estimate actual performance.
I built CompareAIHardware around exactly this problem: comparing GPUs, accelerators, VRAM capacity, memory bandwidth, and model requirements from the perspective of people actually trying to run AI workloads locally.
The next time you see someone ask:
"Can I run a 70B model on my GPU?"
the correct answer probably isn't simply yes or no.
The better answer is:
Which 70B model, which quantization, which context length, and how much of it needs to stay in VRAM?
If you're building a local AI machine, those four questions can save you a very expensive GPU purchase.