# AirLLM Runs a 70B Model on a 4GB GPU. It's True, and That's Not the Interesting Part

> Source: <https://dev.to/arshtechpro/airllm-runs-a-70b-model-on-a-4gb-gpu-its-true-and-thats-not-the-interesting-part-hha>
> Published: 2026-08-03 22:23:56+00:00

AirLLM's README opens with a line that sounds like it can't be true:

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning.

So let's actually check it. Is the claim real? How do you set it up? And — the question nobody asks loudly enough — **should you?**

**TL;DR:** The claim is technically true and the engineering is legitimate.

Here's the thing about a transformer: it's a stack of layers, and it runs them **in order**.

```
Input → Layer 1 → Layer 2 → Layer 3 → ... → Layer 80 → Output
```

When Layer 1 is computing, layers 2 through 80 are just... sitting in your VRAM. Doing nothing. Taking up space.

Normal inference loads all 80 layers into GPU memory because keeping them there is fast. AirLLM asks the obvious follow-up question: what if we didn't? Load Layer 1, run it, throw it away, load Layer 2, run it, throw it away.

Now your VRAM requirement isn't "the size of the model." It's **"the size of the single largest layer."**

For a 70B model at full FP16 precision, that's roughly 1.75GB per layer. Which fits in 4GB with room to spare. The model is still 140GB — it just lives on your disk instead of your GPU, streaming through one slice at a time.

That's it. That's the whole idea. And it genuinely works.

**Yes — with an asterisk the size of the model itself.**

Let's take the claims one at a time.

Real. The math checks out (~1.75GB per layer at FP16), and it's been independently reproduced by enough people that this isn't in dispute.

Real, and this is the genuinely interesting part. Most "run big models on small hardware" tricks work by *making the model worse* — squashing weights from 16 bits to 4, which costs you some accuracy. AirLLM doesn't have to. You get the actual, unmodified, full-precision model.

(Quantization is *optional* here — you can pass `compression='4bit'`

to make it faster. But you don't have to, and that's the distinction they're drawing.)

The README's scaling table looks absurd but follows from the same logic:

| Model | Size | Claimed VRAM |
|---|---|---|
| Llama 3.x 70B | 70B | ~4 GB |
| Llama 3.1 405B | 405B | ~8 GB |
| DeepSeek-V3 | 671B | ~12 GB |
| Qwen3-235B (MoE) | 235B | ~3 GB |
| Kimi K3 (MoE) | 2.8T | ~3.7 GB |

Notice something weird? **The 2.8-trillion-parameter model needs less VRAM than the 671B one.**

That's not an error. Those are Mixture-of-Experts models. An MoE layer contains hundreds of "expert" sub-networks, but each token only routes to a handful of them. Per the v3.1.0 release notes, Kimi K3 holds 896 experts per layer and routes each token to just 16 — so while a full layer's experts expand to ~55GB, a single token only actually needs ~1GB of them. AirLLM streams *just those experts* instead of the whole layer.

Sparser model → smaller working set → less VRAM. Counterintuitive, correct.

Here's the part that isn't in the big bold text. From AirLLM's **own v3.1.0 release notes**, measured on an RTX 6000 Ada:

| Metric | Value |
|---|---|
| Peak VRAM during generation | 3.72 GB |
| One-time init | 900 seconds |
| Generation speed | 292 s/token, disk-bound |

To be clear about what that means: a 100-token response would take **just over 8 hours**.

Credit where it's due — the maintainer publishes this honestly in the release notes. It's just not the number on the marketing line.

For more typical setups, community reports land in the range of:

So the honest version of the claim is:

AirLLM doesn't make 70B fast on a 4GB GPU. It makes 70Bpossibleon a 4GB GPU.

This is worth internalizing, because it explains everything and it's not complicated.

**To generate one token, the model must run every single layer.** Which means AirLLM must read **the entire model off disk** — for **every token**.

So:

```
seconds per token ≈ model size on disk ÷ disk read speed
```

Let's plug in a 70B model at FP16 (~140GB):

| Storage | Speed | Time per token |
|---|---|---|
| Gen4 NVMe SSD | ~7 GB/s | ~20 s |
| Gen3 NVMe SSD | ~3.5 GB/s | ~40 s |
| SATA SSD | ~0.5 GB/s | ~280 s |
| Spinning HDD | ~0.15 GB/s |

**Your disk is your inference engine.** The GPU is barely working — it's sitting idle waiting for data. This is why AirLLM users report their fans screaming and their laptop becoming unusable: the bottleneck is I/O and CPU, not compute.

Two consequences that fall right out of this formula:

`compression='4bit'`

.**Disk space is the #1 thing that will bite you.** AirLLM downloads the model, *then* decomposes it into per-layer shards. For a while, you have **both copies on disk**.

For a 70B FP16 model, budget:

```
~140GB (original download)
+ ~140GB (layer shards)
= ~280GB free space
```

The single most common error in the repo's FAQ — `safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer`

— is, per the maintainers, almost always just **you ran out of disk**.

You'll also want:

```
pip install airllm
```

For the 4-bit compression speedup (recommended — see the math above):

```
pip install -U bitsandbytes
python
from airllm import AutoModel

model = AutoModel.from_pretrained(
    "Qwen/Qwen3-32B",
    compression='4bit',        # ~3x faster; skip for full precision
    delete_original=True,      # deletes the original after splitting — saves ~50% disk
    profiling_mode=True,       # logs per-layer timing so you can see the bottleneck
)

input_text = ['What is the capital of United States?']

input_tokens = model.tokenizer(
    input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=128,
    padding=False,             # avoids a common tokenizer error
)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True,
)

print(model.tokenizer.decode(generation_output.sequences[0]))
```

That's the whole API. Swapping to a 671B model is a one-line change:

```
model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")   # 671B, ~12GB VRAM
```

**Start small.** Please run an 8B model first to validate your setup before committing 280GB and several hours to a 70B download.

| Flag | What it does |
|---|---|
`compression` |
`'4bit'` or `'8bit'` block-wise quantization — biggest speed lever |
`delete_original` |
Deletes the original download after splitting; halves disk usage |
`layer_shards_saving_path` |
Put shards on a different (faster/bigger) drive |
`profiling_mode` |
Logs time consumption per layer |
`hf_token` |
For gated models (Llama, etc.) |
`prefetching` |
Overlaps loading and compute (~10% gain); on by default |

| Error | Actual cause |
|---|---|
`MetadataIncompleteBuffer` |
Out of disk space. It's basically always this. |
`401 Client Error... Repo is gated` |
Pass `hf_token='...'`
|
`Asking to pad but the tokenizer does not have a padding token` |
Set `padding=False`
|
`ValueError: max() arg is an empty sequence` |
Use `AutoModel` , not a specific model class |

Works on Apple Silicon only. Install [mlx](https://github.com/ml-explore/mlx) plus torch, and make sure you're on native (not Rosetta) Python. Same code otherwise.

Depends entirely on which of these you are.

This is the fantasy that draws people in, and it just doesn't work. Interactive chat needs ~20+ tokens/sec. AirLLM gives you seconds-to-minutes *per token*. You will not be having a conversation.

Every token reads tens of GB off your SSD. Consumer NVMe drives are rated for a finite number of terabytes written; hammering one with continuous full-model reads and shard rewrites is not what it was designed for. Also, your machine will be effectively unusable while it runs.

**This is the real use case, and it's underrated.**

The killer insight: the expensive part is *loading a layer*, not *using it*. So if you load Layer 1 and run **50 prompts through it** before moving on, you amortize that cost 50 ways.

One reported benchmark: **35 s/token for a single prompt vs 5.3 s/token when batching 50** — a 6.6x improvement for free.

So if you have 10,000 documents to classify overnight and no GPU budget, AirLLM is a legitimately reasonable tool. Latency doesn't matter when nobody's waiting.

Research on quantization effects, numerical reproducibility, evaluating a model as-published — cases where a 4-bit approximation defeats the point. AirLLM is close to the only way to do this on hardware you already own.

**The claim is real.** 70B on 4GB VRAM, full precision, no tricks in the "lying" sense. The engineering is clever and the MoE expert-streaming work is legitimately impressive.

**The framing is the problem.** "Run 70B on a 4GB GPU" implies you get 70B-quality answers on cheap hardware. What you actually get is 70B-quality answers *eventually* — at a pace measured in minutes per token, with your SSD as the engine and your GPU mostly idle.

AirLLM didn't remove the cost of running a huge model. It **moved** it — out of VRAM, into time and disk I/O. Whether that's a good trade depends entirely on whether you have more time than money.

**Links:**

*Have you actually run this on real hardware? I'd love to hear your tokens/sec and your disk setup in the comments — the community numbers vary wildly and more data points would help.*