# AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required

> Source: <https://dibi8.com/resources/llm-frameworks/airllm-run-huge-llms-tiny-gpu-2026/>
> Published: 2026-08-02 13:15:00+00:00

# AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required

AirLLM is an Apache-2.0 Python library that runs 70B+ LLMs — including Llama 3.1 405B on 8GB and DeepSeek-V3 671B on ~12GB — by streaming one layer (or one MoE expert) onto the GPU at a time instead of loading the whole model, no quantization, distillation, or pruning required.

- ⭐ 25300
- Python
- Apache-2.0
- Updated 2026-08-02

[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/) •
[Ollama: Run LLMs Locally with One Command](https://dibi8.com/resources/llm-frameworks/ollama/)

*Project logo — from github.com/lyogavin/airllm*

## What Is AirLLM? [#](#what-is-airllm)

**AirLLM** dramatically reduces LLM inference memory usage, letting a 70B model run on a single 4GB GPU — without quantization, distillation, or pruning. Per the current README, that same approach scales further: **Llama 3.1 405B on 8GB**, **DeepSeek-V3 (671B) on ~12GB**, and, as of July 2026, **Kimi K3 (2.8T parameters, the largest open-source model released to date) on under 4GB**, because sparse MoE models let AirLLM stream one expert at a time instead of a whole dense layer.

🔗 **GitHub**: [https://github.com/lyogavin/airllm](https://github.com/lyogavin/airllm)
📦 **PyPI**: [https://pypi.org/project/airllm](https://pypi.org/project/airllm/)

Apache-2.0 licensed, at **25,300+ GitHub stars**, with a version history running continuously from its **November 2023** initial release through a **July 2026** commit adding Kimi K3 support — this is a mature project with a genuine multi-year track record, not a recent trend-chaser.

## The Trick: One Layer on the GPU at a Time [#](#the-trick-one-layer-on-the-gpu-at-a-time)

Most “run a huge model on small hardware” techniques (quantization, pruning, distillation) change the model itself to make it smaller. AirLLM instead changes *how much of the model is ever loaded at once*: it decomposes the model layer-by-layer, and during inference only ever keeps **one layer on the GPU** — for MoE models, one **routed expert** — streaming the rest from disk/CPU as needed.

The practical consequence: your required VRAM is driven by the size of the model’s *largest single layer*, not its total parameter count.

| Model | Size | GPU VRAM |
|---|---|---|
| Qwen3 / Mistral / Phi (~8B) | 8B | ~1–2 GB |
| Qwen3-30B / Mixtral (MoE) | 30–47B | ~1–3 GB |
| Qwen3-235B (MoE) | 235B | ~3 GB |
| Llama 3.x 70B (full precision) | 70B | ~4 GB |
| Llama 3.1 405B | 405B | ~8 GB |
| DeepSeek-V3 | 671B | ~12 GB |

Same one line of code (`AutoModel.from_pretrained(...)`

) for every row in that table — no per-model special-casing.

## Quickstart [#](#quickstart)

### Install [#](#install)

```
pip install airllm
```

### Run inference [#](#run-inference)

``` python
from airllm import AutoModel

MAX_LENGTH = 128
# just pass a hugging face repo id — works with almost any popular model:
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")

# go bigger with the exact same one line:
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B")     # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3")  # 671B, runs in ~12GB

input_text = ['What is the capital of United States?']

input_tokens = model.tokenizer(input_text,
    return_tensors="pt",
    return_attention_mask=False,
    truncation=True,
    max_length=MAX_LENGTH,
    padding=False)

generation_output = model.generate(
    input_tokens['input_ids'].cuda(),
    max_new_tokens=20,
    use_cache=True,
    return_dict_in_generate=True)

output = model.tokenizer.decode(generation_output.sequences[0])
print(output)
```

**Disk space matters here**: on first run, the original model is decomposed and saved layer-wise, so you need enough free disk space in your Hugging Face cache directory for both the original download and the split version — this is the single most common source of errors (see FAQ below).

## Optional: Model Compression for 3x Faster Inference [#](#optional-model-compression-for-3x-faster-inference)

*Speed improvement from optional compression — from github.com/lyogavin/airllm*

Separately from the core no-quantization layer-streaming approach, AirLLM offers **opt-in** block-wise quantization compression for up to **3x faster inference** with what the maintainers describe as “almost ignorable accuracy loss” (methodology reference: [arXiv:2212.09720](https://arxiv.org/abs/2212.09720)).

```
pip install -U bitsandbytes
pip install -U airllm
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
                     compression='4bit'  # or '8bit'
                    )
```

**Why this differs from standard quantization**: typical quantization needs both weights *and* activations quantized to meaningfully speed things up, which makes accuracy harder to preserve. Since AirLLM’s bottleneck is disk loading rather than compute, it only needs the *loading size* reduced — so it quantizes weights alone, which is easier to keep accurate.

## Configuration Options [#](#configuration-options)

| Option | What it does |
|---|---|
`compression` | `'4bit'` / `'8bit'` for block-wise quantization, or `None` (default) for no compression |
`profiling_mode` | `True` to print time breakdowns, `False` by default |
`layer_shards_saving_path` | Alternate path to save the split model, if you don’t want it in the default cache |
`hf_token` | Required for gated models like `meta-llama/Llama-2-7b-hf` |
`prefetching` | Overlaps model loading and compute; on by default (currently `AirLLMLlama2` only) |
`delete_original` | Deletes the original downloaded HF model after splitting, keeping only the transformed copy, to save disk space |

## Running on macOS [#](#running-on-macos)

```
1. Install mlx and torch
2. Only Apple Silicon is supported (not Intel Macs)
3. Run the same AutoModel.from_pretrained(...) code as on Linux
```

The README notes you may need a native (non-Rosetta) Python installation for this to work correctly — see the linked [macOS example notebook](https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_on_macos.ipynb) for the full setup.

## Beyond Llama: Broad Model Family Support [#](#beyond-llama-broad-model-family-support)

AirLLM works out of the box with most popular open models — pass the Hugging Face repo ID and `AutoModel`

handles the rest:

```
# ChatGLM
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")
# Qwen
model = AutoModel.from_pretrained("Qwen/Qwen-7B")
# Baichuan / InternLM / Mistral
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
```

Documented families: **Llama** (2/3/3.1/3.3/4) · **Qwen** (1/2/2.5/3, including MoE + FP8) · **DeepSeek** (V2/V3/R1) · **Mistral & Mixtral** · **Phi** · **Gemma** · **ChatGLM** · **Baichuan** · **InternLM** · **Yi** — plus, per the July 2026 update, **Kimi K3**, which additionally requires `pip install compressed-tensors flash-attn`

, a CUDA 12 torch build (no prebuilt flash-attn wheel exists yet for CUDA 13), and `transformers`

4.56.x specifically (its remote code doesn’t load on the 5.x line).

## AirLLM vs. Standard Approaches to Running Huge Models [#](#airllm-vs-standard-approaches-to-running-huge-models)

| Approach | VRAM strategy | Changes model weights? | Setup complexity |
|---|---|---|---|
AirLLM (default) | Stream one layer/expert at a time | No | Low — one `AutoModel.from_pretrained()` call |
4-bit/8-bit quantization (general) | Load whole compressed model | Yes | Low–medium, quantize weights + activations |
Model distillation | Train a smaller model | Yes — different model entirely | High — requires a training run |
Multi-GPU sharding (e.g. vLLM tensor parallel) | Split model across several GPUs | No | Medium–high — needs multiple GPUs |
AirLLM + optional compression | Stream one layer, quantized | Yes (weights only, opt-in) | Low, up to 3x faster than default |

The distinguishing case for AirLLM specifically: you have exactly **one** GPU with limited VRAM and want to run a model that wouldn’t otherwise fit at all, without standing up multi-GPU infrastructure or accepting a quantized/distilled model as your baseline.

## Use Cases [#](#use-cases)

### 1. Running SOTA-Scale Models on a Single Consumer or Hobbyist GPU [#](#1-running-sota-scale-models-on-a-single-consumer-or-hobbyist-gpu)

The headline case: DeepSeek-V3 (671B) on ~12GB, or Llama 3.1 405B on 8GB — hardware most individual developers or small teams can actually own, not a multi-GPU server rack.

### 2. Evaluating a Huge Model Before Committing to Serving Infrastructure [#](#2-evaluating-a-huge-model-before-committing-to-serving-infrastructure)

Since it’s a one-line `AutoModel.from_pretrained()`

swap, AirLLM is a fast way to sanity-check a large model’s output quality on your own hardware before investing in the multi-GPU setup a production deployment would need.

### 3. MoE Models on Minimal Hardware [#](#3-moe-models-on-minimal-hardware)

Kimi K3 (2.8T params) running in under 4GB is only possible because of per-expert streaming — for large MoE models specifically, AirLLM’s approach scales even better than for dense models of similar size.

### 4. Apple Silicon Local Inference [#](#4-apple-silicon-local-inference)

The macOS path (via `mlx`

) extends the same low-VRAM approach to Apple Silicon Macs, for developers who don’t have a dedicated NVIDIA GPU at all.

## Related Repositories [#](#related-repositories)

| Repository | Purpose |
|---|---|
|

[bitsandbytes](https://github.com/TimDettmers/bitsandbytes)## Related Articles [#](#related-articles)

[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)— for comparing AirLLM’s single-tiny-GPU niche against general self-hosting options[Ollama: Run LLMs Locally with One Command](https://dibi8.com/resources/llm-frameworks/ollama/)— the simpler default when your model already fits in VRAM

## Conclusion [#](#conclusion)

**AirLLM** solves a specific, well-defined problem: running a model that’s simply too large for your GPU’s VRAM, without touching the model’s weights by default and without needing multiple GPUs. The layer-and-expert streaming approach trades inference speed for the ability to run models — up to 671B dense-equivalent and beyond for MoE — on hardware that couldn’t otherwise load them at all, with an optional 3x compression mode when speed matters more than exact full-precision output. Nearly three years of continuous version history, current through July 2026’s Kimi K3 support, backs it as a maintained tool rather than an abandoned proof of concept.

**Best for**: Developers with exactly one GPU and limited VRAM who want to run or evaluate a large open model without standing up multi-GPU infrastructure or committing to a quantized/distilled variant.

**GitHub**: [https://github.com/lyogavin/airllm](https://github.com/lyogavin/airllm)

*Last updated: 2026-08-02*