AirLLM is an Apache-2.0 Python library that runs 70B+ LLMs — including Llama 3.1 405B on 8GB and DeepSeek-V3 671B on ~12GB — by streaming one layer (or one MoE expert) onto the GPU at a time instead of the whole model, no quantization, distillation, or pruning required.
- ⭐ 25300
- Python
- Apache-2.0
- Updated 2026-08-02
Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI • Ollama: Run LLMs Locally with One Command
Project logo — from github.com/lyogavin/airllm
What Is AirLLM? # #
AirLLM dramatically reduces LLM inference memory usage, letting a 70B model run on a single 4GB GPU — without quantization, distillation, or pruning. Per the current README, that same approach scales further: Llama 3.1 405B on 8GB, DeepSeek-V3 (671B) on ~12GB, and, as of July 2026, Kimi K3 (2.8T parameters, the largest open-source model released to date) on under 4GB, because sparse MoE models let AirLLM stream one expert at a time instead of a whole dense layer.
🔗 GitHub: https://github.com/lyogavin/airllm 📦 PyPI: https://pypi.org/project/airllm
Apache-2.0 licensed, at 25,300+ GitHub stars, with a version history running continuously from its November 2023 initial release through a July 2026 commit adding Kimi K3 support — this is a mature project with a genuine multi-year track record, not a recent trend-chaser.
The Trick: One Layer on the GPU at a Time # #
Most “run a huge model on small hardware” techniques (quantization, pruning, distillation) change the model itself to make it smaller. AirLLM instead changes how much of the model is ever loaded at once: it decomposes the model layer-by-layer, and during inference only ever keeps one layer on the GPU — for MoE models, one routed expert — streaming the rest from disk/CPU as needed.
The practical consequence: your required VRAM is driven by the size of the model’s largest single layer, not its total parameter count.
| Model | Size | GPU VRAM |
|---|---|---|
| Qwen3 / Mistral / Phi (~8B) | 8B | ~1–2 GB |
| Qwen3-30B / Mixtral (MoE) | 30–47B | ~1–3 GB |
| Qwen3-235B (MoE) | 235B | ~3 GB |
| Llama 3.x 70B (full precision) | 70B | ~4 GB |
| Llama 3.1 405B | 405B | ~8 GB |
| DeepSeek-V3 | 671B | ~12 GB |
Same one line of code (AutoModel.from_pretrained(...)
) for every row in that table — no per-model special-casing.
Quickstart # #
Install #
pip install airllm
Run inference #
from airllm import AutoModel
MAX_LENGTH = 128
model = AutoModel.from_pretrained("Qwen/Qwen3-32B")
#model = AutoModel.from_pretrained("Qwen/Qwen3-235B-A22B") # 235B, runs in ~3GB
#model = AutoModel.from_pretrained("deepseek-ai/DeepSeek-V3") # 671B, runs in ~12GB
input_text = ['What is the capital of United States?']
input_tokens = model.tokenizer(input_text,
return_tensors="pt",
return_attention_mask=False,
truncation=True,
max_length=MAX_LENGTH,
padding=False)
generation_output = model.generate(
input_tokens['input_ids'].cuda(),
max_new_tokens=20,
use_cache=True,
return_dict_in_generate=True)
output = model.tokenizer.decode(generation_output.sequences[0])
print(output)
Disk space matters here: on first run, the original model is decomposed and saved layer-wise, so you need enough free disk space in your Hugging Face cache directory for both the original download and the split version — this is the single most common source of errors (see FAQ below).
Optional: Model Compression for 3x Faster Inference # #
Speed improvement from optional compression — from github.com/lyogavin/airllm
Separately from the core no-quantization layer-streaming approach, AirLLM offers opt-in block-wise quantization compression for up to 3x faster inference with what the maintainers describe as “almost ignorable accuracy loss” (methodology reference: arXiv:2212.09720).
pip install -U bitsandbytes
pip install -U airllm
model = AutoModel.from_pretrained("garage-bAInd/Platypus2-70B-instruct",
compression='4bit' # or '8bit'
)
Why this differs from standard quantization: typical quantization needs both weights and activations quantized to meaningfully speed things up, which makes accuracy harder to preserve. Since AirLLM’s bottleneck is disk rather than compute, it only needs the * size* reduced — so it quantizes weights alone, which is easier to keep accurate.
Configuration Options # #
| Option | What it does |
|---|---|
compression |
'4bit' / '8bit' for block-wise quantization, or None (default) for no compression |
profiling_mode |
True to print time breakdowns, False by default |
layer_shards_saving_path |
Alternate path to save the split model, if you don’t want it in the default cache |
hf_token |
Required for gated models like meta-llama/Llama-2-7b-hf |
prefetching |
Overlaps model and compute; on by default (currently AirLLMLlama2 only) |
delete_original |
Deletes the original downloaded HF model after splitting, keeping only the transformed copy, to save disk space |
Running on macOS # #
1. Install mlx and torch
2. Only Apple Silicon is supported (not Intel Macs)
3. Run the same AutoModel.from_pretrained(...) code as on Linux
The README notes you may need a native (non-Rosetta) Python installation for this to work correctly — see the linked macOS example notebook for the full setup.
Beyond Llama: Broad Model Family Support # #
AirLLM works out of the box with most popular open models — pass the Hugging Face repo ID and AutoModel
handles the rest:
model = AutoModel.from_pretrained("THUDM/chatglm3-6b-base")
model = AutoModel.from_pretrained("Qwen/Qwen-7B")
model = AutoModel.from_pretrained("baichuan-inc/Baichuan2-7B-Base")
#model = AutoModel.from_pretrained("internlm/internlm-20b")
#model = AutoModel.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
Documented families: Llama (2/3/3.1/3.3/4) · Qwen (1/2/2.5/3, including MoE + FP8) · DeepSeek (V2/V3/R1) · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi — plus, per the July 2026 update, Kimi K3, which additionally requires pip install compressed-tensors flash-attn
, a CUDA 12 torch build (no prebuilt flash-attn wheel exists yet for CUDA 13), and transformers
4.56.x specifically (its remote code doesn’t load on the 5.x line).
AirLLM vs. Standard Approaches to Running Huge Models # #
| Approach | VRAM strategy | Changes model weights? | Setup complexity |
|---|---|---|---|
| AirLLM (default) | Stream one layer/expert at a time | No | Low — one AutoModel.from_pretrained() call |
| 4-bit/8-bit quantization (general) | Load whole compressed model | Yes | Low–medium, quantize weights + activations |
| Model distillation | Train a smaller model | Yes — different model entirely | High — requires a training run |
| Multi-GPU sharding (e.g. vLLM tensor parallel) | Split model across several GPUs | No | Medium–high — needs multiple GPUs |
| AirLLM + optional compression | Stream one layer, quantized | Yes (weights only, opt-in) | Low, up to 3x faster than default |
The distinguishing case for AirLLM specifically: you have exactly one GPU with limited VRAM and want to run a model that wouldn’t otherwise fit at all, without standing up multi-GPU infrastructure or accepting a quantized/distilled model as your baseline.
Use Cases # #
1. Running SOTA-Scale Models on a Single Consumer or Hobbyist GPU #
The headline case: DeepSeek-V3 (671B) on ~12GB, or Llama 3.1 405B on 8GB — hardware most individual developers or small teams can actually own, not a multi-GPU server rack.
2. Evaluating a Huge Model Before Committing to Serving Infrastructure #
Since it’s a one-line AutoModel.from_pretrained()
swap, AirLLM is a fast way to sanity-check a large model’s output quality on your own hardware before investing in the multi-GPU setup a production deployment would need.
3. MoE Models on Minimal Hardware #
Kimi K3 (2.8T params) running in under 4GB is only possible because of per-expert streaming — for large MoE models specifically, AirLLM’s approach scales even better than for dense models of similar size.
4. Apple Silicon Local Inference #
The macOS path (via mlx
) extends the same low-VRAM approach to Apple Silicon Macs, for developers who don’t have a dedicated NVIDIA GPU at all.
Related Repositories # #
| Repository | Purpose |
|---|---|
bitsandbytes## Related Articles #
Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI— for comparing AirLLM’s single-tiny-GPU niche against general self-hosting optionsOllama: Run LLMs Locally with One Command— the simpler default when your model already fits in VRAM
Conclusion # #
AirLLM solves a specific, well-defined problem: running a model that’s simply too large for your GPU’s VRAM, without touching the model’s weights by default and without needing multiple GPUs. The layer-and-expert streaming approach trades inference speed for the ability to run models — up to 671B dense-equivalent and beyond for MoE — on hardware that couldn’t otherwise load them at all, with an optional 3x compression mode when speed matters more than exact full-precision output. Nearly three years of continuous version history, current through July 2026’s Kimi K3 support, backs it as a maintained tool rather than an abandoned proof of concept.
Best for: Developers with exactly one GPU and limited VRAM who want to run or evaluate a large open model without standing up multi-GPU infrastructure or committing to a quantized/distilled variant.
GitHub: https://github.com/lyogavin/airllm
Last updated: 2026-08-02