AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required

AirLLM, an Apache-2.0 Python library, enables running large language models like Llama 3.1 405B on 8GB of VRAM and DeepSeek-V3 671B on ~12GB without quantization, by streaming one layer or MoE expert onto the GPU at a time. The project, with 25,300+ GitHub stars and continuous updates since November 2023, also supports Kimi K3 (2.8T parameters) on under 4GB as of July 2026.

AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required AirLLM is an Apache-2.0 Python library that runs 70B+ LLMs — including Llama 3.1 405B on 8GB and DeepSeek-V3 671B on ~12GB — by streaming one layer or one MoE expert onto the GPU at a time instead of loading the whole model, no quantization, distillation, or pruning required. - ⭐ 25300 - Python - Apache-2.0 - Updated 2026-08-02 Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/ • Ollama: Run LLMs Locally with One Command https://dibi8.com/resources/llm-frameworks/ollama/ Project logo — from github.com/lyogavin/airllm What Is AirLLM? what-is-airllm AirLLM dramatically reduces LLM inference memory usage, letting a 70B model run on a single 4GB GPU — without quantization, distillation, or pruning. Per the current README, that same approach scales further: Llama 3.1 405B on 8GB , DeepSeek-V3 671B on ~12GB , and, as of July 2026, Kimi K3 2.8T parameters, the largest open-source model released to date on under 4GB , because sparse MoE models let AirLLM stream one expert at a time instead of a whole dense layer. 🔗 GitHub : https://github.com/lyogavin/airllm https://github.com/lyogavin/airllm 📦 PyPI : https://pypi.org/project/airllm https://pypi.org/project/airllm/ Apache-2.0 licensed, at 25,300+ GitHub stars , with a version history running continuously from its November 2023 initial release through a July 2026 commit adding Kimi K3 support — this is a mature project with a genuine multi-year track record, not a recent trend-chaser. The Trick: One Layer on the GPU at a Time the-trick-one-layer-on-the-gpu-at-a-time Most “run a huge model on small hardware” techniques quantization, pruning, distillation change the model itself to make it smaller. AirLLM instead changes how much of the model is ever loaded at once : it decomposes the model layer-by-layer, and during inference only ever keeps one layer on the GPU — for MoE models, one routed expert — streaming the rest from disk/CPU as needed. The practical consequence: your required VRAM is driven by the size of the model’s largest single layer , not its total parameter count. | Model | Size | GPU VRAM | |---|---|---| | Qwen3 / Mistral / Phi ~8B | 8B | ~1–2 GB | | Qwen3-30B / Mixtral MoE | 30–47B | ~1–3 GB | | Qwen3-235B MoE | 235B | ~3 GB | | Llama 3.x 70B full precision | 70B | ~4 GB | | Llama 3.1 405B | 405B | ~8 GB | | DeepSeek-V3 | 671B | ~12 GB | Same one line of code AutoModel.from pretrained ... for every row in that table — no per-model special-casing. Quickstart quickstart Install install pip install airllm Run inference run-inference python from airllm import AutoModel MAX LENGTH = 128 just pass a hugging face repo id — works with almost any popular model: model = AutoModel.from pretrained "Qwen/Qwen3-32B" go bigger with the exact same one line: model = AutoModel.from pretrained "Qwen/Qwen3-235B-A22B" 235B, runs in ~3GB model = AutoModel.from pretrained "deepseek-ai/DeepSeek-V3" 671B, runs in ~12GB input text = 'What is the capital of United States?' input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH, padding=False generation output = model.generate input tokens 'input ids' .cuda , max new tokens=20, use cache=True, return dict in generate=True output = model.tokenizer.decode generation output.sequences 0 print output Disk space matters here : on first run, the original model is decomposed and saved layer-wise, so you need enough free disk space in your Hugging Face cache directory for both the original download and the split version — this is the single most common source of errors see FAQ below . Optional: Model Compression for 3x Faster Inference optional-model-compression-for-3x-faster-inference Speed improvement from optional compression — from github.com/lyogavin/airllm Separately from the core no-quantization layer-streaming approach, AirLLM offers opt-in block-wise quantization compression for up to 3x faster inference with what the maintainers describe as “almost ignorable accuracy loss” methodology reference: arXiv:2212.09720 https://arxiv.org/abs/2212.09720 . pip install -U bitsandbytes pip install -U airllm model = AutoModel.from pretrained "garage-bAInd/Platypus2-70B-instruct", compression='4bit' or '8bit' Why this differs from standard quantization : typical quantization needs both weights and activations quantized to meaningfully speed things up, which makes accuracy harder to preserve. Since AirLLM’s bottleneck is disk loading rather than compute, it only needs the loading size reduced — so it quantizes weights alone, which is easier to keep accurate. Configuration Options configuration-options | Option | What it does | |---|---| compression | '4bit' / '8bit' for block-wise quantization, or None default for no compression | profiling mode | True to print time breakdowns, False by default | layer shards saving path | Alternate path to save the split model, if you don’t want it in the default cache | hf token | Required for gated models like meta-llama/Llama-2-7b-hf | prefetching | Overlaps model loading and compute; on by default currently AirLLMLlama2 only | delete original | Deletes the original downloaded HF model after splitting, keeping only the transformed copy, to save disk space | Running on macOS running-on-macos 1. Install mlx and torch 2. Only Apple Silicon is supported not Intel Macs 3. Run the same AutoModel.from pretrained ... code as on Linux The README notes you may need a native non-Rosetta Python installation for this to work correctly — see the linked macOS example notebook https://github.com/lyogavin/airllm/blob/main/air llm/examples/run on macos.ipynb for the full setup. Beyond Llama: Broad Model Family Support beyond-llama-broad-model-family-support AirLLM works out of the box with most popular open models — pass the Hugging Face repo ID and AutoModel handles the rest: ChatGLM model = AutoModel.from pretrained "THUDM/chatglm3-6b-base" Qwen model = AutoModel.from pretrained "Qwen/Qwen-7B" Baichuan / InternLM / Mistral model = AutoModel.from pretrained "baichuan-inc/Baichuan2-7B-Base" model = AutoModel.from pretrained "internlm/internlm-20b" model = AutoModel.from pretrained "mistralai/Mistral-7B-Instruct-v0.1" Documented families: Llama 2/3/3.1/3.3/4 · Qwen 1/2/2.5/3, including MoE + FP8 · DeepSeek V2/V3/R1 · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi — plus, per the July 2026 update, Kimi K3 , which additionally requires pip install compressed-tensors flash-attn , a CUDA 12 torch build no prebuilt flash-attn wheel exists yet for CUDA 13 , and transformers 4.56.x specifically its remote code doesn’t load on the 5.x line . AirLLM vs. Standard Approaches to Running Huge Models airllm-vs-standard-approaches-to-running-huge-models | Approach | VRAM strategy | Changes model weights? | Setup complexity | |---|---|---|---| AirLLM default | Stream one layer/expert at a time | No | Low — one AutoModel.from pretrained call | 4-bit/8-bit quantization general | Load whole compressed model | Yes | Low–medium, quantize weights + activations | Model distillation | Train a smaller model | Yes — different model entirely | High — requires a training run | Multi-GPU sharding e.g. vLLM tensor parallel | Split model across several GPUs | No | Medium–high — needs multiple GPUs | AirLLM + optional compression | Stream one layer, quantized | Yes weights only, opt-in | Low, up to 3x faster than default | The distinguishing case for AirLLM specifically: you have exactly one GPU with limited VRAM and want to run a model that wouldn’t otherwise fit at all, without standing up multi-GPU infrastructure or accepting a quantized/distilled model as your baseline. Use Cases use-cases 1. Running SOTA-Scale Models on a Single Consumer or Hobbyist GPU 1-running-sota-scale-models-on-a-single-consumer-or-hobbyist-gpu The headline case: DeepSeek-V3 671B on ~12GB, or Llama 3.1 405B on 8GB — hardware most individual developers or small teams can actually own, not a multi-GPU server rack. 2. Evaluating a Huge Model Before Committing to Serving Infrastructure 2-evaluating-a-huge-model-before-committing-to-serving-infrastructure Since it’s a one-line AutoModel.from pretrained swap, AirLLM is a fast way to sanity-check a large model’s output quality on your own hardware before investing in the multi-GPU setup a production deployment would need. 3. MoE Models on Minimal Hardware 3-moe-models-on-minimal-hardware Kimi K3 2.8T params running in under 4GB is only possible because of per-expert streaming — for large MoE models specifically, AirLLM’s approach scales even better than for dense models of similar size. 4. Apple Silicon Local Inference 4-apple-silicon-local-inference The macOS path via mlx extends the same low-VRAM approach to Apple Silicon Macs, for developers who don’t have a dedicated NVIDIA GPU at all. Related Repositories related-repositories | Repository | Purpose | |---|---| | bitsandbytes https://github.com/TimDettmers/bitsandbytes Related Articles related-articles Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/ — for comparing AirLLM’s single-tiny-GPU niche against general self-hosting options Ollama: Run LLMs Locally with One Command https://dibi8.com/resources/llm-frameworks/ollama/ — the simpler default when your model already fits in VRAM Conclusion conclusion AirLLM solves a specific, well-defined problem: running a model that’s simply too large for your GPU’s VRAM, without touching the model’s weights by default and without needing multiple GPUs. The layer-and-expert streaming approach trades inference speed for the ability to run models — up to 671B dense-equivalent and beyond for MoE — on hardware that couldn’t otherwise load them at all, with an optional 3x compression mode when speed matters more than exact full-precision output. Nearly three years of continuous version history, current through July 2026’s Kimi K3 support, backs it as a maintained tool rather than an abandoned proof of concept. Best for : Developers with exactly one GPU and limited VRAM who want to run or evaluate a large open model without standing up multi-GPU infrastructure or committing to a quantized/distilled variant. GitHub : https://github.com/lyogavin/airllm https://github.com/lyogavin/airllm Last updated: 2026-08-02