{"slug": "airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required", "title": "AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required", "summary": "AirLLM, an Apache-2.0 Python library, enables running large language models like Llama 3.1 405B on 8GB of VRAM and DeepSeek-V3 671B on ~12GB without quantization, by streaming one layer or MoE expert onto the GPU at a time. The project, with 25,300+ GitHub stars and continuous updates since November 2023, also supports Kimi K3 (2.8T parameters) on under 4GB as of July 2026.", "body_md": "# AirLLM: Run a 405B Model on 8GB of VRAM, No Quantization Required\n\nAirLLM is an Apache-2.0 Python library that runs 70B+ LLMs — including Llama 3.1 405B on 8GB and DeepSeek-V3 671B on ~12GB — by streaming one layer (or one MoE expert) onto the GPU at a time instead of loading the whole model, no quantization, distillation, or pruning required.\n\n- ⭐ 25300\n- Python\n- Apache-2.0\n- Updated 2026-08-02\n\n[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/) •\n[Ollama: Run LLMs Locally with One Command](https://dibi8.com/resources/llm-frameworks/ollama/)\n\n*Project logo — from github.com/lyogavin/airllm*\n\n## What Is AirLLM? [#](#what-is-airllm)\n\n**AirLLM** dramatically reduces LLM inference memory usage, letting a 70B model run on a single 4GB GPU — without quantization, distillation, or pruning. Per the current README, that same approach scales further: **Llama 3.1 405B on 8GB**, **DeepSeek-V3 (671B) on ~12GB**, and, as of July 2026, **Kimi K3 (2.8T parameters, the largest open-source model released to date) on under 4GB**, because sparse MoE models let AirLLM stream one expert at a time instead of a whole dense layer.\n\n🔗 **GitHub**: [https://github.com/lyogavin/airllm](https://github.com/lyogavin/airllm)\n📦 **PyPI**: [https://pypi.org/project/airllm](https://pypi.org/project/airllm/)\n\nApache-2.0 licensed, at **25,300+ GitHub stars**, with a version history running continuously from its **November 2023** initial release through a **July 2026** commit adding Kimi K3 support — this is a mature project with a genuine multi-year track record, not a recent trend-chaser.\n\n## The Trick: One Layer on the GPU at a Time [#](#the-trick-one-layer-on-the-gpu-at-a-time)\n\nMost “run a huge model on small hardware” techniques (quantization, pruning, distillation) change the model itself to make it smaller. AirLLM instead changes *how much of the model is ever loaded at once*: it decomposes the model layer-by-layer, and during inference only ever keeps **one layer on the GPU** — for MoE models, one **routed expert** — streaming the rest from disk/CPU as needed.\n\nThe practical consequence: your required VRAM is driven by the size of the model’s *largest single layer*, not its total parameter count.\n\n| Model | Size | GPU VRAM |\n|---|---|---|\n| Qwen3 / Mistral / Phi (~8B) | 8B | ~1–2 GB |\n| Qwen3-30B / Mixtral (MoE) | 30–47B | ~1–3 GB |\n| Qwen3-235B (MoE) | 235B | ~3 GB |\n| Llama 3.x 70B (full precision) | 70B | ~4 GB |\n| Llama 3.1 405B | 405B | ~8 GB |\n| DeepSeek-V3 | 671B | ~12 GB |\n\nSame one line of code (`AutoModel.from_pretrained(...)`\n\n) for every row in that table — no per-model special-casing.\n\n## Quickstart [#](#quickstart)\n\n### Install [#](#install)\n\n```\npip install airllm\n```\n\n### Run inference [#](#run-inference)\n\n``` python\nfrom airllm import AutoModel\n\nMAX_LENGTH = 128\n# just pass a hugging face repo id — works with almost any popular model:\nmodel = AutoModel.from_pretrained(\"Qwen/Qwen3-32B\")\n\n# go bigger with the exact same one line:\n#model = AutoModel.from_pretrained(\"Qwen/Qwen3-235B-A22B\")     # 235B, runs in ~3GB\n#model = AutoModel.from_pretrained(\"deepseek-ai/DeepSeek-V3\")  # 671B, runs in ~12GB\n\ninput_text = ['What is the capital of United States?']\n\ninput_tokens = model.tokenizer(input_text,\n    return_tensors=\"pt\",\n    return_attention_mask=False,\n    truncation=True,\n    max_length=MAX_LENGTH,\n    padding=False)\n\ngeneration_output = model.generate(\n    input_tokens['input_ids'].cuda(),\n    max_new_tokens=20,\n    use_cache=True,\n    return_dict_in_generate=True)\n\noutput = model.tokenizer.decode(generation_output.sequences[0])\nprint(output)\n```\n\n**Disk space matters here**: on first run, the original model is decomposed and saved layer-wise, so you need enough free disk space in your Hugging Face cache directory for both the original download and the split version — this is the single most common source of errors (see FAQ below).\n\n## Optional: Model Compression for 3x Faster Inference [#](#optional-model-compression-for-3x-faster-inference)\n\n*Speed improvement from optional compression — from github.com/lyogavin/airllm*\n\nSeparately from the core no-quantization layer-streaming approach, AirLLM offers **opt-in** block-wise quantization compression for up to **3x faster inference** with what the maintainers describe as “almost ignorable accuracy loss” (methodology reference: [arXiv:2212.09720](https://arxiv.org/abs/2212.09720)).\n\n```\npip install -U bitsandbytes\npip install -U airllm\nmodel = AutoModel.from_pretrained(\"garage-bAInd/Platypus2-70B-instruct\",\n                     compression='4bit'  # or '8bit'\n                    )\n```\n\n**Why this differs from standard quantization**: typical quantization needs both weights *and* activations quantized to meaningfully speed things up, which makes accuracy harder to preserve. Since AirLLM’s bottleneck is disk loading rather than compute, it only needs the *loading size* reduced — so it quantizes weights alone, which is easier to keep accurate.\n\n## Configuration Options [#](#configuration-options)\n\n| Option | What it does |\n|---|---|\n`compression` | `'4bit'` / `'8bit'` for block-wise quantization, or `None` (default) for no compression |\n`profiling_mode` | `True` to print time breakdowns, `False` by default |\n`layer_shards_saving_path` | Alternate path to save the split model, if you don’t want it in the default cache |\n`hf_token` | Required for gated models like `meta-llama/Llama-2-7b-hf` |\n`prefetching` | Overlaps model loading and compute; on by default (currently `AirLLMLlama2` only) |\n`delete_original` | Deletes the original downloaded HF model after splitting, keeping only the transformed copy, to save disk space |\n\n## Running on macOS [#](#running-on-macos)\n\n```\n1. Install mlx and torch\n2. Only Apple Silicon is supported (not Intel Macs)\n3. Run the same AutoModel.from_pretrained(...) code as on Linux\n```\n\nThe README notes you may need a native (non-Rosetta) Python installation for this to work correctly — see the linked [macOS example notebook](https://github.com/lyogavin/airllm/blob/main/air_llm/examples/run_on_macos.ipynb) for the full setup.\n\n## Beyond Llama: Broad Model Family Support [#](#beyond-llama-broad-model-family-support)\n\nAirLLM works out of the box with most popular open models — pass the Hugging Face repo ID and `AutoModel`\n\nhandles the rest:\n\n```\n# ChatGLM\nmodel = AutoModel.from_pretrained(\"THUDM/chatglm3-6b-base\")\n# Qwen\nmodel = AutoModel.from_pretrained(\"Qwen/Qwen-7B\")\n# Baichuan / InternLM / Mistral\nmodel = AutoModel.from_pretrained(\"baichuan-inc/Baichuan2-7B-Base\")\n#model = AutoModel.from_pretrained(\"internlm/internlm-20b\")\n#model = AutoModel.from_pretrained(\"mistralai/Mistral-7B-Instruct-v0.1\")\n```\n\nDocumented families: **Llama** (2/3/3.1/3.3/4) · **Qwen** (1/2/2.5/3, including MoE + FP8) · **DeepSeek** (V2/V3/R1) · **Mistral & Mixtral** · **Phi** · **Gemma** · **ChatGLM** · **Baichuan** · **InternLM** · **Yi** — plus, per the July 2026 update, **Kimi K3**, which additionally requires `pip install compressed-tensors flash-attn`\n\n, a CUDA 12 torch build (no prebuilt flash-attn wheel exists yet for CUDA 13), and `transformers`\n\n4.56.x specifically (its remote code doesn’t load on the 5.x line).\n\n## AirLLM vs. Standard Approaches to Running Huge Models [#](#airllm-vs-standard-approaches-to-running-huge-models)\n\n| Approach | VRAM strategy | Changes model weights? | Setup complexity |\n|---|---|---|---|\nAirLLM (default) | Stream one layer/expert at a time | No | Low — one `AutoModel.from_pretrained()` call |\n4-bit/8-bit quantization (general) | Load whole compressed model | Yes | Low–medium, quantize weights + activations |\nModel distillation | Train a smaller model | Yes — different model entirely | High — requires a training run |\nMulti-GPU sharding (e.g. vLLM tensor parallel) | Split model across several GPUs | No | Medium–high — needs multiple GPUs |\nAirLLM + optional compression | Stream one layer, quantized | Yes (weights only, opt-in) | Low, up to 3x faster than default |\n\nThe distinguishing case for AirLLM specifically: you have exactly **one** GPU with limited VRAM and want to run a model that wouldn’t otherwise fit at all, without standing up multi-GPU infrastructure or accepting a quantized/distilled model as your baseline.\n\n## Use Cases [#](#use-cases)\n\n### 1. Running SOTA-Scale Models on a Single Consumer or Hobbyist GPU [#](#1-running-sota-scale-models-on-a-single-consumer-or-hobbyist-gpu)\n\nThe headline case: DeepSeek-V3 (671B) on ~12GB, or Llama 3.1 405B on 8GB — hardware most individual developers or small teams can actually own, not a multi-GPU server rack.\n\n### 2. Evaluating a Huge Model Before Committing to Serving Infrastructure [#](#2-evaluating-a-huge-model-before-committing-to-serving-infrastructure)\n\nSince it’s a one-line `AutoModel.from_pretrained()`\n\nswap, AirLLM is a fast way to sanity-check a large model’s output quality on your own hardware before investing in the multi-GPU setup a production deployment would need.\n\n### 3. MoE Models on Minimal Hardware [#](#3-moe-models-on-minimal-hardware)\n\nKimi K3 (2.8T params) running in under 4GB is only possible because of per-expert streaming — for large MoE models specifically, AirLLM’s approach scales even better than for dense models of similar size.\n\n### 4. Apple Silicon Local Inference [#](#4-apple-silicon-local-inference)\n\nThe macOS path (via `mlx`\n\n) extends the same low-VRAM approach to Apple Silicon Macs, for developers who don’t have a dedicated NVIDIA GPU at all.\n\n## Related Repositories [#](#related-repositories)\n\n| Repository | Purpose |\n|---|---|\n|\n\n[bitsandbytes](https://github.com/TimDettmers/bitsandbytes)## Related Articles [#](#related-articles)\n\n[Self-Hosted LLM 2026: Ollama vs vLLM vs LocalAI](https://dibi8.com/resources/llm-frameworks/self-hosted-llm-2026-ollama-vllm-localai/)— for comparing AirLLM’s single-tiny-GPU niche against general self-hosting options[Ollama: Run LLMs Locally with One Command](https://dibi8.com/resources/llm-frameworks/ollama/)— the simpler default when your model already fits in VRAM\n\n## Conclusion [#](#conclusion)\n\n**AirLLM** solves a specific, well-defined problem: running a model that’s simply too large for your GPU’s VRAM, without touching the model’s weights by default and without needing multiple GPUs. The layer-and-expert streaming approach trades inference speed for the ability to run models — up to 671B dense-equivalent and beyond for MoE — on hardware that couldn’t otherwise load them at all, with an optional 3x compression mode when speed matters more than exact full-precision output. Nearly three years of continuous version history, current through July 2026’s Kimi K3 support, backs it as a maintained tool rather than an abandoned proof of concept.\n\n**Best for**: Developers with exactly one GPU and limited VRAM who want to run or evaluate a large open model without standing up multi-GPU infrastructure or committing to a quantized/distilled variant.\n\n**GitHub**: [https://github.com/lyogavin/airllm](https://github.com/lyogavin/airllm)\n\n*Last updated: 2026-08-02*", "url": "https://wpnews.pro/news/airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required", "canonical_source": "https://dibi8.com/resources/llm-frameworks/airllm-run-huge-llms-tiny-gpu-2026/", "published_at": "2026-08-02 13:15:00+00:00", "updated_at": "2026-08-02 15:05:52.249249+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["AirLLM", "Llama 3.1 405B", "DeepSeek-V3", "Kimi K3", "Apache-2.0", "GitHub", "PyPI"], "alternates": {"html": "https://wpnews.pro/news/airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required", "markdown": "https://wpnews.pro/news/airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required.md", "text": "https://wpnews.pro/news/airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required.txt", "jsonld": "https://wpnews.pro/news/airllm-run-a-405b-model-on-8gb-of-vram-no-quantization-required.jsonld"}}