AirLLM: Inference 2.8T Kimi K3 on a single 4GB GPU

AirLLM, an open-source inference library, now supports running the 2.8T-parameter Kimi K3 model on a single 4GB GPU, using only 3.72GB of VRAM on an RTX 6000 Ada, by streaming one expert at a time for sparse MoE models. The library also enables running 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Qwen3-235B on ~3GB, without quantization, distillation, or pruning. Version 3.0 added FP8 model support and the latest models, with Kimi K3 support requiring specific dependencies like compressed-tensors, flash-attn, CUDA 12 torch, and transformers 4.56.x.

Quickstart quickstart | | configurations Configurations | macos MacOS | example-python-notebook Example notebooks FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. You can even run 405B Llama 3.1 on 8GB , DeepSeek-V3 671B on ~12GB , and Kimi K3 2.8T — the largest open-source model released to date — on under 4GB , because sparse MoE models stream one expert at a time rather than a whole layer. 2026/07 Kimi K3 2.8T support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: pip install compressed-tensors flash-attn its model code mandates flash attention regardless of what you request , a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformers 4.56.x, as its remote code does not load on 5.x. 2026/06 v3.0 : FP8 model support + the latest models. Run DeepSeek-V3 671B on ~12GB and Qwen3-235B on ~3GB , plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single AutoModel . 2024/08/20 v2.11.0: Support Qwen2.5 2024/08/18 v2.10.1 Support CPU inference. Support non sharded models. Thanks @NavodPeiris for the great work 2024/07/30 Support Llama3.1 405B example notebook https://colab.research.google.com/github/lyogavin/airllm/blob/main/air llm/examples/run llama3.1 405B.ipynb . Support 8bit/4bit quantization . 2024/04/20 AirLLM supports Llama3 natively already. Run Llama3 70B on 4GB single GPU. 2023/12/25 v2.8.2: Support MacOS running 70B large language models. 2023/12/20 v2.7: Support AirLLMMixtral. 2023/12/20 v2.6: Added AutoModel, automatically detect model type, no need to provide model class to initialize model. 2023/12/18 v2.5: added prefetching to overlap the model loading and compute. 10% speed improvement. 2023/12/03 added support of ChatGLM , QWen , Baichuan , Mistral , InternLM 2023/12/02 added support for safetensors. Now support all top 10 models in open llm leaderboard. 2023/12/01 airllm 2.0. Support compressions: 3x run time speed up 2023/11/20 airllm Initial version Quick start quickstart Model Compression model-compression---3x-inference-speed-up Configurations configurations Run on MacOS macos Example notebooks example-python-notebook Supported Models supported-models Acknowledgement acknowledgement FAQ faq First, install the airllm pip package. pip install airllm Then, initialize AirLLMLlama2, pass in the huggingface repo ID of the model being used, or the local path, and inference can be performed similar to a regular transformer model. You can also specify the path to save the splitted layered model through layer shards saving path when init AirLLMLlama2. python from airllm import AutoModel MAX LENGTH = 128 just pass a hugging face repo id — works with almost any popular model: model = AutoModel.from pretrained "Qwen/Qwen3-32B" go bigger with the exact same one line: model = AutoModel.from pretrained "Qwen/Qwen3-235B-A22B" 235B, runs in ~3GB model = AutoModel.from pretrained "deepseek-ai/DeepSeek-V3" 671B, runs in ~12GB or use a model's local path... model = AutoModel.from pretrained "/home/ubuntu/.cache/huggingface/hub/models--Qwen--Qwen3-32B/snapshots/..." input text = 'What is the capital of United States?', 'I like', input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH, padding=False generation output = model.generate input tokens 'input ids' .cuda , max new tokens=20, use cache=True, return dict in generate=True output = model.tokenizer.decode generation output.sequences 0 print output Note: During inference, the original model will first be decomposed and saved layer-wise. Please ensure there is sufficient disk space in the huggingface cache directory. We just added model compression based on block-wise quantization-based model compression. Which can further speed up the inference speed for up to 3x , with almost ignorable accuracy loss see more performance evaluation and why we use block-wise quantization in this paper https://arxiv.org/abs/2212.09720 - Step 1. make sure you have bitsandbytes https://github.com/TimDettmers/bitsandbytes installed by pip install -U bitsandbytes - Step 2. make sure airllm verion later than 2.0.0: pip install -U airllm - Step 3. when initialize the model, passing the argument compression '4bit' or '8bit' : model = AutoModel.from pretrained "garage-bAInd/Platypus2-70B-instruct", compression='4bit' specify '8bit' for 8-bit block-wise quantization Quantization normally needs to quantize both weights and activations to really speed things up. Which makes it harder to maintain accuracy and avoid the impact of outliers in all kinds of inputs. While in our case the bottleneck is mainly at the disk loading, we only need to make the model loading size smaller. So, we get to only quantize the weights' part, which is easier to ensure the accuracy. When initialize the model, we support the following configurations: compression : supported options: 4bit, 8bit for 4-bit or 8-bit block-wise quantization, or by default None for no compression profiling mode : supported options: True to output time consumptions or by default False layer shards saving path : optionally another path to save the splitted model hf token : huggingface token can be provided here if downloading gated models like: meta-llama/Llama-2-7b-hf prefetching : prefetching to overlap the model loading and compute. By default, turned on. For now, only AirLLMLlama2 supports this. delete original : if you don't have too much disk space, you can set delete original to true to delete the original downloaded hugging face model, only keep the transformed one to save half of the disk space. Just install airllm and run the code the same as on linux. See more in Quick Start quickstart . - make sure you installed mlx https://github.com/ml-explore/mlx?tab=readme-ov-file installation and torch - you probably need to install python native see more here https://stackoverflow.com/a/65432861/21230266 - only Apple silicon https://support.apple.com/en-us/HT211814 is supported Example python notebook https://github.com/lyogavin/airllm/blob/main/air llm/examples/run on macos.ipynb https://github.com/lyogavin/airllm/blob/main/air llm/examples/run on macos.ipynb Example colabs here: Details - ChatGLM: python from airllm import AutoModel MAX LENGTH = 128 model = AutoModel.from pretrained "THUDM/chatglm3-6b-base" input text = 'What is the capital of China?', input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH, padding=True generation output = model.generate input tokens 'input ids' .cuda , max new tokens=5, use cache= True, return dict in generate=True model.tokenizer.decode generation output.sequences 0 - QWen: python from airllm import AutoModel MAX LENGTH = 128 model = AutoModel.from pretrained "Qwen/Qwen-7B" input text = 'What is the capital of China?', input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH generation output = model.generate input tokens 'input ids' .cuda , max new tokens=5, use cache=True, return dict in generate=True model.tokenizer.decode generation output.sequences 0 - Baichuan, InternLM, Mistral, etc: python from airllm import AutoModel MAX LENGTH = 128 model = AutoModel.from pretrained "baichuan-inc/Baichuan2-7B-Base" model = AutoModel.from pretrained "internlm/internlm-20b" model = AutoModel.from pretrained "mistralai/Mistral-7B-Instruct-v0.1" input text = 'What is the capital of China?', input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH generation output = model.generate input tokens 'input ids' .cuda , max new tokens=5, use cache=True, return dict in generate=True model.tokenizer.decode generation output.sequences 0 To request other model support: here https://docs.google.com/forms/d/e/1FAIpQLSe0Io9ANMT964Zi-OQOq1TJmnvP-G3 ZgQDhP7SatN0IEdbOg/viewform?usp=sf link AirLLM works out of the box with virtually every popular open LLM — just pass its Hugging Face ID to AutoModel.from pretrained ... . That covers all the major families: Llama 2 / 3 / 3.1 / 3.3 / 4 · Qwen 1 / 2 / 2.5 / 3, including MoE and FP8 · DeepSeek V2 / V3 / R1 · Mistral & Mixtral · Phi · Gemma · ChatGLM · Baichuan · InternLM · Yi — and most new models the day they're released. The trick: AirLLM only ever keeps one layer on the GPU at a time , so the VRAM you need depends on the model's layer size — not its total size. That's how a 671B model fits on a hobbyist card: | Model | Size | GPU VRAM | |---|---|---| | Qwen3 / Mistral / Phi ≈8B | 8B | ~1–2 GB | | Qwen3-30B / Mixtral MoE | 30–47B | ~1–3 GB | | Qwen3-235B MoE | 235B | ~3 GB | | Llama 3.x 70B full precision | 70B | ~4 GB | | Llama 3.1 405B | 405B | ~8 GB | | DeepSeek-V3 | 671B | ~12 GB | Same one line of code for all of them — no special setup. A lot of the code are based on SimJeg's great work in the Kaggle exam competition. Big shoutout to SimJeg: GitHub account @SimJeg https://github.com/SimJeg , the code on Kaggle https://www.kaggle.com/code/simjeg/platypus2-70b-with-wikipedia-rag , the associated discussion https://www.kaggle.com/competitions/kaggle-llm-science-exam/discussion/446414 . safetensors rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer If you run into this error, most possible cause is you run out of disk space. The process of splitting model is very disk-consuming. See this https://huggingface.co/TheBloke/guanaco-65B-GPTQ/discussions/12 . You may need to extend your disk space, clear huggingface .cache https://huggingface.co/docs/datasets/cache and rerun. Most likely you are loading QWen or ChatGLM model with Llama2 class. Try the following: For QWen model: python from airllm import AutoModel <----- instead of AirLLMLlama2 AutoModel.from pretrained ... For ChatGLM model: python from airllm import AutoModel <----- instead of AirLLMLlama2 AutoModel.from pretrained ... Some models are gated models, needs huggingface api token. You can provide hf token: model = AutoModel.from pretrained "meta-llama/Llama-2-7b-hf", hf token='HF API TOKEN' Some model's tokenizer doesn't have padding token, so you can set a padding token or simply turn the padding config off: input tokens = model.tokenizer input text, return tensors="pt", return attention mask=False, truncation=True, max length=MAX LENGTH, padding=False <----------- turn off padding If you find AirLLM useful in your research and wish to cite it, please use the following BibTex entry: @software{airllm2023, author = {Gavin Li}, title = {AirLLM: scaling large language models on low-end commodity computers}, url = {https://github.com/lyogavin/airllm/}, version = {0.0}, year = {2023}, } Bloome is an AI-agent IM platform: build and run AI agent teams in the cloud with zero setup. Add a skill as an agent in a group chat, run it in one click from web or mobile, and share it with your team — think of it as a group chat where your AI assistants are teammates you can @mention and assign tasks to. 👉 Try Bloome https://bloome.im/app?ref=G6BYnov0&utm medium=github&utm source=lyogavin-airllm-ivor-202606 Welcomed contributions, ideas and discussions If you find it useful, please ⭐ or buy me a coffee 🙏