Lyogavin has released AirLLM, an open-source Python library that enables running 70B parameter large language models on GPUs with as little as 4GB of VRAM. By streaming model layers sequentially from …
Independent researcher Alpamys Makazhan released Soup, a Show HN project that fine-tunes a full Llama-3.1-8B model in NF4 quantization with a 3.32 GB VRAM peak at 119.6 tokens/sec on a 4 GB RTX 3050 L…
AirLLM, a new tool from GitHub user lyogavin, enables inference of 70B-parameter models on a single 4GB GPU by aggressively quantizing and offloading layers, potentially reducing hardware costs by 10–…
AirLLM, an open-source inference library, claims to run a 70B-parameter large language model on a 4GB GPU without quantization, distillation, or pruning. The claim is technically true, achieved by loa…
AirLLM, an open-source tool by Gavin Li, can run a 70B-parameter model on a 4GB GPU by streaming layers from disk, but this method is limited by disk bandwidth, resulting in seconds to minutes per tok…
AirLLM, an Apache-2.0 Python library, enables running large language models like Llama 3.1 405B on 8GB of VRAM and DeepSeek-V3 671B on ~12GB without quantization, by streaming one layer or MoE expert …
AirLLM, an open-source inference library, now supports running the 2.8T-parameter Kimi K3 model on a single 4GB GPU, using only 3.72GB of VRAM on an RTX 6000 Ada, by streaming one expert at a time for…
A developer released AirLLM, an open-source tool that enables running 70-billion-parameter AI models on laptops with as little as 8GB RAM, without requiring a GPU. By using memory mapping and layer sw…