lyogavin

Lyogavin has released AirLLM, an open-source Python library that enables running 70B parameter large language models on GPUs with as little as 4GB of VRAM. By streaming model layers sequentially from …

AirLLM, a new tool from GitHub user lyogavin, enables inference of 70B-parameter models on a single 4GB GPU by aggressively quantizing and offloading layers, potentially reducing hardware costs by 10–…