AirLLM: Running 70B Parameter LLMs on a Single 4GB GPU

Lyogavin has released AirLLM, an open-source Python library that enables running 70B parameter large language models on GPUs with as little as 4GB of VRAM. By streaming model layers sequentially from disk, AirLLM reduces VRAM requirements by over 90% while maintaining full 16-bit precision, making it accessible on consumer hardware including Apple Silicon MacBooks.

As open-source Large Language Models LLMs continue to grow in capability, their hardware requirements have ballooned alongside them. Running a 70B or 405B parameter model traditionally demands enterprise cloud GPU servers equipped with hundreds of gigabytes of VRAM. AirLLM is an open-source Python library developed by lyogavin to make massive model inference accessible on standard consumer hardware—allowing developers to run 70B models on GPUs with as little as 4GB of VRAM. Rather than attempting to fit an entire neural network into GPU memory at once, AirLLM uses a "divide and conquer" execution architecture. It streams individual model layers sequentially from disk into memory, computes the output for that specific layer, and then clears it before loading the next. By executing model layers sequentially, AirLLM slashes VRAM requirements by over 90%. It allows developers to run 70B parameter models on a 4GB VRAM GPU, 405B models on 8GB VRAM, and Mixture-of-Experts MoE architectures with minimal memory overhead. Many memory-saving tools rely heavily on 4-bit or 2-bit quantization, which can degrade reasoning capabilities. AirLLM allows developers to execute full 16-bit precision models directly from disk without sacrificing output accuracy. AirLLM is hardware-agnostic. It runs seamlessly on standard desktop PCs with budget graphics cards, cloud instances, and Apple Silicon MacBooks M1, M2, M3, and M4 chips . Integrating AirLLM into an existing Python script requires only a few lines of code: python from airllm import AirLLMLlama model = AirLLMLlama "meta-llama/Meta-Llama-3.1-70B-Instruct" input text = "What is the capital of France?" input tokens = model.tokenizer input text, return tensors="pt" generation output = model.generate input tokens 'input ids' .cuda , max new tokens=20, use cache=True output text = model.tokenizer.decode generation output 0 print output text By decoupling LLM parameter size from VRAM capacity, AirLLM removes one of the largest financial barriers in AI development. It empowers researchers, independent developers, and hobbyists to test state-of-the-art models locally without relying on expensive cloud GPU clusters. Want to run 70B models on your machine? Check out the AirLLM GitHub Repository.