HotPin: Running 120B MoE on 24GB RAM

A new llama.cpp patch called HotPin enables running a 120-billion-parameter mixture-of-experts (MoE) model on just 24GB of RAM, achieving up to 67% memory savings and a 45% speedup over standard swapping, with output that is SHA-256 bit-identical to a full-RAM run. The implementation, created by developer LozzKappa, profiles MoE expert routing frequencies and uses mmap with selective mlock and posix_fadvise prefetching, requiring only about 50 lines of C++ code.

HotPin: Running 120B MoE on 24GB RAM I've been looking into the benchmarks for this llama.cpp patch, and the results on an AMD Ryzen AI 9 HX 370 23.6GB LPDDR5X are pretty revealing for anyone running LLM agents on consumer hardware: gpt-oss:120b: Disk 58.5GB → Min RAM 19.1GB 67% savings at 3.84 tok/s qwen3:30b-a3b: Disk 18.0GB → Min RAM 10.4GB 42% savings at 19.7 tok/s gemma4:26b-a4b: Disk 16.2GB → Min RAM 10.6GB 35% savings at 11.5 tok/s GLM-4.7-Flash: Disk 19.0GB → Min RAM 13.3GB 30% savings at 12.4 tok/s The most impressive part is that the output is SHA-256 bit-identical to a full-RAM run. This isn't a "compressed" version of the model; it's a lossless deployment. The technical implementation is surprisingly lean around 50 lines of C++ . It works by profiling MoE expert routing frequencies, using mmap for the whole model, but only using mlock for the "hottest" experts. It then leverages posix fadvise to prefetch cold experts from the NVMe just before they are needed. For those of us obsessed with local deployment, this solves a massive pain point. If the model exceeds your RAM, you usually see a performance cliff. HotPin actually provides a 45% speedup in those scenarios for gpt-oss, it jumped from 2.64 to 3.84 tok/s compared to standard swapping. If you want to dive into the implementation, the code is available here: https://github.com/LozzKappa/hotpin-llm Next Project Intelligence: Why Git Isn't Enough for AI → /en/threads/3451/