{"slug": "hotpin-running-120b-moe-on-24gb-ram", "title": "HotPin: Running 120B MoE on 24GB RAM", "summary": "A new llama.cpp patch called HotPin enables running a 120-billion-parameter mixture-of-experts (MoE) model on just 24GB of RAM, achieving up to 67% memory savings and a 45% speedup over standard swapping, with output that is SHA-256 bit-identical to a full-RAM run. The implementation, created by developer LozzKappa, profiles MoE expert routing frequencies and uses mmap with selective mlock and posix_fadvise prefetching, requiring only about 50 lines of C++ code.", "body_md": "# HotPin: Running 120B MoE on 24GB RAM\n\nI've been looking into the benchmarks for this llama.cpp patch, and the results on an AMD Ryzen AI 9 HX 370 (23.6GB LPDDR5X) are pretty revealing for anyone running LLM agents on consumer hardware:\n\n**gpt-oss:120b:** Disk 58.5GB → Min RAM 19.1GB (67% savings) at 3.84 tok/s**qwen3:30b-a3b:** Disk 18.0GB → Min RAM 10.4GB (42% savings) at 19.7 tok/s**gemma4:26b-a4b:** Disk 16.2GB → Min RAM 10.6GB (35% savings) at 11.5 tok/s**GLM-4.7-Flash:** Disk 19.0GB → Min RAM 13.3GB (30% savings) at 12.4 tok/s\n\nThe most impressive part is that the output is SHA-256 bit-identical to a full-RAM run. This isn't a \"compressed\" version of the model; it's a lossless deployment.\n\nThe technical implementation is surprisingly lean (around 50 lines of C++). It works by profiling MoE expert routing frequencies, using mmap for the whole model, but only using mlock for the \"hottest\" experts. It then leverages posix_fadvise to prefetch cold experts from the NVMe just before they are needed.\n\nFor those of us obsessed with local deployment, this solves a massive pain point. If the model exceeds your RAM, you usually see a performance cliff. HotPin actually provides a 45% speedup in those scenarios (for gpt-oss, it jumped from 2.64 to 3.84 tok/s) compared to standard swapping.\n\nIf you want to dive into the implementation, the code is available here:\n\n```\nhttps://github.com/LozzKappa/hotpin-llm\n```\n\n[Next Project Intelligence: Why Git Isn't Enough for AI →](/en/threads/3451/)", "url": "https://wpnews.pro/news/hotpin-running-120b-moe-on-24gb-ram", "canonical_source": "https://promptcube3.com/en/threads/3462/", "published_at": "2026-07-26 01:01:24+00:00", "updated_at": "2026-07-26 01:35:26.510088+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["llama.cpp", "HotPin", "AMD Ryzen AI 9 HX 370", "gpt-oss:120b", "qwen3:30b-a3b", "gemma4:26b-a4b", "GLM-4.7-Flash", "LozzKappa"], "alternates": {"html": "https://wpnews.pro/news/hotpin-running-120b-moe-on-24gb-ram", "markdown": "https://wpnews.pro/news/hotpin-running-120b-moe-on-24gb-ram.md", "text": "https://wpnews.pro/news/hotpin-running-120b-moe-on-24gb-ram.txt", "jsonld": "https://wpnews.pro/news/hotpin-running-120b-moe-on-24gb-ram.jsonld"}}