{"slug": "hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram", "title": "Hot experts, cold RAM: a big MoE on a box with more memory than VRAM", "summary": "A developer built Strata, a fork of llama.cpp that keeps only the currently active experts of a Mixture-of-Experts model resident in memory and streams the rest through system DDR instead of GPU VRAM. A further fork, Avalon, tightens the hot-expert set to support longer context and larger models on the same hardware. The developer reports that a machine with 256 GB of DDR and an RTX 5080 can run small and medium private models, with RAM rather than a new GPU as the upgrade path.", "body_md": "Every local deployment spends against two budgets at once: the size of the model and the price of\n\nthe memory holding it. VRAM is the one people reach for first, and it is the one that costs the most\n\nper gigabyte. System DDR is fast, vast and cheap; GPU memory is fast, small and expensive. An\n\ninference engine that can put weights in the second place changes the bill.\n\nThat is what Strata is for. It is forked from llama.cpp and built for memory efficiency: it keeps\n\nresident only the experts that are hot - the ones actually in use - and leaves the rest parked.\n\nThat is how big MoE models run on small RAM.\n\nllama.cpp is the base of everything here: an inference engine in plain C/C++ that runs GGUF models on\n\nhardware you already own - Metal, CUDA, Vulkan, SYCL or CPU only - with 1.5-8 bit quantisation and\n\nhybrid offload for models too big for the GPU. Hybrid offload moves layers between CPU and GPU. An\n\nMoE is a different shape of problem: it is not one network that is too big, it is a fleet of experts\n\nof which only a few matter to a given query.\n\nSo the efficient move is not to place layers. It is to place experts, and to decide which of them\n\ndeserve to be resident right now. Strata does exactly that, and Avalon forks Strata to tune the same\n\nhot set even tighter, which buys longer usable context and larger models on the same hardware.\n\nThe second half of the picture is where the weights sit while they wait. The engine streams model\n\nweights through ordinary system memory - fast, vast DDR instead of expensive GPU VRAM. One machine\n\nwith 256 GB of DDR and an RTX 5080 runs small and medium private AI comfortably.\n\nFor a small organisation the consequence is concrete: the machine you can afford is the machine that\n\nfits the model you want. A 5080-class card with a lot of memory you already bought does more work\n\nthan a bigger card with less, and the upgrade path is RAM, not a new GPU.\n\nExpert selection is not random. Queries cluster: a support inbox about one product, a codebase, a\n\ncorpus of legal text. The gate keeps coming back to the same experts, so the hot set stabilises\n\nafter a while, page-in traffic drops away, and the parked experts stay parked - which is exactly the\n\nefficiency argument for Mixture of Experts in the first place: each query activates only the experts\n\nit needs, and the rest never load into memory at all.\n\nTwo things. First, an NVMe that is actually fast: the cold tier is only cheap if the first read of an\n\nexpert is quick. Second, a model worth the complexity - a dense model that already fits in VRAM is\n\nusually the simpler answer. And measure rather than trust: one model file, weights and prompt length\n\nheld identical across every engine that can load it on that box, reading and writing speed measured\n\napart, versions recorded with the number\n\n([benchmark index](https://localai.hex7.com/benchmark.html)).\n\nBecause inference runs on your own hardware inside the building, personal data never leaves the\n\npremises. No Chapter V transfer to arrange, no standard contractual clauses to keep valid, no US\n\ncloud disclosure to answer for - the accountability question never comes up, because the violation\n\nnever happens. Ask about an on-site assessment:\n\n[privatelocalai@proton.me](mailto:privatelocalai@proton.me).", "url": "https://wpnews.pro/news/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram", "canonical_source": "https://dev.to/private_localai/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram-3pn6", "published_at": "2026-11-12 22:14:00+00:00", "updated_at": "2026-11-12 22:22:48.720903+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "ai-tools", "large-language-models"], "entities": ["Strata", "Avalon", "llama.cpp", "RTX 5080", "GGUF"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram", "markdown": "https://wpnews.pro/news/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram.md", "text": "https://wpnews.pro/news/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram.txt", "jsonld": "https://wpnews.pro/news/hot-experts-cold-ram-a-big-moe-on-a-box-with-more-memory-than-vram.jsonld"}}