Hot experts, cold RAM: a big MoE on a box with more memory than VRAM

A developer built Strata, a fork of llama.cpp that keeps only the currently active experts of a Mixture-of-Experts model resident in memory and streams the rest through system DDR instead of GPU VRAM. A further fork, Avalon, tightens the hot-expert set to support longer context and larger models on the same hardware. The developer reports that a machine with 256 GB of DDR and an RTX 5080 can run small and medium private models, with RAM rather than a new GPU as the upgrade path.

Every local deployment spends against two budgets at once: the size of the model and the price of the memory holding it. VRAM is the one people reach for first, and it is the one that costs the most per gigabyte. System DDR is fast, vast and cheap; GPU memory is fast, small and expensive. An inference engine that can put weights in the second place changes the bill. That is what Strata is for. It is forked from llama.cpp and built for memory efficiency: it keeps resident only the experts that are hot - the ones actually in use - and leaves the rest parked. That is how big MoE models run on small RAM. llama.cpp is the base of everything here: an inference engine in plain C/C++ that runs GGUF models on hardware you already own - Metal, CUDA, Vulkan, SYCL or CPU only - with 1.5-8 bit quantisation and hybrid offload for models too big for the GPU. Hybrid offload moves layers between CPU and GPU. An MoE is a different shape of problem: it is not one network that is too big, it is a fleet of experts of which only a few matter to a given query. So the efficient move is not to place layers. It is to place experts, and to decide which of them deserve to be resident right now. Strata does exactly that, and Avalon forks Strata to tune the same hot set even tighter, which buys longer usable context and larger models on the same hardware. The second half of the picture is where the weights sit while they wait. The engine streams model weights through ordinary system memory - fast, vast DDR instead of expensive GPU VRAM. One machine with 256 GB of DDR and an RTX 5080 runs small and medium private AI comfortably. For a small organisation the consequence is concrete: the machine you can afford is the machine that fits the model you want. A 5080-class card with a lot of memory you already bought does more work than a bigger card with less, and the upgrade path is RAM, not a new GPU. Expert selection is not random. Queries cluster: a support inbox about one product, a codebase, a corpus of legal text. The gate keeps coming back to the same experts, so the hot set stabilises after a while, page-in traffic drops away, and the parked experts stay parked - which is exactly the efficiency argument for Mixture of Experts in the first place: each query activates only the experts it needs, and the rest never load into memory at all. Two things. First, an NVMe that is actually fast: the cold tier is only cheap if the first read of an expert is quick. Second, a model worth the complexity - a dense model that already fits in VRAM is usually the simpler answer. And measure rather than trust: one model file, weights and prompt length held identical across every engine that can load it on that box, reading and writing speed measured apart, versions recorded with the number benchmark index https://localai.hex7.com/benchmark.html . Because inference runs on your own hardware inside the building, personal data never leaves the premises. No Chapter V transfer to arrange, no standard contractual clauses to keep valid, no US cloud disclosure to answer for - the accountability question never comes up, because the violation never happens. Ask about an on-site assessment: privatelocalai@proton.me mailto:privatelocalai@proton.me .