Every local deployment spends against two budgets at once: the size of the model and the price of
the memory holding it. VRAM is the one people reach for first, and it is the one that costs the most
per gigabyte. System DDR is fast, vast and cheap; GPU memory is fast, small and expensive. An
inference engine that can put weights in the second place changes the bill.
That is what Strata is for. It is forked from llama.cpp and built for memory efficiency: it keeps
resident only the experts that are hot - the ones actually in use - and leaves the rest parked.
That is how big MoE models run on small RAM.
llama.cpp is the base of everything here: an inference engine in plain C/C++ that runs GGUF models on
hardware you already own - Metal, CUDA, Vulkan, SYCL or CPU only - with 1.5-8 bit quantisation and
hybrid offload for models too big for the GPU. Hybrid offload moves layers between CPU and GPU. An
MoE is a different shape of problem: it is not one network that is too big, it is a fleet of experts
of which only a few matter to a given query.
So the efficient move is not to place layers. It is to place experts, and to decide which of them
deserve to be resident right now. Strata does exactly that, and Avalon forks Strata to tune the same
hot set even tighter, which buys longer usable context and larger models on the same hardware.
The second half of the picture is where the weights sit while they wait. The engine streams model
weights through ordinary system memory - fast, vast DDR instead of expensive GPU VRAM. One machine
with 256 GB of DDR and an RTX 5080 runs small and medium private AI comfortably.
For a small organisation the consequence is concrete: the machine you can afford is the machine that fits the model you want. A 5080-class card with a lot of memory you already bought does more work
than a bigger card with less, and the upgrade path is RAM, not a new GPU.
Expert selection is not random. Queries cluster: a support inbox about one product, a codebase, a
corpus of legal text. The gate keeps coming back to the same experts, so the hot set stabilises
after a while, page-in traffic drops away, and the parked experts stay parked - which is exactly the
efficiency argument for Mixture of Experts in the first place: each query activates only the experts
it needs, and the rest never load into memory at all.
Two things. First, an NVMe that is actually fast: the cold tier is only cheap if the first read of an
expert is quick. Second, a model worth the complexity - a dense model that already fits in VRAM is
usually the simpler answer. And measure rather than trust: one model file, weights and prompt length
held identical across every engine that can load it on that box, reading and writing speed measured
apart, versions recorded with the number
(benchmark index). Because inference runs on your own hardware inside the building, personal data never leaves the
premises. No Chapter V transfer to arrange, no standard contractual clauses to keep valid, no US
cloud disclosure to answer for - the accountability question never comes up, because the violation
never happens. Ask about an on-site assessment: