AI architecture / Local inference
One shared backbone, thousands of tiny experts, and a smaller context cache.
A research lab announces a breakthrough open model. The weights are public, the benchmarks rival proprietary giants, and developers celebrate. Then they check the system requirements. the model demands hundreds of gigabytes of high-bandwidth memory across a rack of enterprise graphics cards.
The weights are open, but the hardware bill acts as an invisible wall.
This problem stems from the gap between active parameters and total parameters. In modern Mixture of Experts (MoE) architectures, generating a single word might only activate thirty billion parameters. Computationally, that is relatively cheap. But all six hundred billion parameters must stay loaded in memory, waiting to be called. For a hyperscale cloud provider serving thousands of concurrent users, that investment makes economic sense. For an individual developer or a small team running a local assistant, paying for a massive bank of idle memory makes self-hosting practically impossible.
If independent, private AI is going to survive outside of centralized data centers, we need to rethink how these systems are built. I wrote about the hardware barrier in AI Is Getting Better. Running It Yourself Is Getting Harder. This builds on the question behind How Small Can We Slice an AI Model?: how much of a model needs to stay in fast memory at once?
The path forward is not building ever-larger monoliths. Instead, the model can be split into a single frozen backbone paired with a library of thousands of tiny micro-experts.
At the center sits a compressed, quantized base model. This core handles universal grammar, common sense, and general reasoning. It stays permanently loaded in local GPU memory, but it does not try to memorize every niche domain, obscure coding syntax, or corporate policy.
Specialized capabilities live in a sprawling collection of low-rank adapters (LoRA). Instead of an expert being a full-sized neural network weighing gigabytes, each micro-expert is a thin mathematical adjustment compressed through a narrow bottleneck. At low rank, an expert is only two to five megabytes.
Navigating thousands of these adapters without slowing down requires a two-step hierarchical router:
- The Domain Gate: A high-level router evaluates the broad territory of the prompt, narrowing ten thousand candidates down to twenty or thirty relevant specialists.
- The Micro-Router: A secondary router scores only that small cluster and selects the top two or three micro-adapters needed for the immediate sentence.
Parameters are only half the battle. The other silent killer of consumer hardware is the context cache: the memory a model uses to track previous words in a conversation.
In the early days of the Transformer, standard Multi-Head Attention required storing full-resolution keys and values for every token across every attention head. As context windows expanded to tens of thousands of tokens, this scratchpad devoured the fastest memory on the chip. Labs tried forcing heads to share memory (Multi-Query and Grouped-Query Attention), but that compromised subtle reasoning.
The breakthrough came with Multi-Head Latent Attention (MLA). Instead of caching full-resolution vectors, MLA compresses each token's historical state into a compact latent vector (often just 512 numbers wide). Through linear algebra optimizations, the model calculates attention directly against these compressed numbers without unpacking the historical cache back into its bulky form.
This compression works because human communication is governed by topic locality and temporal locality:
- Topic locality: We do not bounce randomly across thousands of unrelated concepts from one sentence to the next. When a discussion focuses on a specific task, our vocabulary, syntax, and logic naturally stay within that territory for dozens of sentences. Most of the model's vast concept space is temporarily irrelevant.
- Temporal locality: Ideas unfold in sequential waves. Words close to one another in time carry the strongest grammatical and logical ties, while older context only needs its central anchor points remembered, not every individual detail.
Because the active meaning of a conversation lives on a much smaller, low-dimensional slice of geometry, the system does not need to keep every past word expanded across hundreds of dimensions just in case. Latent attention preserves the compact coordinates of the discussion, slashing the context cache footprint by roughly ninety percent while keeping reasoning sharp.
The true operational benefit appears when moving data across physical hardware.
Attempting to swap a full two-gigabyte expert from system RAM across a PCIe slot between words takes too long, causing generation speed to stall. But streaming a five-megabyte adapter takes only one or two milliseconds.
Because human tasks stay focused on specific topics for extended periods, the system does not need to juggle thousands of files at once:
- The local graphics card holds the central backbone and the top fifty most common adapters in fast memory.
- When the conversation touches an unusual topic, such as an obscure legal clause or a niche database format, the system streams that five-megabyte adapter from ordinary system RAM or a fast solid-state drive in a fraction of a millisecond.
- Once that task is finished, the slot clears for the next tool.
Frontier labs will continue training dense, multi-hundred-billion-parameter clusters to push benchmark records. But the serving layer is converging on a different reality: one shared base model routing dynamically to thousands of lightweight adapters.
For developers and privacy-focused organizations, this modular design is an escape hatch from cloud dependency. It eliminates the need for expensive multi-GPU clusters just to keep idle weights warm. By combining compressed latent attention, a stable central foundation, and tiny swappable micro-experts, capable AI can run locally on accessible hardware. You do not need to wait for consumer computers to ship with a terabyte of memory; you just need an architecture designed to use the memory we already have.