{"slug": "knowledge-just-moved-out-of-the-gpu-engram-architecture-overview", "title": "Knowledge just moved out of the GPU (Engram architecture overview)", "summary": "DeepSeek published a January 2026 paper and GitHub repository proposing Engram, an architectural subsystem that moves an LLM's factual knowledge out of GPU memory and into a hash-based lookup table in host RAM or on an SSD, and two labs have since shipped industry-grade models with Engram parameters: DeepSeek's v4.1-Flash and Qwen3.8-Flash-Next, a preview of the upcoming Qwen 4 series. The change targets the memory-bound economics of inference, where a dense 27B-parameter model must read 54 GB of weights per token, capping an H100 at roughly 60 tokens per second against a 3.35 TB/s memory bandwidth ceiling. DeepSeek's paper argues the split lowers the HBM required to run a specialized model.", "body_md": "# Knowledge just moved out of the GPU\n\nDeepSeek and Qwen just agreed on where a model's facts should live: in host RAM or on an SSD, not in GPU memory. Here's how that split changes what it costs to run a model, and what it means if you want to own a specialized one.\n\nDeepSeek released [a paper](https://arxiv.org/pdf/2601.07372) and [a repository](https://github.com/deepseek-ai/Engram)\nin January 2026 that proposed a new architectural subsystem for LLMs: moving knowledge outside of the neural\nnetwork parameters and into a hash-based lookup table called Engram.\n\nIn the past few weeks, two major labs have released industry-grade models with a set of Engram parameters: DeepSeek's own v4.1-Flash, and Qwen3.8-Flash-Next (which is a preview of the architectural design of the upcoming Qwen 4 series).\n\nThis is a bigger change than it looks, because it changes the economics of LLM inference. Here is why.\n\n## How LLMs work, with and without Engram\n\n### Every token is a trip through memory\n\nA language model writes one token at a time. To produce the next one, it runs the whole network once: the context goes in, every layer does its arithmetic, and a probability for each word in the vocabulary comes out. Pick one, append it, run the whole network again. A five-hundred-token answer is five hundred passes, each waiting on the last.\n\nThe arithmetic is not the expensive part. The expensive part is that every weight a pass touches has to travel from memory into the GPU compute units, and for a single token each weight is used exactly once and then discarded. A dense 27B-parameter model holds 54 GB of weights, so generating one token means reading 54 GB. An H100 (an enterprise-grade NVIDIA card that costs $25k–$40k) can pull about 3.35 TB/s out of its memory, which puts a hard ceiling of roughly 60 tokens per second on one conversation, before any other overhead. The same chip can do about a thousand trillion floating-point operations per second. The arithmetic takes 0.05 ms. The memory read takes 16 ms. More than 99% of the time the compute units sit idle, just waiting for bytes.\n\nThis is what \"memory-bound\" means, and it is why the speed of an LLM at generation time is a bandwidth number rather than a FLOPS number.\n\nEverything about how accelerators are built follows from this. High bandwidth memory (HBM), the type of memory in GPUs, moves a few terabytes per second. DDR5 does not come close: a desktop tops out near 100 GB/s and a big server near 500, still several times slower than HBM.\n\n### Mixture-of-experts to the rescue\n\nThe first approach to minimize the number of parameters needed to go from HBM to the GPU cores\nwas an architecture called Mixture-of-Experts (MoE). In this architecture, each token is\nrouted only through a subset of parameters (the \"experts\"). If you've ever seen a model name like\n`Qwen3.6-35B-A3B`, that's an MoE model that contains 35B parameters but only\nactivates 3B parameters per token.\n\nOnly a handful of experts run per token, so only their weights need to be read by the GPU cores. But the idle experts still occupy HBM.\n\nThe result of this architecture is an unchanged need for HBM (since all experts need to be\nloaded into the GPU) but lower token generation time since we need to move fewer\nbytes between the HBM and the GPU cores. The tradeoff is that a 35B-A3B model does\nnot have the same intelligence as a dense 35B model. In fact, `Qwen3.6-27B` is widely\nregarded as a better model than `Qwen3.6-35B-A3B` in terms of output quality.\n\n### Engram: the parameters you just look up\n\nEngram, the paper DeepSeek published in January, proposes a different way of reducing the required HBM memory.\n\nLLMs don't use text directly. They first translate it into tokens, which you can think of as pseudowords. The problem is that in human language we have plenty of concepts that are multi-token. For example, \"New York\" is two words, but one concept.\n\nHow do LLMs deal with this? Through compute: the parameters in their neural networks not only encode how to reason about text, but also need to encode knowledge.\n\nDeepSeek looked at this through a different lens: can we move knowledge outside of the neural network? The answer reuses the oldest idea in language modeling: n-grams.\n\nTake the current token and the one or two before it. That short sequence (e.g. `New York`,\n`as a result`, `def __init__`) is a key. Hash the key into a very large table, pull\nout some information (an embedding vector), and add it\nto the internal state of the LLM at an early layer. The lookup is constant time, it\ninvolves no matrix multiplication, and the key depends only on token ids the\nmodel already has.\n\nEach token touches a handful of rows out of billions, and because the key is just a few token ids, the model knows which rows it needs before the pass starts. That is what lets the table live in host RAM, or even on an NVMe SSD.\n\nWhat's the benefit of the Engram table? Once the LLM gets to \"York\" in \"New York\", it will instantly know the text is referring to the US city and not the UK one thanks to the information pulled from the Engram table. In a traditional LLM, that only happens through computation, as the tokens move through the different layers of the neural network.\n\nA telling result from the paper is that once DeepSeek removed the Engram table from their model, factual benchmarks collapsed to 29% to 44% of their original score, while reading comprehension kept 81% to 93%. The facts were in the table. The reasoning was not.\n\n## How this changes the economics of LLMs\n\n### Two kinds of parameters, two price tags\n\nUntil now, every parameter in a model had to live in HBM, and HBM is soldered next to a GPU that costs tens of thousands of dollars. If your model needed more memory, you bought more or bigger GPUs.\n\nEngram splits the model in two. The reasoning parameters still need HBM, because every token reads them and only HBM can feed them fast enough. The knowledge parameters need something else entirely: capacity. A few kilobytes per token, fetched ahead of time, can come from DDR5 you add to the server by the stick, or from an SSD that holds tens of terabytes for a few cents a gigabyte. In V4.1-Flash that is 196B of roughly 750B parameters. In Qwen3.8-Flash-Next, 51B of 176B. About a quarter of each model just stopped needing a GPU. Knowledge that used to cost compute now costs a lookup, so the labs can keep the active parameter count small by using MoE (6B for Qwen and 16B for DeepSeek), without giving up what the model knows.\n\nThe combination of MoE and Engram allows aggressive optimizations,\nlike [running Qwen3.8-Flash-Next on a 12 GB GPU](https://carteakey.dev/blog/running-qwen3-8-flash-next-locally/):\na 125B-parameter model with 6B activated parameters plus a 51B n-gram lookup table.\nThe experts live in host RAM and the lookup table on an NVMe drive, and it\nstill decodes at about 20 tokens per second.\n\n### Self-hosting and fine-tuning\n\nWe argued in [an earlier post](https://three.dev/blog/your-business-can-fit-in-a-smaller-model)\nthat your business fits in a smaller model. Engram makes the expensive part of\na model (the neural network parameters) smaller. The question for a specialized\nmodel is whether 196B of facts about the world help a task that does not need\nany of them. Probably not: a model that reads insurance claims does not need\nto know who discovered penicillin. Nobody has a good recipe for fine-tuning\nEngram models yet, so trimming or replacing that table is still an open problem.\n\n## Why this matters\n\nFor as long as LLMs have been served from GPUs, a parameter was a parameter: every one of them had to sit in the most expensive memory in the building. MoE was the first crack in that rule. It cut the bytes that move per token, but left the HBM bill untouched. Engram is the second. It cuts what has to sit in HBM at all. Put the two together and the GPU holds only the part of the model that reasons, while what the model knows sits in cheap RAM or an SSD.\n\nTwo labs shipping the same parameter split within weeks of each other is a sign that this architecture has evolved from research prototype to industry trend.\n\nIf you run models, capacity planning now has two lines instead of one: HBM bandwidth for reasoning, and cheap capacity for knowledge. If you want to own a specialized model, what the model knows is becoming a separate artifact from how it reasons, one that could be inspected, swapped, or shrunk to fit the job. That second part is the one we are watching most closely.", "url": "https://wpnews.pro/news/knowledge-just-moved-out-of-the-gpu-engram-architecture-overview", "canonical_source": "https://three.dev/blog/knowledge-just-moved-out-of-the-gpu", "published_at": "2026-10-15 09:36:00+00:00", "updated_at": "2026-10-15 09:50:16.989935+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure", "ai-chips"], "entities": ["DeepSeek", "Engram", "Qwen", "DeepSeek v4.1-Flash", "Qwen3.8-Flash-Next", "Qwen 4", "NVIDIA H100", "Mixture-of-Experts"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/knowledge-just-moved-out-of-the-gpu-engram-architecture-overview", "markdown": "https://wpnews.pro/news/knowledge-just-moved-out-of-the-gpu-engram-architecture-overview.md", "text": "https://wpnews.pro/news/knowledge-just-moved-out-of-the-gpu-engram-architecture-overview.txt", "jsonld": "https://wpnews.pro/news/knowledge-just-moved-out-of-the-gpu-engram-architecture-overview.jsonld"}}