# The 890-Byte Token - DeepSeek-V4.1 Flash’s Elegant KV Cache Optimizations

> Source: <https://interestingengineering.substack.com/p/the-890-byte-token-deepseek-v41-flashs>
> Published: 2026-09-18 16:48:58+00:00

Consider this a continuing series as I follow “The Whale”’s continuing software optimization “tricks”. An exceptionally strong, paradigm-shifting response to the hardware crisis. This 552-billion-parameter MoE model introduces an architecture that targets the exact metric breaking data center budgets: the **KV Cache footprin** t! 

Because it **delivers flagship-level agentic performance (even beating their older V4-Pro pipeline) while bypassing the memory squeeze, it represents a major blueprint for how AI software can outpace hardware limitations**.

For long-context or agentic tasks (like coding assistants analyzing a 400,000-token repository), the KV Cache usually balloons, hogging expensive GPU memory (HBM) and preventing concurrent user sessions. V4.1 Flash drastically compresses this overhead by for e.g.: Global HBM Reduction, Footprint drop, applying persistent Storage (SSD) Reductions etc., which we will hear more about further below.

The previous overview was:

This article draws on two recent technical papers:

1. [DeepSeek-V4.1-Flash: Pushing The Limits of KV Cache Compression](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf) ; and
2. [Cross-Model KV Cache Transfers in LLM Families](https://arxiv.org/abs/2608.03893) : A Closed-Form Linear Mapping For Prefill Re-use (NVIDIA)

In November 2023, DeepSeek’s first model needed 389,120 bytes of GPU memory to remember a single token of context. DeepSeek-V4.1-Flash, published this month, needs 890. That 437-fold cut came from decisions about what to store, where to keep it, and what to throw away and rebuild on demand.

NVIDIA’s cross-model paper goes after a different “bill”. Routers swap between small and large models mid-session, and every swap forces the receiving model to re-read the whole conversation. A linear map fitted in about an hour lets the receiver skip that re-read, 2.7 to 25 times faster on the pairs tested.

**Four terms, plain English**

• **KV cache:** the model’s notes on everything it has read, one set per token. Attention consults them at every step.

• **Prefill:** reading the prompt and writing those notes. Expensive, and repeated whenever the notes are lost.

• **Decode:** writing the answer one token at a time, checking the notes each step. Usually capped by memory speed.

• **HBM, host RAM, SSD:** desk, filing cabinet, archive room. Faster tiers cost more per byte and hold less.

For those who prefer a great high-level watch, but adds the technical touch beautifully:

And one less technical from Two Minute Papers:

## 1. The shape of the problem

A token’s notes pass through five stages, and each stage bills a different resource. Reading the prompt burns compute. Holding the notes burns GPU memory. Writing the answer burns memory bandwidth. Parking the session burns SSD and host RAM. Handing the session to another model burns all of it again. **DeepSeek-V4.1-Flash attacks the first four issues. NVIDIA’s mapper attacks the fifth.**

*Figure 1. Five stages of a token’s memory. Badges show the saving each technique claims or implies.*

## 2. Stage by stage

### Stage 1 · Read the prompt (prefill, on GPU)

#### Causal Encoder-Decoder (CED)

• **What it does:** splits the 40-layer network into a 20-layer reader (encoder) and a 20-layer writer (decoder). The writer’s long-range notes are projected straight from the reader’s last layer, so most prompt tokens never pass through the top half.

• **Saves:** about 2× less prefill compute. 8B parameters fire per prompt token, 16B per generated token.

• **Why apply it:** agents re-read large contexts after every tool call. When the cache misses, prefill is the bill, and CED halves it. The idea descends from Microsoft’s YOCO.

#### Decoder SWA Bounded Replay

• **What it does:** every layer also keeps a short local window of the last 128 tokens, built from its own state. Rebuilding the writer’s windows exactly means pushing 20 × 128 = 2,560 tokens through the writer. DeepSeek pushes the last 128 and accepts a close approximation.

• **Saves:** 20× fewer tokens through the decoder at every prefill (derived from the configuration).

• **Why apply it:** without it, CED’s halving leaks back on short follow-up turns. DeepSeek simulates the replay during post-training so the model expects it.

#### Engram lookup memory

• **What it does:** a 196B-parameter table of phrase memories (2- to 4-word patterns) sits in host RAM. Addresses depend only on the input text, so rows are fetched over RDMA before the GPU needs them.

• **Saves:** no speed multiple claimed. It adds knowledge capacity without adding GPU compute or GPU memory per token.

• **Why apply it:** memorised facts move to cheap memory, leaving GPU compute for reasoning.

### Stage 2 · Hold the working memory (global KV, in HBM)

#### CSA2 cross-layer KV and index reuse

• **What it does:** four “Full” layers (2, 8 and 14 in the encoder, 20 in the decoder) write long-range notes. The other 34 attention layers borrow them. Four “Reindex” layers (24, 28, 32, 36) re-rank the borrowed notes with their own question; the remaining 30 “Reuse” layers borrow the ranking as well. Encoder notes also pack two tokens into one entry.

• **Saves:** with FP4, global KV falls from 3,514 to 890 bytes per token: 3.9× less than V4-Flash, 437× less than DeepSeek-V1. Reuse layers run on 15 kernels in prefill and 11 in decode.

• **Why apply it:** at long context, stored notes become the biggest claim on GPU memory. Fewer bytes per token means more concurrent sessions per GPU.

#### FP4 main KV cache

• **What it does:** stores each value in 4 bits with one 8-bit scale per 16 values, trained in during post-training. Values are widened again before attention, so any GPU can run it.

• **Saves:** ≈2× against V4’s FP8 main KV, both in HBM and on SSD.

• **Why apply it:** after normalisation, values cannot exceed about 22.6 (observed peak around 10), so a simple 4-bit format loses nothing measurable. The 128-token local window stays at FP8 because it is more sensitive.

*Figure 2. Four DeepSeek generations of per-token global KV, and a reconstruction of the 890 bytes from the report’s configuration.*

An independent teardown by zartbot, working from the released configuration file, reaches the same four components and the same 890-byte total. The config confirms the layer assignments the report only describes in prose: cache-writing layers 2, 8, 14 and 20; index-producing layers 2, 8, 14, 20, 24, 28, 32 and 36.

#### Where the 3.9× actually comes from

Splitting the drop into steps changes the story. Cross-layer reuse on its own would have landed at 652 bytes, better than the final figure. DeepSeek then spent 2.5× of that back by loosening sequence compression from 4:1 in V4-Flash to 2:1 in the encoder and 1:1 in the decoder, buying one cache entry per token where the decoder needs to address individual positions. FP4 recovered the difference.

*Figure 3. The 3,514 → 890 drop broken into three steps. The middle step goes the wrong way on purpose.*

**The trade?**

• Judged on bytes alone, V4.1-Flash left compression on the table. Judged on what the cache can do, it bought a finer-grained decoder cache and paid for it in precision rather than in capacity. Reading the 3.9× as pure compression misses the design decision underneath it.

#### Three jobs, three refresh rates

The clearest way to read CSA2 is to split attention into three jobs. Content is the compressed notes, and building them means scanning the whole history: the expensive job, done 4 times. Address is the shortlist of 512 positions a layer will read, re-picked 8 times. Query is the question a layer asks, a cheap projection of its own state, recomputed in all 40 layers. Cost falls in the same order the refresh rate rises.

• **Full mode** rebuilds all three: a cache miss with a full write-back.

• **Reindex mode** keeps the notes and re-picks where to look: the data is resident, the address is recomputed.

• **Reuse mode** keeps both and only rewrites the question: a full hit, and the mode 30 of the 40 layers run in.

The economy is real but bounded. A reuse layer cannot reach a position its source layer left off the shortlist, and a selection mistake made upstream cannot be corrected downstream. It is the same failure mode NVIDIA documents in Section 5: what matters is where the error lands, not how large it is.

*Figure 4. The three jobs inside CSA2 and how often each is rebuilt. The three modes are the three combinations.*

### Stage 3 · Write the answer (decode, on GPU)

#### Hierarchical Sparse Indexer

• **What it does:** the first decoder layer scores every past position, keeps its top 512, and marks the best 2,048 blocks of 8 positions: a 16,384-position shortlist. Later re-ranking layers search only the shortlist.

• **Saves:** deeper indexers do constant work instead of work that grows with context. At one million tokens that is ≈61× fewer positions scored per query (derived). Across all changes, decode FLOPs rise only about 25% from 4K to 1M context.

• **Why apply it:** long agent sessions spend their hours decoding against huge histories.

#### Single-Pass mHC and the Mega-mHC kernel

• **What it does:** mHC keeps four parallel residual lanes between blocks. Using the previous block’s mixing weights removes a wait, so one fused kernel reads and writes the lanes once.

• **Saves:** 2× less activation memory traffic, from (4n+4)d to (2n+2)d per block.

• **Why apply it:** decode is usually memory-bound, so moving half the bytes speeds every token. Quality cost reported as negligible.

#### DSpark speculative decoding

• **What it does:** a three-block drafter proposes five tokens in one pass; a confidence head and scheduler choose how many the main model verifies, given current load.

• **Saves:** not quantified in the report.

• **Why apply it:** more accepted tokens per expensive forward pass, in serving and in RL rollouts.

### Stage 4 · Park and resume (SSD and host RAM)

#### Split persistent cache with Encoder SWA Bounded Replay

• **What it does:** long-range notes go to SSD with a guaranteed life of at least 72 hours. Local windows go to a pool built from 10% of each machine’s DRAM and expire within minutes. If a user returns after the window expires, the model replays the last 128 tokens instead of 40 × 128 = 5,120 to rebuild it approximately.

• **Saves:** persistent cache ≈8× smaller than V4-Flash (≈2× from dropping local windows, ≈4× from smaller global KV). 40× fewer replay tokens on a miss (derived).

• **Why apply it:** local windows are dead minutes after a turn ends, yet V4 kept them on SSD for days. DeepSeek calls bounded replay the cornerstone that turns a costly miss into a cheap one.

#### KVTC, a separate line of NVIDIA work

• **What it does:** JPEG-style coding of idle caches: a PCA rotation, a bit budget per component, then DEFLATE.

• **Saves:** up to ≈20× on FP16 caches of Llama, Mistral and Qwen models in NVIDIA’s tests.

• **Why it stays dashed:** DeepSeek’s cache is already 4-bit, low-rank and shared across layers. The extra gain on 890 B is unmeasured.

*Figure 5. Every multiple in the document on one log scale, labelled by the resource it saves.*

## 3. Stage 5 · The handover

**DeepSeek’s report** stays inside one model. **NVIDIA’s paper** starts from the other end: a router that moves a live session between a small and a large sibling.

### Cross-model KV cache transfer

• **What it does:** a per-head linear map converts the sender’s cache into the receiver’s format. The receiver starts writing without re-reading.

• **How it is built:** strip RoPE from keys; for each receiver layer, stack the k sender layers that predict it best; solve ridge regression in closed form on 500 FineWeb-Edu passages of 1,024 tokens; re-apply the receiver’s RoPE. 47–87 minutes on one 8×H100 node, no gradient training.

• **Saves:** 2.7–25× less time than re-prefill. Qwen3 14B→32B at 32K tokens: 278 ms against 6,975 ms. Storage cost: 1.0–3.4B mapper parameters (4–12 GB) per direction, up to P(P−1) mappers for a fleet of P models.

• **Why apply it:** cost-quality cascades, escalation and mid-conversation switching all read the same context twice. The map removes the second read.

### Quality, fallback and screening

• **Retention:** four of six pairs keep 73–98% of the receiver’s own accuracy; two Ministral pairs fall to 42–44%. Maths degrades first: GSM8K keeps 18% on Llama 3.1 8B→70B.

• **Fallback:** a small MLP recovers +24 to +37 points of HellaSwag retention on the failing pairs, and does slightly worse where linear already works.

• **Screening:** fit quality (R²) misleads across pairs (r = −0.20). Attention-output cosine tracks retention (r = +0.57). Where the error lands matters more than how big it is.

• **Multi-turn:** on Qwen3 14B↔32B, the small-to-large gap widens 1.7 points over ten turns; large-to-small drift runs 0.33 points per turn.

*Figure 6. The five-step mapper and its retention across six same-family pairs, small to large.*

## Putting the five stages on one turn

Stages 1 to 5 describe “the machine”. What a deployment actually experiences is a sequence of turns, each one arriving to find the cache in a different state. Three questions settle: whether the long-range notes survived on SSD, whether the local window survived in host RAM, and whether the router moved the session to another model since the last turn.

The answers are asymmetric by design. The notes that cost 890 bytes per token are guaranteed for 72 hours, so the first question usually answers yes. The local window expires within minutes, so the second usually answers no, and that miss is exactly the one bounded replay makes cheap. The third question is the one neither DeepSeek nor a single-model deployment has to ask, and the one NVIDIA’s mapper exists to answer.

*Figure 7. The four paths a resuming turn can take, and what each one costs.*

## 4. Six things easy to get wrong

These numbers travel fast and arrive mangled. Each row below is a shorthand in circulation and what the underlying reports actually say.

## 5. Question: Could the two papers stack?

This is something I will test at some point. Here is the split between what is known, what follows, and what hasn’t been measured.

• **Known:** NVIDIA tested dense, full-attention, matched-KV pairs in three families and lists sliding-window hybrids as future work. DeepSeek-V4.1-Flash mixes sparse global attention with a 128-token window in every layer and stores a shared 512-channel latent in FP4.

• **Inferred, favourable:** only four DeepSeek layers produce global KV, so a mapper would fit four targets plus indexer K, far fewer than one per layer. Local windows could then be rebuilt with 128-token bounded replay.

• **Inferred, risky:** sparse selection amplifies key errors. A small error can change which 512 entries a layer reads, exactly the error-placement failure NVIDIA documents. Reuse layers make this worse: they can only rewrite their question, so they cannot recover a position their source layer dropped from the shortlist.

• **Unknown:** whether any DeepSeek sibling pair shares KV shape; whether a 4-bit latent maps linearly; how mapping error and replay approximation compound; whether KVTC adds anything below 890 B.

**For a regulated client**

• **Reproducibility:** DeepSeek states that replayed states depend on where the cache hit landed and differ across positions. Identical prompts with different cache histories can yield slightly different outputs. Log cache state wherever outputs must be replayable.

• **Retention:** prompt-derived global KV is designed to persist on SSD for at least 72 hours. Treat it as customer data in derived form under retention and residency policy.

• **Model swaps:** mapped caches keep 73–98% of accuracy on good pairs and far less on others, with arithmetic failing first. Validate each pair on the client’s own tasks before routing.

**Formula for the DeepSeek lineage** 

V4-Flash + CED + CSA2 (cross-layer KV and index reuse, hierarchical indexer) + FP4 main KV + SWA Bounded Replay + Single-Pass mHC + Engram + DSpark − HCA − MTP module − CSA’s overlapping windows = V4.1-Flash: 552B backbone, 8B/16B active, 890 B per token, 1M context.

## 6. How the derived numbers were built

• **890 B reconstruction:** main KV entry = 512 channels × 4 bits (256 B) + 32 one-byte scales = 288 B; indexer-K entry = 128 × 4 bits (64 B) + 4 scales = 68 B. Encoder: 3 Full layers at 2:1 compression; decoder: 1 Full layer at 1:1. (3×288 + 3×68)/2 + 288 + 68 = 890. Assumes MXFP4 (one scale per 32) for indexer K.

• **≈61× indexer:** 1,000,000 positions ÷ 16,384 candidates. Applies to decoder Reindex layers only; the first decoder layer still scans everything.

• **40× and 20× replay:** 40 layers × 128 tokens ÷ 128, and 20 decoder layers × 128 ÷ 128. Counts replayed tokens; FLOP savings will differ.

• **Corroboration:** zartbot’s independent teardown of the released configuration reaches the same 890 B and the same four components, and supplies the V4-Flash comparison used in Figure 3: 41 cache-holding layers, 584 B per main-KV entry (448 channels FP8 + 64 channels BF16 + scales), 68 B per indexer-K entry, 4:1 sequence compression.

• **Waterfall steps in Figure 3:** 4 sources × (584 + 68) ÷ 4 = 652; then 3 encoder sources ÷ 2 plus 1 decoder source ÷ 1 = 1,630; then main-KV entries at 288 B give 3 × 356 ÷ 2 + 356 = 890.

• **Reported, not derived:** 437×, 3.9×, ≈8×, ≈2× prefill, 2× mHC traffic, ≈2× FP4, +25% decode FLOPs (DeepSeek); 2.7–25×, retention and correlation figures (NVIDIA).

## References

**[1]** DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression. Technical report, 2026. [https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)

**[2]** Heo, T. et al. (NVIDIA). Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse. arXiv:2608.03893, 2026. [https://arxiv.org/abs/2608.03893](https://arxiv.org/abs/2608.03893)

**[3]** NVIDIA. KV Cache Transform Coding for Compact Storage in LLM Inference (KVTC). ICLR 2026, arXiv:2511.01815. [https://arxiv.org/abs/2511.01815](https://arxiv.org/abs/2511.01815)

**[4]** DeepSeek-AI. DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence. arXiv:2606.19348, 2026. [https://arxiv.org/abs/2606.19348](https://arxiv.org/abs/2606.19348)

**[5]** Sun, Y. et al. You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO). NeurIPS 2024. [https://arxiv.org/abs/2405.05254](https://arxiv.org/abs/2405.05254)

**[6]** Chen, L. et al. PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention. arXiv:2503.03588. [https://arxiv.org/abs/2503.03588](https://arxiv.org/abs/2503.03588)

**[7]** Xie, Z. et al. mHC: Manifold-Constrained Hyper-Connections. arXiv:2512.24880. [https://arxiv.org/abs/2512.24880](https://arxiv.org/abs/2512.24880)

**[8]** Cheng, X. et al. Conditional Memory via Scalable Lookup (Engram). arXiv:2601.07372. [https://arxiv.org/abs/2601.07372](https://arxiv.org/abs/2601.07372)

**[9]** Cheng, X. et al. DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation. arXiv:2607.05147. [https://arxiv.org/abs/2607.05147](https://arxiv.org/abs/2607.05147)

**[10]** Xu, Y. et al. HiSA: Efficient Hierarchical Indexing for Fine-Grained Sparse Attention. arXiv:2603.28458. [https://arxiv.org/abs/2603.28458](https://arxiv.org/abs/2603.28458)

**[11]** Alvarez, E. et al. Introducing NVFP4 for Efficient and Accurate Low-Precision Inference. NVIDIA blog, 2025. [https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)

**[12]** Rouhani, B. D. et al. Microscaling Data Formats for Deep Learning (MXFP4). arXiv:2310.10537. [https://arxiv.org/abs/2310.10537](https://arxiv.org/abs/2310.10537)

**[13]** zartbot. DeepSeek-V4.1 Flash: Pushing the Limits of KV Cache Compression. Architecture teardown, September 2026. [https://zartbot.github.io/blog/model_arch/dsv41flash_arch/en.html](https://zartbot.github.io/blog/model_arch/dsv41flash_arch/en.html)

**[14]** Bai, Y. et al. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse. arXiv:2603.12201. [https://arxiv.org/abs/2603.12201](https://arxiv.org/abs/2603.12201)

**[15]** Sun, Y. et al. You Only Index Once: Cross-Layer Sparse Attention with Shared Routing. arXiv:2606.06467. [https://arxiv.org/abs/2606.06467](https://arxiv.org/abs/2606.06467)

**[16]** Brandon, W. et al. Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. NeurIPS 2024. [https://arxiv.org/abs/2405.12981](https://arxiv.org/abs/2405.12981)

**[17]** WeLM / Tencent. Building Effective Sparse MoE Models with Moderate Resources (KV-Mirror). [https://welm.weixin.qq.com/en/posts/building-effective-sparse-moe-models-with-moderate-resources/](https://welm.weixin.qq.com/en/posts/building-effective-sparse-moe-models-with-moderate-resources/)

**[18]** VentureBeat. Nvidia says it can shrink LLM memory 20x without changing model weights. March 2026. [https://venturebeat.com/orchestration/nvidia-shrinks-llm-memory-20x-without-changing-model-weights](https://venturebeat.com/orchestration/nvidia-shrinks-llm-memory-20x-without-changing-model-weights)