{"slug": "deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget", "title": "DeepSeek-V4.1-Flash: Frontier Agents on a Smaller Memory Budget", "summary": "DeepSeek released DeepSeek-V4.1-Flash, a 552B-parameter MoE model built on a new Causal Encoder-Decoder (CED) architecture that activates about 8B parameters per token during prefill and 16B during decoding. The model's Compressed Sparse Attention 2 (CSA2), SWA Bounded Replay, and FP4 KV quantization cut the global KV cache to 890 bytes per token, roughly one quarter of DeepSeek-V4-Flash's, while retaining a 1M-token context, native vision, and agentic capabilities after training on 45T multimodal tokens.", "body_md": "## DeepSeek-V4.1-Flash: Frontier Agents on a Smaller Memory Budget\n\nA lot of recent progress in language models has focused on better reasoning, larger reinforcement-learning runs, and increasingly sophisticated agent training. One of the many remaining bottlenecks is the context. Coding agents, research agents, and tool-using systems repeatedly process hundreds of thousands of tokens while generating comparatively little output. At that point, prefill compute and the KV cache become the most important infrastructure problems.\n\n**DeepSeek-V4.1-Flash** is essentially an attempt to redesign the Transformer around that workload. Its 552B-parameter MoE backbone uses a new **Causal Encoder-Decoder (CED)** architecture, activating only about 8B parameters per token during prefill and 16B during decoding. **Compressed Sparse Attention 2 (CSA2)** shares KV representations and sparse-attention decisions across layers; **SWA Bounded Replay** avoids persistently storing sliding-window KV states; and FP4 KV quantization pushes the global cache down to only **890 bytes per token — about one quarter of DeepSeek-V4-Flash’s**. The model still supports a 1M-token context, native vision, and strong agentic capabilities. It is trained on 45T multimodal tokens, with much of the improvement coming from scaling automatically generated agent tasks, environments, and rollouts.\n\n### The Architecture\n\n#### Causal Encoder-Decoder\n\nCED splits the 40-layer MoE transformer into a 20-layer causal encoder and a 20-layer decoder. The encoder is causal, so generation stays autoregressive. The key change is where the decoder gets its global KV cache.\n\nNormally every layer computes its own keys and values from its own hidden states, so prefill requires running the prompt through the entire network. In CED, each decoder layer instead projects its global KV from **the final encoder hidden state**, using layer-specific projection weights. The decoder can therefore be skipped during prefill while still obtaining a distinct KV cache for every decoder layer.\n\nThis roughly halves prefill compute: the model activates about **8B parameters** per token during prefill, versus **16B** during decoding. Sliding-window attention, with a 128-token window, is still computed independently at every layer. The trade-off is representational depth: decoder global KV comes from a single encoder representation rather than from hidden states progressively refined through the decoder itself.\n\n#### CSA2: the layer and byte dimensions\n\n[DeepSeek-V4](https://andlukyane.com/blog/paper-review-deepseekv4) introduced **CSA**, which compresses the KV cache along the sequence dimension and then applies sparse selection over the compressed entries. CSA2 keeps that and adds the layer dimension. The first two layers are pure sliding-window; every remaining layer is assigned one of three static modes:\n\n- **Full** computes its own main KV, its own indexer keys, and fresh Top-K indices.\n- **Reindex** reuses a preceding layer’s main KV and indexer keys, but rescores them with its own indexer query to pick its own Top-K.\n- **Reuse** takes both the main KV and the Top-K selection from a preceding layer and only runs sparse attention.\n\nThese save different resources. Sharing KV reduces memory; sharing Top-K selections also avoids repeatedly scoring the full history.\n\nIn the encoder, the remaining 18 layers form three groups of one Full layer and five Reuse layers. In the decoder, the 20 layers form five groups: the first starts with a Full layer, while the other four start with Reindex layers. As a result, **one decoder layer creates the main KV shared by all twenty decoder layers**, while several layers can still choose different sparse positions.\n\nThe cost of the rescoring layers is bounded by a **Hierarchical Sparse Indexer**. The first Full decoder layer builds a candidate pool of at most 16384 positions; later Reindex layers search only within that pool rather than the entire context. This bounds their indexing cost, although it also means positions excluded from the initial pool cannot be recovered later.\n\nFinally, the shared main KV cache is quantized to FP4 using quantization-aware training. The more sensitive sliding-window cache remains at FP8.\n\n#### SWA Bounded Replay\n\nEach layer’s window is small, but there are 40 of them, and prefix caching means that state must survive between turns, which pushes it onto SSD or host memory.\n\nSWA Bounded Replay stops persisting it: on a cache hit the model rebuilds the window state by replaying a single window’s worth of tokens instead of the window multiplied by the layer count that a faithful reconstruction would need. The result is close but not identical, because a layer’s window should hold keys and values derived from hidden states that themselves depended on tokens now outside the replay span. The paper calls the quality impact negligible.\n\n#### Other Architectural Components\n\nV4.1-Flash keeps DeepSeek’s highly sparse MoE design: each layer has **384 routed experts plus one shared expert**, with only six routed experts activated per token. This lets the 552B backbone provide large capacity while keeping per-token compute relatively small.\n\nIt also uses **Single-Pass [mHC](https://andlukyane.com/blog/paper-review-mhc)**, an optimized version of the manifold-constrained Hyper-Connections introduced in DeepSeek-V4. mHC maintains several parallel residual streams and learns how to mix them around each Transformer block. V4.1 shifts one set of mixing coefficients by a layer, removing a sequential dependency and allowing the operation to be fused into a single **Mega-mHC** kernel with substantially less activation memory traffic.\n\n**Engram** is a 196B-parameter conditional memory separate from the 552B backbone. Two Engram modules hash local 2-, 3-, and 4-token patterns into huge learned embedding tables and inject the retrieved vectors through a learned gate. Only a tiny fraction of this memory is accessed for each token, so Engram adds enormous parameter capacity without corresponding matrix-multiplication cost — effectively separating **memorization capacity from compute**.\n\nFinally, **DSpark** accelerates generation through speculative decoding. A smaller three-stage MoE drafter predicts blocks of up to five future tokens, while the main model verifies them; a confidence head controls how aggressively the drafts are accepted. Unlike the architectural changes above, DSpark primarily improves inference throughput rather than model quality.\n\n### Post-training\n\nPost-training follows the standard SFT → RL → on-policy distillation pipeline, and DeepSeek explicitly states that it introduces no new post-training algorithms. Instead, essentially all of the reported gains come from scaling the quantity, diversity, and verifiability of synthesized tasks and agent environments. Coding environments are automatically constructed from difficult coding-agent sessions submitted by employees and external partners, as well as public GitHub repositories, with specialized agents building and verifying each task. The report also gives an unusual glimpse into RL failure modes: agents attempt reward hacking, exploit real vulnerabilities in their sandboxes, delete critical binaries and system files, and sometimes even remove the filesystem entirely.\n\n### Experiments\n\nThe base model is positioned as comparable to DeepSeek-V4-Pro-Base at a third of the total parameters and a quarter of the activated ones. It leads all three code benchmarks and scores **74.1 on MMLU-Pro** against V4-Pro-Base’s **73.5**, but the world knowledge and long-context retrieval favor the larger model.\n\nThe instruct model is much stronger on agents. At maximum reasoning effort, V4.1-Flash reaches **90.6 on Terminal-Bench 2.1**, ahead of Opus 5’s 89.1, and **74.2 on DeepSWE v1.1**, essentially tied with Opus 5 at 74.0.\n\nThe weakest part is the lack of **architectural ablations**. CED is described as preserving baseline quality, FP4 KV quantization as causing only marginal degradation, and bounded replay as having little effect on responses, yet the report does not provide controlled benchmark tables isolating these changes.\n\n### Conclusions\n\nMany modern LLMs add one or two improvements at a time. [MiniMax’s MSA](https://andlukyane.com/blog/paper-review-minimax-sparse-attention) keeps exact softmax attention and sparsifies only which blocks it runs over; [Kimi K3](https://andlukyane.com/blog/paper-review-kimik3) changes what is cached, replacing three of every four global layers with linear attention carrying a fixed-size state. Quantization is usually added on afterward, and between-turn persistence is left to model-agnostic infrastructure. V4.1-Flash combines many ideas, among others the Causal Encoder-Decoder design, which makes **prefill and decoding fundamentally asymmetric workloads inside the model itself** rather than merely optimizing the same Transformer stack at serving time.\n\nThe result is not simply a “new best model”. DeepSeek’s own evaluations show a trade-off. V4.1-Flash has higher metrics on Terminal-Bench 2.1, but lower on Terminal-Bench 3.0/4.0 and HLE. These are also model vs. harness evaluations. The important result is that an open-weight model can remain in roughly the same capability regime while being designed around dramatically cheaper long-context inference. That efficiency/capability trade-off is probably the most important idea in the paper.\n\n[paperreview](https://andlukyane.com/tag/paperreview)\n\n[deeplearning](https://andlukyane.com/tag/deeplearning)\n\n[llm](https://andlukyane.com/tag/llm)\n\n[moe](https://andlukyane.com/tag/moe)\n\n[attention](https://andlukyane.com/tag/attention)\n\n[multimodal](https://andlukyane.com/tag/multimodal)\n\n[agent](https://andlukyane.com/tag/agent)", "url": "https://wpnews.pro/news/deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget", "canonical_source": "https://andlukyane.com/blog/paper-review-deepseek-v41-flash", "published_at": "2026-09-21 00:00:00+00:00", "updated_at": "2026-09-21 09:54:16.443051+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents", "ai-infrastructure"], "entities": ["DeepSeek", "DeepSeek-V4.1-Flash", "DeepSeek-V4-Flash", "Causal Encoder-Decoder", "Compressed Sparse Attention 2", "SWA Bounded Replay", "Hierarchical Sparse Indexer"], "alternates": {"html": "https://wpnews.pro/news/deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget", "markdown": "https://wpnews.pro/news/deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget.md", "text": "https://wpnews.pro/news/deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget.txt", "jsonld": "https://wpnews.pro/news/deepseek-v4-1-flash-frontier-agents-on-a-smaller-memory-budget.jsonld"}}