When you send a long prompt to an LLM API, the service has to store the key-value (KV) cache for your session somewhere. For short conversations, GPU HBM handles it fine. But as context windows stretch to hundreds of thousands of tokens — and agentic workflows keep sessions alive for minutes or hours — the KV cache grows large enough that it has to spill off the GPU entirely.
The standard answer has been a tiered hierarchy: GPU HBM → host DRAM → NVMe SSD. DRAM acts as a warm buffer, catching recently evicted blocks before they land on slower SSD. It's the same logic as CPU cache hierarchies, applied to LLM serving. And for a long time, it worked well enough.
A new paper from researchers at Tsinghua University and Zhipu AI argues that this assumption is now wrong. Their system, FlatCake (arXiv:2610.38355), replaces the hierarchy with a flat architecture that treats DRAM and SSD as peer media in a single unified address space.
The tiered model rests on a key assumption: recently evicted KV blocks are likely to be reused soon, so keeping them in DRAM gives you a fast second chance before they hit SSD. This is the temporal locality assumption — the same one that makes CPU caches work.
Modern LLM workloads increasingly violate it. The authors identify two dominant patterns in production traces:
One-shot blocks: A user sends a long document for summarization. The KV cache is computed once, used once, and never touched again. Staging it through DRAM adds latency without any reuse benefit.
Massive session footprints: An agentic workflow keeps a 200K-token context alive for 30 minutes. The entire session footprint exceeds DRAM capacity, so blocks cycle through DRAM constantly — but the "warm buffer" never actually helps because the working set is too large to fit.
In both cases, DRAM temporal filtering fails. The result is what the authors call "staging contention": DRAM bandwidth gets consumed by blocks moving in and out, while the actual reads and writes that matter — serving live requests — compete for the same bandwidth.
There's a second problem: SSD layout. Traditional tiered systems write KV blocks to SSD in eviction order, not session order. When a session needs to be restored, the blocks for that session are scattered across the SSD, requiring multiple small reads instead of one large sequential read. This read amplification — measured at 6.5x to 6.7x in Mooncake, a widely-used KV cache system that won Best Paper at FAST 2025 — is a significant source of latency. The Mooncake paper describes this architecture in detail.
FlatCake's core insight is that KV blocks are immutable and write-once. Once a block is computed, it never changes. This means you can decide at creation time exactly where a block should live — DRAM or SSD — and never move it again.
The architecture has three main components:
FC-GVA (Global Virtual Address Manager): Manages a unified 64-bit address space that spans both DRAM and SSD. Each block gets a single address that encodes both the medium (DRAM or SSD) and the offset within that medium. Because blocks never migrate, there's no need for eviction-ordered layout or cross-tier movement.
FC-MAS (Medium-Aware Request Scheduler): Maintains separate request queues for DRAM and SSD. By knowing which medium each block lives on (encoded in the address), the scheduler can prevent head-of-line blocking — a slow SSD read doesn't stall a fast DRAM read that's waiting behind it.
FC-CIO (Coalescing Direct I/O Engine): Writes KV blocks to SSD in session-contiguous, layer-major order. When a session needs to be restored, all its blocks are laid out sequentially on disk, enabling a single large read via GPUDirect Storage (GDS) that streams directly into a GPU window — bypassing host DRAM entirely. GDS establishes a direct DMA path between NVMe storage and GPU memory, eliminating the CPU bounce buffer that traditional I/O requires.
The placement policy is straightforward: blocks that are likely to be reused within a short window go to DRAM; everything else goes directly to SSD. Because the address is immutable, there's no migration overhead and no staging contention.
The authors evaluated FlatCake against Mooncake using production traces from DeepSeek-V4-Flash on NVIDIA B200 GPUs, and against Mooncake and Mooncake-GDS using Llama-3.1-70B on A100 GPUs.
On the DeepSeek-V4-Flash production traces:
On Llama-3.1-70B across 8K to 128K context lengths:
The 30-point cache hit rate gap (94% vs 64%) means FlatCake serves far more requests from cached context rather than recomputing — directly translating to lower latency and higher throughput.
The shift toward agentic workflows is changing LLM serving workloads in ways traditional infrastructure wasn't designed for. A single agentic session might involve dozens of tool calls, each appending to a growing context. The KV cache for that session needs to persist across calls, potentially for minutes.
This is exactly the workload profile where tiered storage struggles most. The session footprint is large, the reuse pattern is irregular, and the cost of a cache miss — recomputing hundreds of thousands of tokens — is high.
FlatCake handles this naturally. Because blocks are placed directly on their target medium at creation time, there's no staging overhead. Because SSD layout is session-contiguous, restoration is fast. And because DRAM and SSD are managed as peers rather than a hierarchy, the system adapts to different workload patterns without relying on temporal locality.
KV cache management has become one of the central infrastructure problems in LLM serving. Systems like Mooncake, vLLM's PagedAttention, and NVIDIA's Dynamo have each addressed different aspects of the problem. Research from NVIDIA has also highlighted how compression techniques often collide with serving infrastructure assumptions. FlatCake's contribution is specifically to the storage layer — arguing that the tiered model's assumptions no longer hold and that a flat, placement-at-creation-time approach is better suited to current workloads.
FlatCake is integrated into the SGLang serving framework and includes optimized GPU kernels for H200 and B200 hardware. For teams running long-context or agentic workloads at scale, the results suggest meaningful gains in both latency and throughput without requiring changes to the model itself.
The key design principle — immutable blocks placed once, never migrated — is a useful mental model for KV cache infrastructure more broadly. If your workload doesn't exhibit the temporal locality that tiered storage assumes, you're paying the overhead of a hierarchy that isn't helping you.
Whether FlatCake's approach becomes standard or gets absorbed into existing systems, the underlying argument is worth taking seriously: as context windows grow and agentic sessions become longer, the assumptions baked into LLM serving infrastructure need to be revisited.