The KV Cache Has No ABI The KV cache has no standard ABI, with vLLM's FlashAttention backend alone reporting its cache shape as a four-dimensional tensor that varies by backend, attention variant, and model family, complicating disaggregated inference. The cache is paged, packs K and V together, and supports multiple dimension orders (NHD and HND), requiring negotiation methods for layout. This lack of a universal representation means that moving a KV cache between prefill and decode machines is far more complex than a simple handoff. The KV Cache Has No ABI The last post argued that prefill and decode want different computers, and that the industry has started buying them separately. This one is about the handoff, which sounds like the easy part but it is easier said than done. Stated at the level of a slide, disaggregation is simple. Prefill runs the prompt through the model and produces a KV cache. Ship that cache to the decode machine. Decode generates tokens from it. One artifact crosses one wire, once, per request. The problem is the word “it.” There is no single representation of a KV cache, no standard describing one, and inside the most widely deployed inference engine there are dozens of different answers to what shape the thing is. What is actually in the cache Let’s look at the ordinary case, multi-head or grouped-query attention on a GPU. vLLM’s FlashAttention backend reports its cache shape as a four-dimensional tensor: 1 2 3 4 5 6 7 8 9 10 python vllm/v1/attention/backends/flash attn.py @staticmethod def get kv cache shape num blocks: int, block size: int, num kv heads: int, head size: int, cache dtype str: str = "auto", - tuple int, ... : return num blocks, num kv heads, block size, 2 head size Four things to notice here out which none of them are universal. The cache is paged . It is not a contiguous per-sequence buffer but a pool of fixed-size blocks, with a per-request block table mapping logical positions to physical blocks. Any consumer needs the block table as well as the blocks, and needs to agree on block size . vLLM requires that to be a multiple of 16. K and V are packed together into the trailing dimension, which is why it is 2 head size rather than a separate tensor for each. A consumer that expects separate K and V tensors reads interleaved garbage. Head count is a parameter. Grouped-query attention means num kv heads is smaller than the query head count, by a model-specific ratio. And the dimension order is a choice , where vLLM supports two: 1 2 NHD: num blocks, block size, num kv heads, 2 head size HND: num blocks, num kv heads, block size, 2 head size These hold identical numbers in different memory order. The engine knows they are not interchangeable, which is why the connector interface has a method for asking: 1 2 php def get required kvcache layout - str | None: """Returns "HND", "NHD", or None if no specific layout is required.""" A negotiation method for a layout question, inside one engine, on one vendor’s hardware. 1 That is the shape of the problem in miniature, before any vendor boundary is involved. Why have one shape when you can have dozens. Search vLLM for "def get kv cache shape" and you get dozens of concrete implementations 2, each returning a different tensor shape for a cache the disaggregation slide treats as a single object. They divide along several axes at once. Backend: FlashAttention, FlashInfer, Triton, ROCm, CPU, XPU. Attention variant: standard MHA/GQA, MLA, sparse MLA, sliding-window, differential KV. Model family: DeepSeek V4, Kimi K3, MiniMax M3, each with bespoke variants. And vendor — the same model carries separate implementations under models/inkling/amd/ and models/inkling/nvidia/ . Not all of them are live in any one deployment. vLLM has been migrating into the vllm/v1 tree, so several sit in older model executor paths or in model-specific directories that only load for a single architecture. That helps less than it sounds: most of them are under vllm/v1/attention/ alone, and a consumer does not get to choose which one the producer was configured with. Multi-head Latent Attention is the sharpest divergence, because it changes the rank of the tensor. DeepSeek’s MLA does not store keys and values at all. It stores a compressed latent plus a separate positional component, so the head dimension disappears: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 vllm/v1/attention/backends/mla/flashmla sparse.py @staticmethod def get kv cache shape num blocks: int, block size: int, num kv heads: int, assumed to be 1 for MLA head size: int, cache dtype str: str = "auto", - tuple int, ... : if cache dtype str == "fp8 ds mla": V3.2 main MLA: 656-byte custom storage format. See module docstring. return num blocks, block size, 656 else: return num blocks, block size, head size Read the parameter list against the body. num kv heads is accepted and then ignored, with a comment explaining it is assumed to be 1. The signature describes a world this implementation does not live in. 3 fn:2 And then there is 656. A shape whose last dimension is a byte count That constant is not a number of elements. It is a number of bytes, and the surrounding kernel spells out the contract: 1 2 3 4 5 6 7 8 9 // csrc/libtorch stable/cache kernels.cu if kv cache dtype == "fp8 ds mla" { STD TORCH CHECK kv lora rank == 512, "kv lora rank must be 512 for fp8 ds mla" ; STD TORCH CHECK pe dim == 64, "pe dim must be 64 for fp8 ds mla" ; STD TORCH CHECK kv cache.size 2 == 656 / kv cache.element size , "kv cache.size 2 must be 656 bytes for fp8 ds mla" ; STD TORCH CHECK kv c.element size == 2, ... ; STD TORCH CHECK k pe.element size == 2, ... ; } Together with the kernel comment about how the tiles are written, the 656 bytes decompose: | Region | Contents | Bytes | |---|---|---| | NoPE latent | 512 elements, fp8 | 512 | | Scales | 4 tiles × fp32 scale | 16 | | RoPE component | 64 elements, bf16 | 128 | Total | logical width 576 | 656 | A 576-element logical vector stored in 656 bytes, in three regions of two different dtypes, with quantization scales interleaved between them at tile granularity determined by how a warp writes its lanes. This is not a tensor layout. It is a struct, defined by a CUDA kernel, whose field offsets are load-bearing. You can watch the engine admit it, because there is a dedicated kernel whose entire job is turning the packed form back into something else: 1 2 3 4 5 void cp gather and upconvert fp8 kv cache torch::stable::Tensor const& src cache, // NUM BLOCKS, BLOCK SIZE, 656 torch::stable::Tensor const& dst, // TOT TOKENS, 576 torch::stable::Tensor const& block table, // BATCH, BLOCK INDICES ... 656 in, 576 out. 4 If you receive the source buffer without that kernel, you have 656 bytes per token of something you cannot interpret. Note also what lives in that struct: the RoPE component is stored in the cache as a distinct region . Whether positional encoding is already applied to what you receive, and where it sits, is a per-variant answer. There is no general rule to rely on. The scales are inside the tensor Quantization compounds it, because the scale factors are not metadata alongside the cache. They are packed into the cache. vLLM’s Triton backend returns num blocks, num kv heads, block size, 2 head size for ordinary quantization, but under per-token-head modes it pads the trailing dimension to make room for inline scales, with the padding computed from the ratio of dtype sizes. Same logical cache, same model, same engine, physically different layout depending on a quantization flag. So “the cache is fp8” is not a sufficient description. You need to know the scale granularity, whether scales are inline or separate, where they sit, and what dtype they are. The connector is not a wire format At this point the obvious objection is that vLLM already has an abstraction for shipping caches between instances. It does. Read its actual signatures rather than its name. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 vllm/distributed/kv transfer/kv connector/v1/base.py @abstractmethod def start load kv self, forward context: "ForwardContext", kwargs - None: """Start loading the KV cache from the connector to vLLM's paged KV buffer.""" @abstractmethod def wait for layer load self, layer name: str - None: """Block until the KV for a specific layer is loaded into vLLM's paged buffer.""" @abstractmethod def save kv layer self, layer name: str, kv layer: torch.Tensor, attn metadata: "AttentionMetadata", kwargs - None: """Start saving a layer of KV cache from vLLM's paged buffer to the connector.""" @abstractmethod def get num new matched tokens self, request: "Request", num computed tokens: int - tuple int | None, bool : ... @abstractmethod def update state after alloc self, request: "Request", blocks: "KVCacheBlocks", num external tokens: int : ... @abstractmethod def build connector meta self, scheduler output: SchedulerOutput - KVConnectorMetadata: ... Every one of those signatures is written in vLLM’s own object model. ForwardContext , AttentionMetadata , Request , KVCacheBlocks , SchedulerOutput are Python classes defined inside vLLM. The payload type is torch.Tensor . The docstrings do not say “the KV buffer,” they say “vLLM’s paged KV buffer.” This is a good interface for what it is: a plugin point letting you swap the transport underneath two vLLM instances. UCX, RDMA, TCP, NVMe-oF, object storage. What it is not is a description of bytes on a wire that a foreign runtime could implement against. To satisfy this interface, a Cerebras or Trainium runtime would have to reproduce vLLM’s scheduler objects, its block-table representation, its per-layer save/load lifecycle, and its notion of a forward context. The connector abstracts the transport . It does not abstract the format , because the format was never separated from the engine in the first place. 5 fn:4 graph LR subgraph V "What the connector abstracts" A "vLLM prefill" -- B "KVConnector" B -- C "UCX / RDMA / TCP / NVMe-oF / S3" C -- D "KVConnector" D -- E "vLLM decode" end subgraph X "What it does not abstract" F "layout, rank, packing