{"slug": "fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches", "title": "Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches", "summary": "Researchers introduced Fathom, a per-query read-depth key scan for sparse decoding over offloaded KV caches, which at one million tokens on Qwen3-8B runs a decode step 1.67x faster in GPU time than the 136-bit scans of Double Sparsity, Loki and SparQ r=32. In the same GPU time as SparQ's 68-bit read (r=16), Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings, and on real coding-agent sessions it reaches the step agreement of the most accurate 136-bit scan at 92 bits. The method stores the 4-bit K cache channel-major as bit planes and spends each query's bit budget by reverse water-filling over variance-weighted channel importance, and it is not faster when the index is resident in GPU memory.", "body_md": "arXiv:2609.17652v1 Announce Type: new \nAbstract: When agentic sessions run to a million tokens with many sessions resident at once, the KV cache and the index that ranks it live in host memory, and the scan that ranks all n keys for a top-k step becomes the traffic that bounds decoding. We present Fathom, a key scan in which each query decides how many bits of each key channel to read. The 4-bit K cache is stored channel-major as bit planes, so a prefix of t planes is exactly the channel's t-bit quantizer, and the query spends its bit budget by reverse water-filling over the variance-weighted importance of its channels. At one million tokens on Qwen3-8B a decode step is 1.67x faster in GPU time than with the 136-bit scans of Double Sparsity, Loki and SparQ r=32, and in the same GPU time as SparQ's 68-bit read (r=16) Fathom reads 18% fewer bytes with lower attention error on six of seven model and context settings. On RULER-style tasks every per-token scan matches exact top-k decoding, and on real coding-agent sessions Fathom reaches the step agreement of the most accurate 136-bit scan at 92 bits. The store is the 4-bit K copy a quantized serving stack already holds, and the method is not faster when the index is resident in GPU memory.", "url": "https://wpnews.pro/news/fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches", "canonical_source": "https://arxiv.org/abs/2609.17652", "published_at": "2026-09-17 04:00:00+00:00", "updated_at": "2026-09-17 04:25:58.192358+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Fathom", "Qwen3-8B", "Double Sparsity", "Loki", "SparQ", "RULER"], "alternates": {"html": "https://wpnews.pro/news/fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches", "markdown": "https://wpnews.pro/news/fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches.md", "text": "https://wpnews.pro/news/fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches.txt", "jsonld": "https://wpnews.pro/news/fathom-per-query-read-depth-for-sparse-decoding-over-offloaded-kv-caches.jsonld"}}