HyperCache – Distributed KV-cache quantization and relative RoPE reuse

HyperCache has been introduced as an external distributed KV-cache layer for multi-node LLM inference that combines wire quantization with relative RoPE block reuse, positioning itself as "memcached for KV tensors" beneath existing vLLM, SGLang and TRT-LLM workers. The system targets the recompute break-even threshold: for a 70B model at 320 KB/token in BF16, a 16k-token prefix is about 5 GB, and copying it across PCIe, host memory and a standard TCP stack takes roughly 310-530 ms versus about 389 ms for an H100 to recompute the 16k tokens from scratch. HyperCache addresses the gaps in local prefix caching, disaggregated prefill/decode and host-RAM/TCP offloading by using per-tenant keyed BLAKE2b causal and modular hashes over 16- or 64-token blocks, so reordered RAG chunks such as [Doc A, Doc B] versus [Doc B, Doc A] can still hit the cache.

Distributed KV-cache management, wire quantization, and relative RoPE block reuse for multi-node LLM inference. In multi-turn agent loops, codebase analysis, and RAG workloads, most prompt tokens are shared across requests system prompts, tool definitions, repository files, and retrieved documents . Current serving stacks handle this in one of three ways, each with a specific bottleneck: 1. Local in-engine prefix caching vLLM APC, SGLang RadixAttention : Stores KV blocks in a prefix tree in local GPU VRAM. This works well when consecutive requests hit the same worker and fit in HBM. In a multi-node cluster behind a load balancer, a request routed to Node B cannot read KV blocks computed on Node A. Moreover, because prefix trees require exact token-for-token prefix alignment from index 0, reordered RAG chunks Doc A, Doc B vs Doc B, Doc A miss the cache entirely. 2. Disaggregated prefill/decode architectures Mooncake, DistServe, Splitwise : Separate prefill nodes from decode nodes and stream KV caches over RDMA. Effective at large scale, but requires replacing the serving runtime and cluster scheduler rather than dropping into an existing vLLM deployment. 3. Host-RAM and TCP offloading LMCache, DeepSpeed-FastGen : Evicts KV blocks over PCIe to host memory or streams them over standard kernel TCP sockets. The bottleneck here is the recompute break-even threshold: for a 70B model 320 KB/token in BF16 , a 16k-token prefix is ~5 GB. Copying 5 GB across PCIe, host memory, and a standard TCP stack takes ~310-530 ms, which is roughly the same time an H100 takes to simply recompute the 16k tokens from scratch ~389 ms . HyperCache is designed as an external cache layer "memcached for KV tensors" that sits underneath existing inference workers. Incoming LLM Request | +-------------+-------------+ | | v v Prompt Tokenizer Global Radix / Hash DAG | HMAC-BLAKE2b + Modular | | | Cache Lookup | | v v +---------------------+ +---------------------+ | Compute New Suffix | | Fetch Shared KV | | + 15% CacheBlend | | Prefix Tensors | +----------+----------+ +----------+----------+ | | | +----------+----------+ | | RDMA / TCP-FP8 | | | Layer-Wise Overlap | | +----------+----------+ v v +------------------------------------+ | Decode Generation Loop | | vLLM / SGLang / TRT-LLM Hook | +------------------------------------+ Standard Transformer self-attention is strictly causal: the K and V tensors at token BLAKE2b digests per token block default 16 or 64 tokens : - Chained Causal Hash causal hash : $$H k = \text{BLAKE2b} {K {\text{tenant}}} \text{"CAUSAL"} \parallel H {k-1} \parallel \text{tokens} k $$ Used for exact prefix hits with zero recomputation. If a middle block changes A, B, C - A, X, C , the chain breaks at X and C is never falsely matched as an exact causal hit. - Position-Independent Modular Hash modular hash : $$M k = \text{BLAKE2b} {K {\text{tenant}}} \text{"MODULAR"} \parallel \text{tokens} k $$ Used when a known document or file block appears at a new token offset or after a different predecessor. All hashes are keyed by a per-tenant secret derived via HMAC-SHA256 to prevent cross-tenant timing side-channel attacks where a user infers whether another tenant has cached a proprietary file by measuring TTFT . An explicit "public" namespace is available for global system prompts and shared tool schemas. When a cached block is reused at a new token offset Because 2D RoPE rotations belong to This means HyperCache does not need invasive forward-pass hooks to intercept pre-RoPE keys. It takes standard post-RoPE keys directly from the engine's PagedAttention buffer at max err ~ 4.7e-7 in float32 . To account for cross-chunk causal attention drift in layers hypercache/core/attention engine.py : Layer 0 K/V depends only on token embeddings making rotated Layer 0 KV exact , and the top ~15% of tokens with the largest Layer-0 hidden-state drift are selectively recomputed through deeper layers following the CacheBlend approach . Because network and PCIe links are an order of magnitude slower than HBM3e, blocks are compressed before leaving L1: - L1 Active HBM / Working Memory : Native FP16 / BF16 . - L2 Peer Memory / Pinned Host RAM / Network Wire : Per-token, per-head dynamic FP8 int8 payload + fp16 scales , cutting transfer volume by 1.97x 0.9999 cosine similarity vs raw float32 . - L3 NVMe / Disk Spillover : Symmetric INT4 bit-packed 2 values per uint8 byte + fp16 scales written to .hckv binary block files with a 32-bit CRC32 frame checksum 3.88x compression, 0.993 cosine similarity . Waiting for all 80 layers of a 64k-token KV cache ~10 GB in FP8 to transfer before starting prefill leaves the GPU idle. By pipelining transfers layer-by-layer—fetching Layer Modeled break-even comparison for Llama-3-70B across 80GB H100 nodes 400Gbps RoCEv2, ~45.5 GB/s effective : | Context Length | Raw BF16 KV | Full GPU Recompute | Host-RAM + TCP Uncompressed | HyperCache FP8 + RDMA + Layer Overlap | |---|---|---|---|---| | 16k tokens | 5.00 GB | 389 ms | 530 ms | 22.4 ms | | 32k tokens | 10.00 GB | 942 ms | 1,049 ms | 44.6 ms | | 64k tokens | 20.00 GB | 2,546 ms | 2,096 ms | 95.7 ms | | 128k tokens | 40.00 GB | 7,753 ms | 4,219 ms | 221.9 ms | To keep the repository easy to inspect and test on any machine without requiring an InfiniBand cluster: - Implemented and executed locally zero mocks : - hypercache/core/hasher.py : Chained causal and modular HMAC-BLAKE2b block chunker and hasher. - hypercache/core/quantizer.py : Per-head FP8 and bit-packed INT4 KV quantizer/dequantizer, plus the$SO 2 $ relative RoPE phase rotator. - hypercache/core/attention engine.py : A 4-layer Causal Grouped-Query Attention GQA + RoPE Transformer in NumPy that runs end-to-end forward passes on reordered chunks Sys, DocB, DocA, Query , comparing ground-truth prefill against naive KV stitching, RoPE-only rotation, and RoPE + 15% selective recomputation. - hypercache/core/storage engine.py : Thread-safe L1 - L2 FP8 - L3 INT4 .hckv disk files with CRC32 tamper detection physical tensor storage engine. - hypercache/transport/tcp server.py : Multi-threaded binary TCP socket server and client HyperCacheTCPServer / HyperCacheTCPClient with TCP NODELAY framing that streams compressed KV blocks over the wire and applies relative RoPE shifts on fetch. - hypercache/core/radix dag.py : Thread-safe distributed Radix/modular block directory with LRFU demotion and hot-spot tracking. - - Analytical / simulated for cluster capacity planning: - hypercache/transport/fabric.py and hypercache/cli/auditor.py : Analytical latency models for 400Gbps RoCEv2/InfiniBand GPUDirect RDMA, NVLink P2P, and H100/A100/B200 prefill compute scaling. Native ibverbs / UCX C++ bindings and CUDA stream hooks for vLLM's C++ block manager are planned for v0.2. - Requires Python 3.10+ and numpy . pip install -e . Executes the local 4-layer GQA attention verification on reordered chunks, physical L1/L2/L3 .hckv disk round-trips, live loopback TCP socket transfers, and prints the cluster break-even table: python -m hypercache benchmark --gpu h100 Example output: ========================================================================================== PART 1: LIVE MEASURED NUMERICAL & SYSTEMS VERIFICATION EXECUTED LOCALLY ========================================================================================== A. Multi-Layer Causal GQA Attention + RoPE on Reordered RAG Chunks Sys, DocB, DocA, Q Naive KV Stitching No RoPE / No Recompute : Cosine Sim = 0.7423 Rel L2 Err: 0.715 - BROKEN Relative RoPE Phase Rotation Only 0% Rec. : Cosine Sim = 0.9701 HYPERCACHE RoPE Shift + 15% CacheBlend : Cosine Sim = 0.9783 Rel L2 Err: 0.208 | Saved 85.3% Prefix FLOPs ------------------------------------------------------------------------------------------ B. Physical Tensor Quantization, L3 Disk Spillover & Live TCP Socket Transport Relative RoPE Shift Error Pos 1024 - 8192 : 4.77e-07 Exact SO 2 identity FP8 E4M3 Wire Quantization : 1.97x compression | Cosine Sim: 0.999979 INT4 Grouped L3 Disk .hckv + CRC32 : 3.88x compression | Cosine Sim: 0.993258 67,653 B on disk Live TCP Socket Fetch + On-The-Fly RoPE : 133,188 B frame in 1.14 ms | Cosine Sim vs GT: 0.999979 ========================================================================================== Runs all 9 unit, adversarial, CRC32 tamper detection, multi-threaded TCP concurrency, and GQA attention tests: python tests/test hypercache.py Evaluates cache hit rates and TTFT across Siloed APC, Host-RAM+TCP offload, and HyperCache on a JSONL log file or a built-in multi-agent / RAG workload : python -m hypercache audit --workload multi-agent-swe --nodes 4 --gpu h100 Or replay your own prompt logs: python -m hypercache audit --log-file requests.jsonl --nodes 4 --gpu h100 python -m hypercache top --duration 1.5 MIT