HyperCache – Distributed KV-cache quantization and relative RoPE reuse

Distributed KV-cache management, wire quantization, and relative RoPE block reuse for multi-node LLM inference.

In multi-turn agent loops, codebase analysis, and RAG workloads, most prompt tokens are shared across requests (system prompts, tool definitions, repository files, and retrieved documents).

Current serving stacks handle this in one of three ways, each with a specific bottleneck:

1. Local in-engine prefix caching (vLLM APC, SGLang RadixAttention): Stores KV blocks in a prefix tree in local GPU VRAM. This works well when consecutive requests hit the same worker and fit in HBM. In a multi-node cluster behind a load balancer, a request routed to Node B cannot read KV blocks computed on Node A. Moreover, because prefix trees require exact token-for-token prefix alignment from index 0, reordered RAG chunks ([Doc A, Doc B] vs[Doc B, Doc A] ) miss the cache entirely.
2. Disaggregated prefill/decode architectures (Mooncake, DistServe, Splitwise): Separate prefill nodes from decode nodes and stream KV caches over RDMA. Effective at large scale, but requires replacing the serving runtime and cluster scheduler rather than dropping into an existing vLLM deployment.
3. Host-RAM and TCP off (LMCache, DeepSpeed-FastGen): Evicts KV blocks over PCIe to host memory or streams them over standard kernel TCP sockets. The bottleneck here is the recompute break-even threshold: for a 70B model (320 KB/token in BF16), a 16k-token prefix is ~5 GB. Copying 5 GB across PCIe, host memory, and a standard TCP stack takes ~310-530 ms, which is roughly the same time an H100 takes to simply recompute the 16k tokens from scratch (~389 ms).

HyperCache is designed as an external cache layer ("memcached for KV tensors") that sits underneath existing inference workers.

                        [ Incoming LLM Request ]
                                   |
                     +-------------+-------------+
                     |                           |
                     v                           v
          [ Prompt Tokenizer ]         [ Global Radix / Hash DAG ]
                     |                  (HMAC-BLAKE2b + Modular)
                     |                           |
                     |                      Cache Lookup
                     |                           |
                     v                           v
          +---------------------+      +---------------------+
          |  Compute New Suffix |      |  Fetch Shared KV    |
          |  + 15% CacheBlend   |      |  Prefix Tensors     |
          +----------+----------+      +----------+----------+
                     |                            |
                     |                 +----------+----------+
                     |                 |  RDMA / TCP-FP8     |
                     |                 | (Layer-Wise Overlap)|
                     |                 +----------+----------+
                     v                            v
                 +------------------------------------+
                 |       Decode Generation Loop       |
                 |   (vLLM / SGLang / TRT-LLM Hook)   |
                 +------------------------------------+

Standard Transformer self-attention is strictly causal: the K and V tensors at token BLAKE2b digests per token block (default 16 or 64 tokens):

Chained Causal Hash (causal_hash): $$H_k = \text{BLAKE2b}{K {\text{tenant}}}(\text{"CAUSAL"} \parallel H_{k-1} \parallel \text{tokens}_k)$$ Used for exact prefix hits with zero recomputation. If a middle block changes ([A, B, C] ->[A, X, C] ), the chain breaks atX andC is never falsely matched as an exact causal hit. #

Position-Independent Modular Hash (modular_hash): $$M_k = \text{BLAKE2b}{K {\text{tenant}}}(\text{"MODULAR"} \parallel \text{tokens}_k)$$ Used when a known document or file block appears at a new token offset or after a different predecessor.

All hashes are keyed by a per-tenant secret derived via HMAC-SHA256 to prevent cross-tenant timing side-channel attacks (where a user infers whether another tenant has cached a proprietary file by measuring TTFT). An explicit "public" namespace is available for global system prompts and shared tool schemas.

When a cached block is reused at a new token offset

Because 2D RoPE rotations belong to

This means HyperCache does not need invasive forward-pass hooks to intercept pre-RoPE keys. It takes standard post-RoPE keys directly from the engine's PagedAttention buffer at max_err ~ 4.7e-7 in float32).

To account for cross-chunk causal attention drift in layers hypercache/core/attention_engine.py): Layer 0 K/V depends only on token embeddings (making rotated Layer 0 KV exact), and the top ~15% of tokens with the largest Layer-0 hidden-state drift are selectively recomputed through deeper layers (following the CacheBlend approach).

Because network and PCIe links are an order of magnitude slower than HBM3e, blocks are compressed before leaving L1:

• L1 (Active HBM / Working Memory): NativeFP16 /BF16 .
• L2 (Peer Memory / Pinned Host RAM / Network Wire): Per-token, per-head dynamicFP8 (int8 payload +fp16 scales), cutting transfer volume by1.97x (>0.9999 cosine similarity vs raw float32).
• L3 (NVMe / Disk Spillover): SymmetricINT4 bit-packed (2 values peruint8 byte +fp16 scales) written to.hckv binary block files with a 32-bitCRC32 frame checksum (3.88x compression,>0.993 cosine similarity).

Waiting for all 80 layers of a 64k-token KV cache (~10 GB in FP8) to transfer before starting prefill leaves the GPU idle. By pipelining transfers layer-by-layer—fetching Layer

Modeled break-even comparison for Llama-3-70B across 80GB H100 nodes (400Gbps RoCEv2, ~45.5 GB/s effective):

To keep the repository easy to inspect and test on any machine without requiring an InfiniBand cluster:

Implemented and executed locally (zero mocks): - hypercache/core/hasher.py : Chained causal and modularHMAC-BLAKE2b block chunker and hasher. #

hypercache/core/quantizer.py : Per-headFP8 and bit-packedINT4 KV quantizer/dequantizer, plus the$SO(2)$ relative RoPE phase rotator. #

hypercache/core/attention_engine.py : A 4-layer Causal Grouped-Query Attention (GQA) + RoPE Transformer in NumPy that runs end-to-end forward passes on reordered chunks ([Sys, DocB, DocA, Query] ), comparing ground-truth prefill against naive KV stitching, RoPE-only rotation, and RoPE + 15% selective recomputation. #

hypercache/core/storage_engine.py : Thread-safe L1 -> L2 (FP8 ) -> L3 (INT4``.hckv disk files withCRC32 tamper detection) physical tensor storage engine. #

hypercache/transport/tcp_server.py : Multi-threaded binary TCP socket server and client (HyperCacheTCPServer /HyperCacheTCPClient ) withTCP_NODELAY framing that streams compressed KV blocks over the wire and applies relative RoPE shifts on fetch. #

hypercache/core/radix_dag.py : Thread-safe distributed Radix/modular block directory with LRFU demotion and hot-spot tracking. #

Analytical / simulated for cluster capacity planning: - hypercache/transport/fabric.py andhypercache/cli/auditor.py : Analytical latency models for 400Gbps RoCEv2/InfiniBand GPUDirect RDMA, NVLink P2P, and H100/A100/B200 prefill compute scaling. Nativeibverbs / UCX C++ bindings and CUDA stream hooks for vLLM's C++ block manager are planned for v0.2. #

Requires Python 3.10+ and numpy.

pip install -e .

Executes the local 4-layer GQA attention verification on reordered chunks, physical L1/L2/L3 .hckv disk round-trips, live loopback TCP socket transfers, and prints the cluster break-even table:

python -m hypercache benchmark --gpu h100

Example output:

A. Multi-Layer Causal GQA Attention + RoPE on Reordered RAG Chunks [Sys, DocB, DocA, Q] * Naive KV Stitching (No RoPE / No Recompute) : Cosine Sim = 0.7423 (Rel L2 Err: 0.715 -> BROKEN) * Relative RoPE Phase Rotation Only (0% Rec.) : Cosine Sim = 0.9701 * HYPERCACHE (RoPE Shift + 15% CacheBlend) : Cosine Sim = 0.9783 (Rel L2 Err: 0.208 | Saved 85.3% Prefix FLOPs) #

B. Physical Tensor Quantization, L3 Disk Spillover & Live TCP Socket Transport * Relative RoPE Shift Error (Pos 1024 -> 8192): 4.77e-07 (Exact SO(2) identity) * FP8 (E4M3) Wire Quantization : 1.97x compression | Cosine Sim: 0.999979 * INT4 (Grouped) L3 Disk (.hckv + CRC32) : 3.88x compression | Cosine Sim: 0.993258 (67,653 B on disk)

Runs all 9 unit, adversarial, CRC32 tamper detection, multi-threaded TCP concurrency, and GQA attention tests:

python tests/test_hypercache.py

Evaluates cache hit rates and TTFT across Siloed APC, Host-RAM+TCP offload, and HyperCache on a JSONL log file (or a built-in multi-agent / RAG workload):

python -m hypercache audit --workload multi-agent-swe --nodes 4 --gpu h100 python -m hypercache audit --log-file requests.jsonl --nodes 4 --gpu h100 python -m hypercache top --duration 1.5

MIT