[Research/Code] Verified 384x KV-Cache Compression with Linear Complexity using Finite Scalar Quantization (UL-SMF)

A developer has open-sourced the Unified Latent-State Memory Fabric (UL-SMF), a hardware-software co-designed architecture that compresses standard FP32 KV caches into a 16-dimensional discrete latent space using Finite Scalar Quantization (FSQ), achieving a 384x compression ratio (reducing a 48 MB FP32 block to 0.12 MB) with 94.15% to 95.84% cosine similarity retention on real LLM attention tensors and 14.1 ms to 19.6 ms end-to-end CUDA event verified latency on a CUDA T4 GPU. The self-contained Python package allows monkey-patching standard Transformer attention layers with two lines of code, and is dual-licensed under AGPLv3 with commercial options.

Hey everyone, As context windows scale to 128k+ tokens, KV cache VRAM consumption becomes a massive bottleneck for real-time inference. I’ve open-sourced the Unified Latent-State Memory Fabric UL-SMF , a hardware-software co-designed architecture that compresses standard FP32 KV caches into a 16-dimensional discrete latent space using Finite Scalar Quantization FSQ . Live Benchmark Results CUDA T4 / FP32 : • Compression Ratio: 384x reduction Reduces a 48 MB FP32 block to 0.12 MB • Semantic Retention: 94.15% - 95.84% Cosine Similarity on real LLM attention tensors • Pipeline Latency: ~14.1 ms to 19.6 ms end-to-end CUDA event verified latency The self-contained Python package allows you to monkey-patch standard Transformer attention layers with two lines of code. Technical Specification Zenodo DOI : Live Verification: The Unified Latent-State Fabric: Resolving the Inference Memory Wall via GLRP v2.0 and Aegis-KV | Zenodo https://zenodo.org/records/21896117 Feedback, testing, and benchmarks on longer context models are welcome Dual-licensed under AGPLv3 with commercial options available.