Hey everyone,
As context windows scale to 128k+ tokens, KV cache VRAM consumption becomes a massive bottleneck for real-time inference.
I’ve open-sourced the Unified Latent-State Memory Fabric (UL-SMF), a hardware-software co-designed architecture that compresses standard FP32 KV caches into a 16-dimensional discrete latent space using Finite Scalar Quantization (FSQ).
Live Benchmark Results (CUDA T4 / FP32): • Compression Ratio: 384x reduction (Reduces a 48 MB FP32 block to 0.12 MB)
• Semantic Retention: 94.15% - 95.84% Cosine Similarity on real LLM attention tensors
• Pipeline Latency: ~14.1 ms to 19.6 ms end-to-end CUDA event verified latency
The self-contained Python package allows you to monkey-patch standard Transformer attention layers with two lines of code.
Technical Specification (Zenodo DOI): Live Verification: The Unified Latent-State Fabric: Resolving the Inference Memory Wall via GLRP v2.0 and Aegis-KV | Zenodo Feedback, testing, and benchmarks on longer context models are welcome! Dual-licensed under AGPLv3 with commercial options available.