Unlocking Non-Uniform KV Cache for Efficient Multi-Turn LLM Serving
Researchers introduced Tangram, a serving system that enables non-uniform Key-Value cache compression for multi-turn large language model inference. The system uses deterministic budget allocation, head group page cluste…