Revolutionizing Memory in AI: A Deep Dive into Compressing Key-Value Cache
Researchers have developed a method to compress key-value caches in AI models using contractive iterated-map codes, achieving 36-54x memory reduction on GPT-2 with a perplexity cost of 11-15%. The technique also enables …