CoMem

A developer has implemented CoMem, a technique from a recent paper that reduces the memory footprint of long-context LLM inference by caching intermediate layer states instead of full KV caches. The a…

Researchers introduced CoMem (Comprehension Memory), a method that leverages the depth division of labor in large language models to enable unbounded-context memory. In evaluations, CoMem achieved 97.…