lmsys/longchat-7b-v1.5-32k

Proxima, an out-of-tree vLLM plugin implementing STAR-KV low-rank KV cache compression, serves 4x more concurrent requests at 8192 context and boots at 16384 context where plain vLLM refuses to start,…