{"slug": "show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm", "title": "Show HN: Proxima serves 4x more requests with no hardware change on vLLM", "summary": "Proxima, an out-of-tree vLLM plugin implementing STAR-KV low-rank KV cache compression, serves 4x more concurrent requests at 8192 context and boots at 16384 context where plain vLLM refuses to start, on the same Modal L4 24GB GPU with no hardware change. The plugin, developed by Tenosra, compresses the key/value cache via learned soft-thresholding, enabling up to 3.11x more KV-cache blocks and a max context length of 32768 tokens versus 8192 for stock vLLM 0.10.1.1.", "body_md": "Low-rank KV Cache Compression for vLLM\n\nAn out-of-tree vLLM plugin implementing STAR-KV — low-rank KV cache compression via\nlearned soft-thresholding for adaptive rank control\n[Bhatnagar et al., ICML 2026 Spotlight](https://arxiv.org/abs/2606.08382) .\n\nSTAR-KV replaces a transformer's full-dimension key/value cache with a per-layer low-rank projection (head-wise decomposition for K, joint decomposition for V), calibrated offline via short knowledge distillation, and stored in a compressed paged format. This repo reproduces that pipeline and wires it into vLLM's V1 continuous-batching, paged-attention serving stack via dedicated Triton kernels.\n\n**KV-cache compression: fit more concurrent requests, and longer contexts, in the same GPU\nmemory.** Measured end to end on real GPU hardware (Modal L4 24GB):\n\n| plain vLLM | STAR-KV | delta | |\n|---|---|---|---|\n| Max concurrent requests @ 8192 context, zero errors/timeouts | 1 | 4 | 4x |\n| Max concurrent requests @ 16384 context | 0 (refuses to boot) | 2 | boots+serves where plain can't even start |\nGPU KV-cache blocks, same memory budget (`max_model_len=4096` ) |\n756 | 2353 | 3.11x |\nMax `max_model_len` this L4 can boot at |\n8192 | 32768 | boots where plain refuses |\n\nFull breakdown, methodology, and every number behind this table is in\n[Measured results](#measured-results) below.\n\n- You are\n**memory-bound**, not latency-bound: you need more concurrent sequences or longer contexts than stock vLLM fits on the GPU you have, and can tolerate slower per-token decode in exchange for not OOMing or refusing to boot. - You are running\n**long-context workloads**(16k-32k+ tokens) on GPUs where plain vLLM's boot-time KV-cache pre-flight check refuses to start. - You are\n**not** chasing single-request latency — at batch 1, STAR-KV's decode kernel is slower than FlashAttention-2 (launch-overhead bound, 0.36-0.87x); it crosses over to faster than FA2 around batch 8-16 and stays 1.2-1.46x faster at real serving batch sizes.\n\nNot on PyPI. Install from source:\n\n```\ngit clone https://github.com/Tenosra/Proxima\ncd Proxima\npip install -e .   # pins vllm==0.10.1.1, see docs/vllm_patch_notes.md for why\n# 1. Calibrate a STAR-KV checkpoint from a base HF model (offline, one-time, no vLLM involved)\npython -m proxima_vllm.calibration.distill \\\n    --base-model lmsys/longchat-7b-v1.5-32k \\\n    --output ./checkpoints/longchat-7b-star-kv \\\n    --profiles aggressive,balanced,conservative\n\n# 2. Serve it with vLLM\nvllm serve ./checkpoints/longchat-7b-star-kv \\\n    --additional-config '{\"proxima_vllm\": {\"kv_compression\": \"star_kv\", \"rank_profile\": \"balanced\", \"use_triton_kernels\": true}}'\n```\n\nThe reference calibrated checkpoint used for the numbers below is published (private) at\n`iampoppyxx/longchat-7b-starkv-proxima`\n\non the HF Hub.\n\nBaseline is vLLM 0.10.1.1's own V1 `FlashAttentionBackend`\n\n. On L4 (sm89/Ada) vLLM falls back to\n**FlashAttention-2**, not FA3 (FA3 requires Hopper) — that's what every number below is compared\nagainst. This section keeps the headline wins; full raw sweep logs (including the messier\nkernel-level tradeoffs) live in git history if you need them.\n\nSame GPU, same model, distinct (non-shared-prefix) prompts per concurrent request,\n`enable_prefix_caching=False`\n\n, fixed 512-token output, 120s request timeout. Capacity = serves\nevery request without erroring or timing out. `max_concurrency`\n\n= highest concurrency tested with\nzero errors before the next doubling fails.\n\n| context | plain vLLM max concurrency | STAR-KV max concurrency | delta |\n|---|---|---|---|\n| 8192 | 1 | 4 |\n4x |\n| 16384 | 0 (boot refused) | 2 |\nSTAR-KV serves here; plain can't boot at all |\n\nDetail per level:\n\n**8192**: plain vLLM — conc=1 ok (13.1 tok/s, ttft 2.5s); conc=2 both requests time out. Boot log:`Available KV cache memory: 6.99 GiB`\n\n, 14,304 tokens. STAR-KV — conc=1/2/4 all ok (up to 20.2 tok/s, ttft 9.7s at conc=4); conc=8 all time out. Boot log:`Available KV cache memory: 7.05 GiB`\n\n, 44,800 tokens.**16384**: plain vLLM refuses to boot at every`gpu_memory_utilization`\n\ntried (0.9/0.85/0.8/0.75):`ValueError: ... 8.26 GiB KV cache is needed, which is larger than the available KV cache memory (6.99 GiB)`\n\n. STAR-KV boots (`gpu_memory_utilization=0.85`\n\n, 5.95 GiB / 37,792 tokens) and serves conc=1/2 ok (up to 9.8 tok/s, ttft 14.1s); conc=4 all time out.**32768**: plain vLLM boot-refused at every util level (known from the KV-cache-capacity numbers below); STAR-KV concurrency ceiling at this context not yet measured under this protocol.\n\n`balanced`\n\nprofile, same model/checkpoint, same memory budget (`max_model_len=4096`\n\n,\n`gpu_memory_utilization=0.85`\n\n, block size 16):\n\n| GPU blocks | vs stock vLLM | |\n|---|---|---|\n| plain vLLM | 756 | 1x |\n| STAR-KV balanced | 2353 | 3.11x |\n\nAt `max_model_len=32768`\n\n, plain vLLM's boot-time pre-flight check refuses to start at any\n`gpu_memory_utilization`\n\n(needs 16.00 GiB KV cache, only ~5.45 GiB available). STAR-KV boots at\n`gpu_memory_utilization=0.83`\n\nand serves real traffic — 20 concurrent requests at 178.6 tok/s.\nAcross the full context sweep (prompts sized to `max_model_len - 256`\n\n):\n\n| engine | 4096 | 8192 | 16384 | 32768 |\n|---|---|---|---|---|\n| plain vLLM | ok | ok | boot refused |\nboot refused |\n| STAR-KV | ok | ok | ok | ok |\n\nPlain vLLM refuses to boot above 8192 on this L4; STAR-KV boots and serves at every context up to the model's 32768 limit.\n\n`scripts/bench/bench_attn_step.py`\n\n; 32-head MHA, head_size 128, rank 64, fp16. Speedup > 1 means\nSTAR-KV is faster:\n\n| seq len | batch 1 | batch 16 | batch 64 |\n|---|---|---|---|\n| 1024 | 0.36x | 1.23x | 1.40x |\n| 2048 | 0.57x | 1.33x | 1.43x |\n| 4096 | 0.84x | 1.40x | 1.43x |\n| 8192 | 0.87x | 1.44x | 1.46x |\n\n1.2-1.46x faster than FA2 at the batch sizes real continuous-batching serving actually runs at (>=16); the crossover point is around batch 8-16 regardless of sequence length.\n\n`max_model_len=4096`\n\n, `gpu_memory_utilization=0.85`\n\n, sweeping `max_concurrency`\n\n:\n\nSTAR-KV keeps admitting more concurrent decode sequences — mean batch size climbs to ~100-110 before saturating KV cache, vs plain vLLM saturating around batch ~35-47 — and output tok/s converges to near parity (460-483 tok/s) once both are fully saturated, so the capacity win isn't paid for in aggregate throughput at the top end.\n\nSingle long-generation request(s), `concurrency=4`\n\nat `max_model_len=9216`\n\n, `concurrency=2`\n\nat\n`17408`\n\n:\n\n| engine | context | max_tokens | out tok/s | tpot p99 (ms) | kv used |\n|---|---|---|---|---|---|\n| plain | 9216 | 8192 | 7.6 | 525.8 | 1.00 |\n| STAR-KV | 9216 | 8192 | 27.8 | 143.9 | 0.98 |\n| STAR-KV | 17408 | 16384 | 12.3 | 143.8 | 0.93 |\n\nPlain vLLM saturates KV and preempts heavily under sustained long generation at this concurrency (tpot p99 525.8ms); STAR-KV holds steady tpot p99 (~115-144ms across all tested lengths) including at 17408 context, which plain cannot boot at all.\n\nVerified end to end on `lmsys/longchat-7b-v1.5-32k`\n\n, real ShareGPT serving traffic, on Modal L4\nGPU hardware. Multi-model support (Qwen/Mistral/Gemma/Llama-3) is written but not yet validated\non hardware beyond this one model.\n\nApache-2.0. Not affiliated with the STAR-KV paper authors or the vLLM project.", "url": "https://wpnews.pro/news/show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm", "canonical_source": "https://github.com/Tenosra/Proxima", "published_at": "2026-08-11 15:04:01+00:00", "updated_at": "2026-08-11 15:12:03.783305+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "ai-research"], "entities": ["Proxima", "vLLM", "STAR-KV", "Tenosra", "Modal L4 24GB", "FlashAttention-2", "lmsys/longchat-7b-v1.5-32k", "iampoppyxx/longchat-7b-starkv-proxima"], "alternates": {"html": "https://wpnews.pro/news/show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm", "markdown": "https://wpnews.pro/news/show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm.md", "text": "https://wpnews.pro/news/show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm.txt", "jsonld": "https://wpnews.pro/news/show-hn-proxima-serves-4x-more-requests-with-no-hardware-change-on-vllm.jsonld"}}