Post-training quantization (PTQ) of large language models fails under extreme low-bit constraints because dynamic activation outliers scale beyond 1⁰⁵, causing representation collapse, according to a …
A new post-training method called Codec-Gauge, introduced in arXiv:2607.20538v1, learns small orthogonal channel transforms for Transformer KV caches to improve compression fidelity. Across six models…
A new benchmark evaluates KV-cache optimization techniques—quantization, pruning, and merging—for long-context LLM serving, finding that compression ratio alone poorly predicts end-to-end performance.…
The Continue plugin v1.2.20 patches memory leaks, unhandled exceptions, and JCEF message chunking crashes across JetBrains and VS Code adapters, fixing crash vectors that cause sidebar hangs and autoc…
A developer deploying a 70B Llama-3 model on 8x H100s found that scaling from 8k to 32k context windows causes the KV cache to balloon to 10.7 GB per request, forcing memory paging to CPU at 200 concu…