A new benchmark evaluates KV-cache optimization techniques—quantization, pruning, and merging—for long-context LLM serving, finding that compression ratio alone poorly predicts end-to-end performance.…
A developer released TurboQuant, a pip-installable adapter for mlx-lm on Apple Silicon that uses a randomized Hadamard transform for data-oblivious, calibration-free quantization of weights and KV cac…
A 27B-parameter Qwen3.6-Coder model running on a consumer AMD Strix Halo mini-PC successfully fixed a bug in the LLMKube Kubernetes operator by correcting a hardcoded timeout that caused request failu…
Huawei released KVarN, a native KV-cache quantization back end for vLLM that delivers up to 5x more cache capacity and 1.3x the throughput of FP16 while maintaining FP16-level accuracy. The calibratio…
Amazon Web Services announced a new method combining Amazon FSx for Lustre with NVIDIA GPUDirect Storage (GDS) to reduce large language model cold-start loading times from minutes to seconds on AWS GP…
Researchers have developed Speculative KV coding, a lossless compression method that reduces the size of large language model KV caches by up to approximately 4x using a smaller, faster predictor mode…