LLM inference optimization can reduce serving costs by 5-10x and latency by 3-5x, often determining whether an AI feature ships. The bottleneck is memory bandwidth during autoregressive decoding, and …
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…
Baidu released Unlimited OCR, an open-source 3-billion-parameter model that uses Reference Sliding Window Attention to parse multi-page documents in a single pass, eliminating the need for page-by-pag…
A developer built Modelplane, an open-source inference control plane using Crossplane, to manage GPU fleets across clouds, neoclouds, and on-premise environments. The platform allows platform teams to…
UC San Diego researchers developed DFlash, a speculative decoding method that uses a lightweight block diffusion model to draft entire token blocks in parallel, achieving up to 6.08x speedup on Qwen3-…
IBM Research, Red Hat, and NxtGen Cloud Technologies demonstrated that using llm-d to serve AI models on mixed GPU hardware can boost inference speeds by 3 to 5 times and double throughput, enabling e…
SkyPilot launched SkyPilot Endpoints, a production-ready LLM inference system that deploys and manages inference across multiple Kubernetes clusters from a single YAML specification. The system handle…
SGLang achieved a 5x throughput improvement for serving DeepSeek-V4 on NVIDIA GB300 hardware since its Day-0 launch, reaching ~11,200 tok/s/GPU at 50 tok/s/user through kernel and runtime optimization…
NVIDIA announced that DFlash speculative decoding boosts inference performance on Blackwell GPUs by up to 15x for gpt-oss-120b and nearly doubles interactivity for Llama 3.1 8B compared to EAGLE-3. Th…
NVIDIA's confidential computing (CC) on Blackwell GPUs introduces performance overheads ranging from 1% to over 100% for inference and 18–32% for training, according to benchmarks from Tinfoil. The pr…
Baidu released Unlimited-OCR, a one-shot long-horizon parsing model that extends DeepSeek-OCR, on June 22, 2026. The open-source model supports single-image and multi-page PDF parsing with configurabl…
Modelplane, an open-source control plane for AI inference built on Crossplane, is being released to manage GPU clusters as a single inference fleet, handling provisioning, model placement, autoscaling…
Researchers improved SGLang's offloading engine to achieve 7x cheaper prefill costs for DeepSeek V4 Flash on Grace Hopper systems, leveraging high CPU-GPU bandwidth to hide weight transfers behind com…
MoonMath AI open-sourced a bf16 forward attention kernel for AMD MI300X GPUs that outperforms AMD's AITER v3 on every tested shape and rounding mode, achieving up to 1.26x speedup. The kernel, written…
Local AI serving engines like Ollama and vLLM offer different trade-offs between ease of use and throughput, with Ollama ideal for single users and vLLM for high-concurrency production workloads. The …
Modal Labs released state-of-the-art DFlash speculators for Qwen 3.5 and Qwen 3.6 models on Hugging Face, achieving 5-20% additional speedups and enabling Qwen 3.5 122B-A10B to run at over 1000 tok/s …
Modal and Z Lab released DFlash, a speculative decoding model for Qwen 3.5 397B-A17B, achieving over 4.3x throughput versus baseline and 1.5x versus MTP on HumanEval at concurrency 1. The model uses a…
Terminal-Bench introduces Challenges, long-horizon, token-intensive, single-task benchmarks requiring agents to build entire codebases from scratch. Three initial challenges—Rust Compiler Speedup, Inf…
M* (M-Star), a new serving system for multimodal models, matches or beats specialized systems by up to 2.7× on speech and image serving and 12.5× on world-model rollouts. It uses a Walk Graph abstract…
Zhipu AI released GLM-5.2, a large language model with a 1M-token context window, flexible effort levels, and an MIT license, targeting long-horizon coding tasks. The model introduces IndexShare, an a…