Baidu released Unlimited-OCR, a one-shot long-horizon parsing model that extends DeepSeek-OCR, on June 22, 2026. The open-source model supports single-image and multi-page PDF parsing with configurabl…
Researchers improved SGLang's offloading engine to achieve 7x cheaper prefill costs for DeepSeek V4 Flash on Grace Hopper systems, leveraging high CPU-GPU bandwidth to hide weight transfers behind com…
MoonMath AI open-sourced a bf16 forward attention kernel for AMD MI300X GPUs that outperforms AMD's AITER v3 on every tested shape and rounding mode, achieving up to 1.26x speedup. The kernel, written…
Local AI serving engines like Ollama and vLLM offer different trade-offs between ease of use and throughput, with Ollama ideal for single users and vLLM for high-concurrency production workloads. The …
Modal Labs released state-of-the-art DFlash speculators for Qwen 3.5 and Qwen 3.6 models on Hugging Face, achieving 5-20% additional speedups and enabling Qwen 3.5 122B-A10B to run at over 1000 tok/s …
Modal and Z Lab released DFlash, a speculative decoding model for Qwen 3.5 397B-A17B, achieving over 4.3x throughput versus baseline and 1.5x versus MTP on HumanEval at concurrency 1. The model uses a…
Terminal-Bench introduces Challenges, long-horizon, token-intensive, single-task benchmarks requiring agents to build entire codebases from scratch. Three initial challenges—Rust Compiler Speedup, Inf…
M* (M-Star), a new serving system for multimodal models, matches or beats specialized systems by up to 2.7× on speech and image serving and 12.5× on world-model rollouts. It uses a Walk Graph abstract…
Zhipu AI released GLM-5.2, a large language model with a 1M-token context window, flexible effort levels, and an MIT license, targeting long-horizon coding tasks. The model introduces IndexShare, an a…
A new open-source GUI tool, Local-LLM-Launcher-GUI, lets users run large language models locally via vLLM or llama.cpp without memorizing command-line flags. The browser-based interface provides hardw…
MiniMax released the M3 open-weight model, claiming it costs 5% of Claude Opus per task, achieves 59% on SWE-Bench Pro, and supports a 1-million-token context window at one-twentieth the compute of it…
Z.ai released GLM-5.2, an open-weights AI model under an MIT license, positioning it between Claude Opus 4.7 and 4.8 in performance while costing less than a fifth of Opus on output tokens. The model …
Researchers released IndexCache, a patch for SGLang and vLLM that accelerates sparse attention in DeepSeek-V3.2 and GLM-5 models by reusing index computations across layers, achieving up to 1.82× pref…
Inference engineering, the craft of optimizing GPU operations during AI model inference, can cut costs by up to 80% by addressing the split between prefill and decode phases. Two teams using the same …
A new Rust-based reverse proxy called Sors reorders prompt content to maximize prefix cache hits in LLM inference engines like vLLM and SGLang, improving latency by placing static content before dynam…
Jian Chen, Yesheng Liang, and Zhijian Liu integrated Z Lab's DFlash block diffusion speculative decoding method into SGLang, collaborating with Modal and LMSYS. The team reports up to 4.31x higher thr…
A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
A developer published token-sec-calc, an open-source Python CLI tool that benchmarks LLM inference throughput, latency, time-to-first-token, and queue wait against any OpenAI-compatible endpoint. The …
Google's DiffusionGemma, an experimental open model using discrete diffusion for text generation, offers a parallel approach that can outperform token-by-token LLMs in throughput-sensitive workloads. …
MiniMax released its M3 multimodal model on NVIDIA's accelerated infrastructure, offering a free public endpoint via NVIDIA's API catalog. The 428-billion-parameter model processes text, images, and v…