A new open-source GUI tool, Local-LLM-Launcher-GUI, lets users run large language models locally via vLLM or llama.cpp without memorizing command-line flags. The browser-based interface provides hardw…
MiniMax released the M3 open-weight model, claiming it costs 5% of Claude Opus per task, achieves 59% on SWE-Bench Pro, and supports a 1-million-token context window at one-twentieth the compute of it…
Z.ai released GLM-5.2, an open-weights AI model under an MIT license, positioning it between Claude Opus 4.7 and 4.8 in performance while costing less than a fifth of Opus on output tokens. The model …
Researchers released IndexCache, a patch for SGLang and vLLM that accelerates sparse attention in DeepSeek-V3.2 and GLM-5 models by reusing index computations across layers, achieving up to 1.82× pref…
Inference engineering, the craft of optimizing GPU operations during AI model inference, can cut costs by up to 80% by addressing the split between prefill and decode phases. Two teams using the same …
A new Rust-based reverse proxy called Sors reorders prompt content to maximize prefix cache hits in LLM inference engines like vLLM and SGLang, improving latency by placing static content before dynam…
Jian Chen, Yesheng Liang, and Zhijian Liu integrated Z Lab's DFlash block diffusion speculative decoding method into SGLang, collaborating with Modal and LMSYS. The team reports up to 4.31x higher thr…
A developer traced a hybrid Mamba-Transformer MoE inference run and found that MoE all-to-all collective stalls dominate the tail latency, with a 69x tail ratio, despite dashboards showing 96% GPU uti…
A developer published token-sec-calc, an open-source Python CLI tool that benchmarks LLM inference throughput, latency, time-to-first-token, and queue wait against any OpenAI-compatible endpoint. The …
Google's DiffusionGemma, an experimental open model using discrete diffusion for text generation, offers a parallel approach that can outperform token-by-token LLMs in throughput-sensitive workloads. …
MiniMax released its M3 multimodal model on NVIDIA's accelerated infrastructure, offering a free public endpoint via NVIDIA's API catalog. The 428-billion-parameter model processes text, images, and v…
Researchers from UC Berkeley and other institutions released CommBench, a benchmark of over 100 GPU communication problems, to evaluate whether large language models can generate correct and efficient…
A developer found that deploying a 70B Llama model with RAG features caused time-to-first-token (TTFT) to jump from 180 ms to 1.4 seconds, as the model recomputed identical attention states for repeat…
The open-source LLM inference landscape in 2025–2026 has split into three specialized tiers: throughput-oriented serving engines for production GPU clusters, portability-focused runtimes for consumer …
NVIDIA's DGX Spark single-node AI appliance can run OpenAI's open-weight GPT-OSS-120B sparse MoE model at ~50 tokens/s using optimized engines like SGLang or llama.cpp, making it viable as a local cod…
A developer released Thaw, an open-source tool that snapshots a live large language model inference session to create forked branches without re-running prefill, addressing the computational waste of …
Liquid AI released LFM2.5-8B-A1B, an edge model designed for fast tool calling on consumer hardware, with a 128K context window and pretraining scaled to 38 trillion tokens. The model, available on Hu…
StepFun released Step 3.7 Flash, a 198B-parameter Mixture-of-Experts vision-language model with 11B active parameters, for enterprise deployment on NVIDIA-accelerated infrastructure. The multimodal AI…
Researchers have developed a stateful inference architecture for multi-agent tool calling that reduces per-turn computational cost from full reprocessing to delta-only updates, achieving 2.1x faster p…
Researchers have developed PAT, an adaptive tensor parallelism method that dynamically reconfigures GPU resource allocation during the generation stage of synchronous RLHF training to address the bott…