Modal Labs released state-of-the-art DFlash speculators for Qwen 3.5 and Qwen 3.6 models on Hugging Face, achieving 5-20% additional speedups and enabling Qwen 3.5 122B-A10B to run at over 1000 tok/s …
A developer building vLLM from source on an AWS g5 instance with Ubuntu 26.04 and Python 3.14 encountered multiple version-skew, driver, and toolchain issues, including a pitfall where missing nvidia-…
A critical bug in Google's Gemma 4 causes it to malform tool calls under real load, affecting vLLM, llama.cpp, Ollama, and oobabooga. A developer open-sourced a diagnosis, repair, and experimental LoR…
OpenCode, an open-source terminal-first coding agent, now supports self-hosted large language models via OpenAI-compatible APIs, enabling users to connect it to a vLLM server running NVIDIA's Nemotron…
Profile v2.1.4, a physics-aware optimizer for vLLM inference servers, achieved a 15x throughput increase from 31 to 470 tok/s and a 93% cost reduction on an NVIDIA A100 GPU. The tool uses roofline mat…
Depot now supports SOCI v2 for container builds, enabling lazy-pulling of images to drastically reduce startup times. The feature generates a SOCI index during the build process, allowing containers t…
M* (M-Star), a new serving system for multimodal models, matches or beats specialized systems by up to 2.7× on speech and image serving and 12.5× on world-model rollouts. It uses a Walk Graph abstract…
The Continue plugin v1.2.20 patches memory leaks, unhandled exceptions, and JCEF message chunking crashes across JetBrains and VS Code adapters, fixing crash vectors that cause sidebar hangs and autoc…
Alibaba's Qwen3-80B and Qwen3-4B models were successfully co-located on a single NVIDIA DGX Spark using vLLM containers behind a LiteLLM proxy, but the 80B model's inability to emit tool calls in auto…
Seven open-source AI projects—Ollama, Open WebUI, Browser Use, vLLM, Unsloth, CrewAI, and Continue—are reshaping production software development in June 2026. Ollama, with 174,000+ GitHub stars, now o…
Google Cloud and Anyscale announced optimizations for Ray Serve LLM on Google Kubernetes Engine (GKE) that deliver up to 5x higher throughput and 8x lower latency for large language model inference. T…
Zhipu AI released GLM-5.2, a large language model with a 1M-token context window, flexible effort levels, and an MIT license, targeting long-horizon coding tasks. The model introduces IndexShare, an a…
Ray Serve LLM, in partnership with Google Kubernetes Engine, announced major performance improvements achieving up to 4.4x higher throughput on prefill-heavy workloads and 24x higher on decode-heavy w…
Nexus Labs enabled speculative decoding in vLLM for a fine-tuned 8B model, achieving a 1.9x throughput gain, but discovered that greedy decoding with a draft model is not bit-identical to greedy decod…
Researchers from Hao AI Lab introduced JetFlow, a speculative decoding framework that breaks the scaling ceiling of autoregressive LLMs by combining one-forward drafting efficiency with branch-wise ca…
A new open-source GUI tool, Local-LLM-Launcher-GUI, lets users run large language models locally via vLLM or llama.cpp without memorizing command-line flags. The browser-based interface provides hardw…
MiniMax released the M3 open-weight model, claiming it costs 5% of Claude Opus per task, achieves 59% on SWE-Bench Pro, and supports a 1-million-token context window at one-twentieth the compute of it…
Z.ai released GLM-5.2, an open-weights AI model under an MIT license, positioning it between Claude Opus 4.7 and 4.8 in performance while costing less than a fifth of Opus on output tokens. The model …
By 2026, AI workloads will consume roughly 40% of enterprise Kubernetes clusters, but the default scheduler is ill-suited for GPU-intensive tasks, leading to 30-45% GPU utilization rates and wasted co…
The PyTorch Foundation has opened nominations for its 2026 Contributor Awards, recognizing individuals who strengthen projects like PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors through techn…