Researchers from Hao AI Lab introduced JetFlow, a speculative decoding framework that breaks the scaling ceiling of autoregressive LLMs by combining one-forward drafting efficiency with branch-wise ca…
A new open-source GUI tool, Local-LLM-Launcher-GUI, lets users run large language models locally via vLLM or llama.cpp without memorizing command-line flags. The browser-based interface provides hardw…
MiniMax released the M3 open-weight model, claiming it costs 5% of Claude Opus per task, achieves 59% on SWE-Bench Pro, and supports a 1-million-token context window at one-twentieth the compute of it…
Z.ai released GLM-5.2, an open-weights AI model under an MIT license, positioning it between Claude Opus 4.7 and 4.8 in performance while costing less than a fifth of Opus on output tokens. The model …
By 2026, AI workloads will consume roughly 40% of enterprise Kubernetes clusters, but the default scheduler is ill-suited for GPU-intensive tasks, leading to 30-45% GPU utilization rates and wasted co…
The PyTorch Foundation has opened nominations for its 2026 Contributor Awards, recognizing individuals who strengthen projects like PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors through techn…
VLLM, a model-serving engine for large language models, introduced a small op-level IR to resolve the tension between acting as a compiler target and a hand-tuned kernel dispatcher. The IR allows vLLM…
Researchers released IndexCache, a patch for SGLang and vLLM that accelerates sparse attention in DeepSeek-V3.2 and GLM-5 models by reusing index computations across layers, achieving up to 1.82× pref…
Researchers introduced cuTile Rust, a tile-based system for safe, idiomatic GPU kernel authoring in Rust that extends Rust's ownership discipline to GPU kernels. On the NVIDIA B200 GPU, cuTile Rust ac…
Researchers have discovered that the key/value cache in large language models can be edited and composed like notebook notes, enabling direct modification of model outputs without full recomputation. …
Ray Data LLM, a library for large-scale batch inference, achieves 2x throughput over vLLM's synchronous LLM engine in production-scale workloads by optimizing hardware utilization and providing fault …
Amazon Web Services announced container image caching for Amazon SageMaker AI inference, reducing end-to-end latency by up to 2x for generative AI models during scale-out events by eliminating contain…
JetBrains open-sourced Mellum2, a 12B Mixture-of-Experts coding model under Apache 2.0, designed for air-gapped and compliance-locked environments where external API calls are prohibited. The model us…
A developer deployed a Llama 3 inference endpoint on Oracle Cloud Infrastructure's OKE cluster using vLLM and NVIDIA A10 GPUs in about 20 minutes. The setup cost $1.52/hr on-demand or $0.46/hr preempt…
Flama 2.0 introduces first-class support for generative AI, enabling users to download, package, and serve large language models (LLMs) via a single command line. The framework allows fetching models …
Inference engineering, the craft of optimizing GPU operations during AI model inference, can cut costs by up to 80% by addressing the split between prefill and decode phases. Two teams using the same …
A developer built Trigix, an open-source (MIT), self-hostable workflow automation platform with a Rust execution engine and AI nodes that can run against local models. The platform features ~180 node …
Anthropic CEO Dario Amodei said reinforcement learning shows the same log-linear scaling as pre-training, but RL system efficiency is critical to afford enough training. Experiments on open models sho…
A new Rust-based reverse proxy called Sors reorders prompt content to maximize prefix cache hits in LLM inference engines like vLLM and SGLang, improving latency by placing static content before dynam…
Enterprises are burning millions on cloud LLM tokens due to inefficient agentic systems, prompting a shift to open-weight models on dedicated infrastructure to eliminate marginal token costs and achie…