The PyTorch Foundation has opened nominations for its 2026 Contributor Awards, recognizing individuals who strengthen projects like PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors through techn…
VLLM, a model-serving engine for large language models, introduced a small op-level IR to resolve the tension between acting as a compiler target and a hand-tuned kernel dispatcher. The IR allows vLLM…
Researchers released IndexCache, a patch for SGLang and vLLM that accelerates sparse attention in DeepSeek-V3.2 and GLM-5 models by reusing index computations across layers, achieving up to 1.82× pref…
Researchers introduced cuTile Rust, a tile-based system for safe, idiomatic GPU kernel authoring in Rust that extends Rust's ownership discipline to GPU kernels. On the NVIDIA B200 GPU, cuTile Rust ac…
Researchers have discovered that the key/value cache in large language models can be edited and composed like notebook notes, enabling direct modification of model outputs without full recomputation. …
Ray Data LLM, a library for large-scale batch inference, achieves 2x throughput over vLLM's synchronous LLM engine in production-scale workloads by optimizing hardware utilization and providing fault …
Amazon Web Services announced container image caching for Amazon SageMaker AI inference, reducing end-to-end latency by up to 2x for generative AI models during scale-out events by eliminating contain…
JetBrains open-sourced Mellum2, a 12B Mixture-of-Experts coding model under Apache 2.0, designed for air-gapped and compliance-locked environments where external API calls are prohibited. The model us…
A developer deployed a Llama 3 inference endpoint on Oracle Cloud Infrastructure's OKE cluster using vLLM and NVIDIA A10 GPUs in about 20 minutes. The setup cost $1.52/hr on-demand or $0.46/hr preempt…
Flama 2.0 introduces first-class support for generative AI, enabling users to download, package, and serve large language models (LLMs) via a single command line. The framework allows fetching models …
Inference engineering, the craft of optimizing GPU operations during AI model inference, can cut costs by up to 80% by addressing the split between prefill and decode phases. Two teams using the same …
A developer built Trigix, an open-source (MIT), self-hostable workflow automation platform with a Rust execution engine and AI nodes that can run against local models. The platform features ~180 node …
Anthropic CEO Dario Amodei said reinforcement learning shows the same log-linear scaling as pre-training, but RL system efficiency is critical to afford enough training. Experiments on open models sho…
A new Rust-based reverse proxy called Sors reorders prompt content to maximize prefix cache hits in LLM inference engines like vLLM and SGLang, improving latency by placing static content before dynam…
Enterprises are burning millions on cloud LLM tokens due to inefficient agentic systems, prompting a shift to open-weight models on dedicated infrastructure to eliminate marginal token costs and achie…
PagedAttention, a memory optimization technique in the vLLM inference server, applies virtual memory concepts to manage the KV cache in large language models, improving throughput by reducing fragment…
Engineers achieved up to 67% cost savings and 2.7x better goodput by using Prefill-Decode disaggregation with Ray and vLLM on AMD MI325X GPUs, separating prefill and decode phases onto dedicated hardw…
Researchers introduced CacheWise, a KVCache management layer for LLM coding agents, reducing evictions by 2-2.6x and improving session completion time by up to 3.5x in vLLM, according to a June 2026 a…
NVIDIA researchers developed an agentic system for deploying machine learning models to ephemeral SageMaker endpoints, generating runtime code at deployment time from prose artifacts rather than reusa…
Phlox, an open-source self-hosted agentic web chat application, has been released on GitHub. It supports any model provider including AWS Bedrock and OpenAI-compatible endpoints, and features agentic …