Llm-d is a Kubernetes-native distributed inference serving stack, released open source under Apache 2.0, that adds inference-aware orchestration on top of vLLM and Kubernetes without replacing either.…
A Kubernetes Dynamic Resource Allocation (DRA) driver makes composable CXL memory a schedulable cluster resource for cross-node KV-cache reuse in LLM serving, according to an arXiv paper (2609.10790v1…
A former B2B SaaS web developer who switched to inference engineering about a month ago published a skills map arguing the field requires only three service categories: routing and scheduling via Nvid…
IBM Research and Red Hat used the open-source llm-d framework to deploy GLM-5.2, an approximately 753-billion-parameter mixture-of-experts model, on 544 NVIDIA H100 GPUs, serving up to 3,000 concurren…
Shaide, a Kubernetes-native AI platform for distributed multi-model inference, has been released under the Apache 2.0 license. It uses vLLM for inference, llm-d for orchestration, KV-cache-aware sched…
Axem co-founders open sourced Shaide, a Kubernetes-native AI platform for distributed multi-model inference, under the Apache 2.0 license. The platform uses vLLM for inference and llm-d for orchestrat…
A vLLM RFC from the llm-d team warns that disaggregated inference deployments, where prefill and decode run on separate schedulers, can trigger recomputation-based preemption inside the decode instanc…
Red Hat's head of product for AI platforms, Tushar Katarki, says open source models and platforms are improving faster than enterprises planned, giving organizations more control over cost, data, and …
The Cloud Native Computing Foundation (CNCF) accepted llm-d, an open-source framework for distributed LLM inference on Kubernetes, as a sandbox project in March 2026, with contributions from IBM Resea…
Google Cloud announced the general availability of Managed Lustre, a high-performance storage solution powered by DDN's EXAScaler, offering throughput from 125 MB/s to 1000 MB/s per TiB and scaling up…
The llm-d project introduces co-operative time-slicing to interleave independent reinforcement learning jobs on shared hardware, increasing aggregate accelerator duty cycles from ~40% to 70% without i…
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…
NVIDIA open-sourced Grove, a Kubernetes API for managing multi-component AI inference stacks, at KubeCon Europe 2026. Grove introduces custom resources for gang scheduling, topology-aware placement, a…
IBM Research, Red Hat, and NxtGen Cloud Technologies demonstrated that using llm-d to serve AI models on mixed GPU hardware can boost inference speeds by 3 to 5 times and double throughput, enabling e…