Amazon Web Services announced Amazon SageMaker HyperPod Inference Gateway, a Kubernetes-native, GPU-aware routing system that deploys as a single EKS managed addon on existing HyperPod infrastructure …
Llm-d is a Kubernetes-native distributed inference serving stack, released open source under Apache 2.0, that adds inference-aware orchestration on top of vLLM and Kubernetes without replacing either.…
Microsoft's Azure Kubernetes Service engineering team published a reference implementation on June 29 that separates agent-request routing into three layers: semantic model selection via RouteLLM, gat…
A new open-source tool called llm-d acts as an LLM-aware load balancer for distributed inference, intelligently routing requests across vLLM instances based on KV cache locality and GPU utilization. B…