Microsoft Outlines a Three-Layer Agent Routing Stack for AKS

Microsoft's Azure Kubernetes Service engineering team published a reference implementation on June 29 that separates agent-request routing into three layers: semantic model selection via RouteLLM, gateway policy via agentgateway, and inference-aware GPU placement via the Gateway API Inference Extension and KAITO, all behind one OpenAI-compatible endpoint. The design aims to control costs and policy for AI agent workloads by making each layer independently replaceable and observable.

Microsoft Outlines a Three-Layer Agent Routing Stack for AKS Microsoft's AKS engineering team published a reference implementation on June 29 that separates agent-request routing into three decisions: which model answers, how traffic is governed, and which GPU replica serves the request. The design combines RouteLLM, agentgateway, the Gateway API Inference Extension, KAITO and Azure OpenAI behind one OpenAI-compatible endpoint. Microsoft's Azure Kubernetes Service engineering team published a reference implementation on June 29 for routing the repeated model calls generated by AI agents. Instead of treating routing as one choice, the design separates semantic model selection, gateway policy and inference-aware GPU placement. Three decisions, three owners At the first layer, RouteLLM evaluates each prompt and chooses between a lower-cost self-hosted model and a stronger Azure OpenAI deployment. The example sends simple steps to Phi-4 Mini running through vLLM while escalating harder work to a frontier model. Microsoft presents that split as a way to keep routine calls from multiplying frontier-model costs inside long agent loops. The second layer is agentgateway, an open-source, OpenAI-compatible gateway running in the cluster. It handles provider abstraction, authentication, rate limits, budgets, guardrails and OpenTelemetry without deciding which prompt is difficult. Both the strong and weak routes pass through this policy layer. The third layer handles placement only after a model has been selected. For the self-hosted path, the Gateway API Inference Extension's Endpoint Picker uses live signals such as queue depth and KV-cache occupancy to choose among vLLM replicas provisioned by KAITO. This avoids sending a small request to a busy pod simply because it is next in a round-robin sequence. What teams should validate The architecture is a worked implementation, not a new managed AKS routing product. Microsoft says the manifests were validated in mid-2026 against Gateway API Inference Extension v1.0.0 and agentgateway v1.3.1, while warning that young APIs and configuration fields can change between releases. The reference also leaves important operational work with the adopter. The semantic threshold must be calibrated against the team's own prompt mix and strong/weak model pair. The gateway needs production-grade identity and budget policies. The Endpoint Picker's example plaintext gRPC connection must remain cluster-internal and access-restricted, or be re-encrypted with a service mesh. For platform teams, the useful idea is the separation of responsibilities: prompt content chooses the model, policy governs the call, and GPU state chooses the replica. That makes each layer independently replaceable and observable while presenting one endpoint to agent frameworks. Key Points - 1The June 29 AKS reference separates semantic model choice, gateway governance and GPU-replica placement. - 2RouteLLM selects the strong or weak path, agentgateway applies policy, and the Endpoint Picker uses live vLLM state for placement. - 3Microsoft describes a worked open-source architecture whose versions, routing threshold, identity controls and internal networking must be validated by adopters. Scoring Rationale The reference gives platform teams a concrete, end-to-end pattern for controlling cost, policy and GPU placement across agent workloads on AKS. Its practical value is high, but it composes existing open-source and managed components rather than announcing a new managed routing service. Sources Primary source and supporting public references used for this report. Practice interview problems based on real data 1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with. Try 250 free problems /problems