{"slug": "tetrate-adds-ability-to-route-inference-requests-via-ai-gateway", "title": "Tetrate Adds Ability to Route Inference Requests via AI Gateway", "summary": "Tetrate added a token broker service to its AI gateway to route inference requests to the most suitable large language model based on capability and cost. The company also introduced a distributed inference control plane for usage policies that redirect requests to approved alternatives. These capabilities aim to help organizations manage AI model consumption costs and ensure governance.", "body_md": "Tetrate this week [added a token broker service to its artificial intelligence (AI) gateway](https://www.prnewswire.com/news-releases/tetrate-adds-token-brokering-capability-for-ai-code-gen-cost-management-through-agent-router-enterprise-302826151.html) to make it simpler to route inference requests to the large language model (LLM) that is best suited from a capability and cost perspective to process them.\n\nAdditionally, Tetrate has added a distributed inference control plane to its Tetrate Agent Router Enterprise gateway that enables IT teams to define specific usage policies that redirect a request to an approved alternative rather than returning an error or forcing anyone to rewrite code.\n\nDavid Wang, head of product for Tetrate, said these extensions to the company’s Agent Router Enterprise gateway enable organizations to apply rules to inference requests that could, for example, route many of them to a lower-cost, open source AI model rather than a more expensive proprietary alternative.\n\nAdditionally, organizations might decide that certain classes of inference requests uniquely require the capabilities of the latest AI models made available by OpenAI or Anthropic, he added.\n\nThe token brokering capability that Tetrate has added at no additional cost can be invoked via a command line interface (CLI), software development kit (SDK) or application programming interface (API).\n\nBased on the open source Envoy AI Gateway project that Tetrate is advancing in collaboration with Bloomberg and Nutanix, the [latest 1.0 version of that open source software has added a single OpenAI-compatible interface](https://www.prnewswire.com/news-releases/envoy-ai-gateway-reaches-v1-0--establishing-the-open-source-standard-for-enterprise-ai-traffic-302808088.html) for invoking AI models along with a native Model Context Protocol (MCP) server and support for token-aware traffic management, centralized credential management, distributed tracing capabilities based on OpenTelemetry GenAI semantic conventions and stable API.\n\nGoing forward, the project’s maintainers plan to add dollar-based controls alongside existing token-based controls, as well as deeper MCP authorization and identity management capabilities.\n\nThe Envoy AI Gateway itself is based on the open source Envoy proxy, originally developed by Lyft, that is now being advanced under the auspices of the Cloud Native Computing Foundation (CNCF).\n\nIn general, the cost of a token continues to decline, but as usage increases, organizations are now moving to control costs. Instead of trying to encourage adoption of AI, also known as tokenmaxxing, organizations are now trying to optimize consumption of AI models. The overall goal is to not reduce the number of use cases for AI, but rather maximize the return on investment, said Wang.\n\nRouting inference requests efficiently is especially critical when deploying AI agents, which in the absence of any governance policies will as part of an effort to complete a task without any regard for cost will launch queries against multiple AI models.\n\nMany organizations are also now trying to ensure that the consumption of AI models also aligns with any digital sovereignty requirements that might limit what AI models can actually be used.\n\nWhile no two organizations have the same level of AI maturity, it’s now only a matter of time before inference requests are routinely routed to the AI model that an IT team has determined is best capable of handling that request. The challenge, as always, is providing that capability in a way that is seamless to the end user that originally created the request.", "url": "https://wpnews.pro/news/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway", "canonical_source": "https://techstrong.ai/articles/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway/", "published_at": "2026-07-17 13:12:23+00:00", "updated_at": "2026-07-17 13:41:15.377562+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-products", "ai-tools", "ai-agents", "ai-policy"], "entities": ["Tetrate", "Envoy AI Gateway", "Bloomberg", "Nutanix", "OpenAI", "Anthropic", "Cloud Native Computing Foundation", "David Wang"], "alternates": {"html": "https://wpnews.pro/news/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway", "markdown": "https://wpnews.pro/news/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway.md", "text": "https://wpnews.pro/news/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway.txt", "jsonld": "https://wpnews.pro/news/tetrate-adds-ability-to-route-inference-requests-via-ai-gateway.jsonld"}}