Envoy AI Gateway is now Agent Router and has joined the Agentic AI Foundation. For why the project is moving and what it means for existing users, read the announcement. This post is about what Agent Router is and what it does for the agent you are building.
| TL;DR |
|---|
| Agent Router gives agent builders the traffic handling Envoy has proven in production for a decade, without requiring Envoy proxy expertise. Your agent gets one integration point for every model and MCP tool; provider APIs, credentials, failover, tool wiring, and token limits are handled by the gateway configuration rather than your code. It starts with a single command on your laptop, and the same pattern runs for your cluster deployment. |
Powerful and usable are not the same thing, and we need both #
Envoy is a remarkably powerful traffic-handling proxy, just as Unix is a remarkably powerful operating system. However, shipping a terminal to interact with Unix did not make personal computers running a Unix-based operating system take off. Personal computers took off with a different interface, built for a different audience, and the operating system underneath was no less powerful for it. Powerful and usable are not the same thing, nor are they mutually exclusive.
Envoy is at the same point for traffic handling, and Agent Router is the configuration interface that makes it tailored for agent developers.
Agents generate traffic complex enough to require the power, and while building agents, we believe in ease of use. An agent working on a task makes a series of model calls and tool calls. One task may need a few requests, another two hundred. Responses can stream for minutes. Every provider has its own request format, its own credentials, its own failure modes. Every tool server has its own endpoint and its own idea of who is allowed to call what. We believe powerful can be usable; it just requires a good interface.
Built to solve a real, shared problem #
In 2024, Bloomberg's engineers identified their inference integration needs as a traffic problem: a unified API across providers, failover between them, token counting with rate limits, and authentication with each upstream provider. They brought that list to the Envoy community, where Tetrate, with maintainers on both Envoy Proxy and Envoy Gateway, joined them in building it. While the foundational traffic parts were already solved, provider translation and token accounting became the work the teams shipped together.
Agent Router grew out of that Bloomberg and Tetrate collaboration: one integration point for your agent, Envoy underneath handling the traffic, and the full Envoy configuration surface still there for everyone who wants it.
Start with one command; keep the agent you have #
OPENAI_API_KEY=sk-your-key aigw run # gateway on localhost:1975
export OPENAI_BASE_URL=http://localhost:1975/v1 # the only change your agent needs
Your existing client connects by changing one variable. Any client that reads OPENAI_BASE_URL works as is; MCP clients connect to the gateway's MCP endpoint, and aigw run accepts the same mcpServers file that Claude Desktop, Cursor, and VS Code already use.
Your local Agent Router configuration follows the same pattern as your production configuration. What you write here is what you deploy on a dedicated gateway or a Kubernetes cluster later; only the packaging changes. See the standalone guide and configuration resources.
5 things your agents shouldn't have to focus on #
Every agent codebase answers the same questions somewhere: which provider API to speak to, where the keys live, what happens when a call fails, which tools are wired in, and how much is being spent. Sometimes the answer is your code, sometimes a dependency, sometimes a gap. Agent Router moves all five into gateway configuration.
The differences between inference provider APIs. Instead, your agent sees one OpenAI-compatible endpoint. Agent Router translates each request into the upstream's format, so changing the model is changing the model name. Model name guide.
Inference provider credentials. The only thing your agent authenticates to is the gateway. Provider keys and short-lived cloud credentials stay with Agent Router, which authenticates upstream on the agent's behalf. Upstream authentication.
Failover strategies. Your agent sends one request and gets one answer. Behind that request, Agent Router retries a failed call at the next provider, in that provider's format, with that provider's credentials. Fallback guide.
MCP server multiplexing. What your agent gets is one tool catalog, sized for it. Agent Router builds that catalog from selected tools across many MCP servers, filtered by identity at discovery and again at invocation. A support agent gets a small catalog, which keeps context small and avoids wrong-tool calls. MCP guide.
Usage controls beyond request counts. Your agent never counts a token. Agent Router does, and enforces limits on input, output, or total tokens per team, application, or model. Usage-based rate limiting.
The powerful half, already built #
Long-lived streams, uneven demand, and upstream failures are not new traffic problems. Envoy has handled them at scale for a decade, and agent traffic inherits that handling unchanged.
The failure modes are already handled. Timeouts, retries, health checks, outlier detection, and circuit breakers exist in Envoy because years of high-volume production deployments needed them. A stalled provider is detected and routed around. A burst of concurrent tasks hits a connection cap instead of taking down a backend.
The security record is public. Envoy is a CNCF-graduated project with a written threat model, a security release process, a declared security posture for every extension, and a standing bug bounty through Google's Patch Reward Program.
You can see what your agent actually did. Every model and tool call passes through the gateway, so each one is logged and traced with no instrumentation in your agent: which model answered, how many tokens, time to first token, where a fallback kicked in, which tool was called. When one request in two hundred goes wrong, the trace shows you which. Signals follow the OpenTelemetry GenAI conventions and land in the tools you already run. Observability.
When you need something the gateway does not do #
At some point you will need something the gateway does not do: a header rewrite for one tenant, a call to an internal approval service, a step that removes sensitive data before the prompt reaches the provider, a custom check on a tool call's arguments.
Agent Router routes are Envoy routes, so Envoy's extension frameworks are available to you as they are. Through Envoy Gateway, which Agent Router builds on, you can attach Lua for small changes, dynamic modules in Rust or Go, or an external processor when the logic belongs in a service you already run. Client authentication uses the same mechanism, whether you validate a JWT, run an OIDC flow, or call your own authorization service. Every one of these extension points has been running in production proxies for years; the same mechanism that handles the traffic runs your custom step, so you are not adding a second system to maintain. Start with the Envoy Gateway extensibility guides or Built on Envoy.
Where the project is headed #
Agent Router's new chapter within AAIF is bringing the power of Envoy to the agentic stack. Unix did not become less powerful when it reached people who never open a shell, and Envoy does not become less powerful because it handles agent traffic; what changes is the configuration interface. Connecting an existing agent, setting a traffic policy, and understanding a configuration error should require less knowledge of proxy internals than it does today. Being inside AAIF puts that work next to the people building the agentic stack and the protocols agents speak, within a community dedicated to enabling agent builders.
Agent builders bring something back to Envoy, too. Every agent on Agent Router is a new traffic pattern and a new failure case, and when one of those needs something the data plane cannot yet provide, the Agent Router and Envoy communities work together on the enhancements in Envoy. The proxy keeps improving for the traffic that agents actually send.
Join the community, help shape what is next #
Do not hesitate to show up with a problem, it is a valuable contribution. Often the most useful contributions start at a real problem: an agent setup that is hard to connect, a rule that is hard to express, or a failure you can reproduce.
Community meetings are held weekly on Mondays and are open to everyone; larger changes go through public proposals, while smaller fixes and enhancements go straight to issues and PRs.
Agent Router exists because of the Envoy and Envoy Gateway communities and the Gateway API and Inference Extension projects it builds on. Thank you.
Links:
Authors