A year ago, most routing logic was a few lines of if model == "gpt-4": ... elif model == "claude-sonnet": ...
shoved inside a orchestration framework like LangChain or custom middleware. It handled fallback and maybe a simple cost cap. Today, dedicated services handle model selection, latency-aware dispatch, budget-constrained routing, semantic router hints, and observability you can alert on. That shift from "utility" to "infrastructure-as-a-service" defines a new category.
What makes the category real? Stand-alone products. Portkey, Helicone, AI Gateway (open-source), and newer entrants treat routing as a separate layer. They expose a unified API that proxies to any provider, and they add what a simple if-else
never could: automatic retry with backoff, cost tracking per request, request-level fallback chains, and caching across providers. One of the projects I maintain replaced a fifty-line custom router with a five-line configuration block, and the observability alone saved me two incidents last month.
The technical architecture is also clean. Most follow an ingress-proxy-evaluate-egress pattern. The ingress accepts a single endpoint, the proxy inspects the request (model hint, latency budget, user, etc.), an evaluation engine picks the provider and model, and the egress fires the request and returns the response. Some add a feedback loop so the router learns from latency or accuracy signals. That pattern is now mature enough that you can deploy it as a sidecar, a gateway, or even a serverless function.
I see strong parallels with the API gateway movement in microservices — the same "glue" layer that started as one-offs and coalesced into Kong, Tyk, or AWS API Gateway. LLM routers are the gateway for model calls. And just like API gateways, they'll need to handle auth, rate limiting, IP allowlists, and audit logs for compliance. Some already do.
The key takeaway: if you're still hard-coding model endpoints in your app code or treating provider selection as an afterthought in your agent chain, you're missing a whole class of production hardening. The router-as-service gives you observability, redundancy, and cost control in a single configurable proxy. That's no longer a hack — it's a category. Pick a managed router or deploy the open-source one, but don't skip the layer.
Next GitHub's menu icon turned into a pancake emoji – anyone else? →