An AI gateway is one control plane for every LLM request your application makes, regardless of which provider serves it. Without one, the pieces scatter: provider keys in one place, rate limits in another, caching somewhere else, and separate homes for permissions, spend reporting, and logs. That sprawl gets harder to govern with every new service that starts calling a model.
The things worth evaluating are fairly concrete — does it consolidate provider access, cap spend, deduplicate calls with caching, keep durable audit records, and let you see what production is actually doing when latency, cost, or output quality moves.
This post compares six gateways across provider coverage, governance, caching, cost visibility, audit logging, and observability. They break down into two rough camps, and the right pick depends mostly on which constraint is actually binding for you.
An AI gateway sits between your application code and your model providers. Every LLM request passes through it on the way out.
Without that layer, each service handles its own provider keys, retries, rate limits, caching, and logging. With it, services hit a single endpoint that applies one consistent set of access rules, quotas, cost accounting, and audit records across every provider behind it.
"AI gateway" and "LLM gateway" generally describe the same thing. The first term shows up more when the conversation is about governing production; the second when it's about developers routing between models. Model routing and a unified API are features of a gateway, not separate product categories — worth knowing when you're reading vendor pages.
Most options fall into one of two camps:
Infrastructure-first gateways extend an existing API management, Kubernetes, or edge platform to cover AI traffic. You inherit the deployment model, the policy engine, and the operational overhead you already have.
LLM-native gateways start from model access, prompt traffic, observability, and provider abstraction. You get AI-specific features first and general-purpose traffic management second.
Neither is better in the abstract. Pick based on whether your real constraint is infrastructure control, provider routing, or tying production traffic to some measure of quality.
Provider and model breadth. Broad coverage means one endpoint reaches every model you care about, so adopting a new one doesn't mean writing another provider-specific integration and shipping it to every service.
Rate limiting and quotas. Limits scoped per key, per team, and per model are what stop a runaway agent loop, a staging environment, or one hungry feature from quietly eating the month's budget.
Access control and governance. Role-based permissions, virtual keys, and policy enforcement in one place give engineering, security, and finance a shared handle on model usage instead of three different partial views.
Caching. Caching responses cuts both cost and latency on repeat calls. The details matter more than the checkbox: can you control expiration, scope, and per-request behavior, and can you tell whether a given response was served from cache?
Cost tracking and attribution. Spend should slice by team, feature, model, user, or environment. A single org-wide number tells you that you have a problem but not where it is.
Audit logging. Durable records make it possible to reconstruct which requests ran, who made them, what the payloads were, and how they resolved — during an incident, or during a compliance review six months later.
Actionable observability. Production logs get considerably more useful when a trace can become an evaluation case, a scorer, or a release check, rather than just a row in a dashboard.
That last criterion is the one most comparisons skip, and it's where these tools differ most. A gateway already holds the inputs, outputs, latency, token counts, and cost for every call. Whether that data can feed an evaluation workflow — or just a metrics view — determines whether production traffic helps you catch regressions before the next prompt or model change ships, or only helps you explain them afterward.
Best for: teams shipping production AI applications who want model routing joined up with tracing and evaluation in one system.
Braintrust Gateway gives you a single API for routing across providers: OpenAI, Anthropic, Google, AWS, Azure, Mistral, and others on its supported list. Setup is about as light as it gets — point an existing SDK at https://gateway.braintrust.dev
, authenticate with a Braintrust API key, and call supported models without carrying separate provider integrations through every service. Provider keys are configured at the organization or project level, and custom providers let you route to self-hosted models, fine-tuned models, and proprietary services through the same OpenAI-compatible client.
The distinguishing property is that gateway traffic lands in the observability and evaluation layer by default. Calls through the gateway are traced without extra SDK instrumentation, so inputs, outputs, latency, token usage, cost, and caching behavior are all inspectable. Those spans can then be scored, annotated with metadata or feedback, added to datasets, compared across experiments, and used in CI checks — which is a meaningfully different workflow from exporting logs and rebuilding that pipeline yourself.
The operational controls are present too. Response caching cuts repeat calls, with request headers, TTL settings, and cache-status visibility governing when a cached response gets served. Cached data is encrypted with AES-GCM under a key derived from the caller's API key, and cached results are scoped to that user by default. Response headers expose cache status, provider endpoint, error origin, request ID, and logged span ID — enough to correlate routing behavior with debugging and cost attribution.
Pros
Cons
Pricing: Free Starter plan (1 GB processed data, 10K scores, unlimited users). Pro at $249/month. Custom enterprise pricing. The gateway itself is free during beta.
Best for: enterprise teams that want a managed gateway bundled with governance, guardrails, prompt management, and compliance controls.
Portkey packages an AI Gateway alongside Observability, Guardrails, Governance, and Prompt Management. Its Universal API covers routing, fallbacks, load balancing, retries, virtual keys, logs, traces, feedback, metadata, filters, and alerts.
It fits when you want gateway controls and guardrails from one managed vendor, especially where the requirements list includes RBAC, SSO, granular budget and rate limits, private cloud deployment, VPC hosting, data export, or compliance controls. Both deterministic and LLM-based guardrails are supported, which is one of the stronger parts of the product. If you gate releases on evaluation results, you'll likely still run that workflow elsewhere.
Pros
Cons
Pricing: Free tier with 10K logged requests. Paid plans from $49/month. Custom enterprise pricing.
Best for: teams with DevOps bandwidth who want an open-source gateway they run themselves.
LiteLLM ships a Python SDK and a Proxy Server that route requests to 100+ providers behind an OpenAI-compatible interface. The open-source plan includes provider integrations, virtual keys, budgets, teams, load balancing, RPM and TPM limits, guardrails, and logging integrations — a genuinely broad feature set for something you can pip install
.
It's the right pick if you want infrastructure control and accept the operational cost of running the gateway. Logging integrations cover Braintrust, Langfuse, Arize Phoenix, LangSmith, and OpenTelemetry, so it composes reasonably well with whatever observability stack you already have. Enterprise adds cloud or self-hosted deployment, support, custom SLAs, JWT auth, SSO, and audit logs.
Pros
Cons
Pricing: Free and open source for self-hosted use. Custom enterprise pricing for hosted management and enterprise features.
Best for: enterprises already running Kong who want AI controls inside the API management stack they operate today.
Kong AI Gateway layers AI-specific capabilities onto Kong Gateway and Kong Konnect via plugins: universal LLM API routing, rate limiting, semantic caching, semantic routing, MCP and A2A traffic gateway support, RAG injection, data governance, guardrails, prompt engineering, load balancing, audit logs, LLM metrics, metering, cost controls, and secrets management.
The appeal is consolidation. If your platform team already governs APIs through Kong, AI traffic can be governed by the same infrastructure, the same policies, and the same on-call rotation. The cost is that you're adopting AI features inside a full API management platform, with the setup and pricing that implies. Structured evaluation, scorer development, and CI release gates will need a separate system.
Pros
Cons
Pricing: 30-day free trial. Plus plan billed per gateway. Custom enterprise pricing.
Best for: Kubernetes-native enterprises that need on-prem control over MCP servers, AI services, and private AI infrastructure.
SUSE AI Universal Proxy is an open-source (Apache 2.0) project for managing and proxying MCP servers. It sits inside SUSE's broader cloud-native AI direction and focuses on discovery, registration, authentication, governance, cost tracking, logging, and MCP service control across enterprise environments.
Note the scope carefully: this is more an MCP governance layer than an LLM router. It's a strong fit for teams standardizing AI services on Kubernetes where shadow AI discovery, private deployment, and centralized MCP governance are the priorities. For plain routing across hosted model APIs, the LLM-native gateways are a more direct path.
Pros
Cons
Pricing: Open source, deployed on your own Kubernetes infrastructure.
Best for: teams that want caching, rate limiting, and usage visibility running at the edge with nothing to operate.
Cloudflare AI Gateway routes AI requests through Cloudflare for providers including OpenAI, Anthropic, Google, Replicate, and Workers AI. The feature set covers analytics, logging, caching, rate limiting, request retry, model fallback, persistent logs with plan-specific storage limits, DLP scanning, guardrails, and unified billing.
The pitch is low friction: core features are free on all plans, and there's no proxy for you to run. Observability centers on request metrics, logs, usage analytics, and cost visibility, so structured evaluation and CI gating need a separate workflow. Some capabilities are in beta or depend on other Cloudflare subscriptions, which is worth checking against your plan before committing.
Pros
Cons
Pricing: Core features are free on all plans.
Infrastructure governance is the constraint → Kong or SUSE. Kong if API management is already standardized there. SUSE if you're Kubernetes-native and need private deployment, MCP governance, and shadow-AI discovery.
You want to own the proxy → LiteLLM. Broadest provider coverage of the group, and free, as long as you have the engineering capacity to run it in production.
You want governance, guardrails, and prompt management from one vendor → Portkey.
You want edge caching and usage analytics with nothing to operate → Cloudflare.
You want routing to feed evaluation and release checks → Braintrust. It's the only option here where gateway traffic becomes evaluation input without a second pipeline.
Five of these six gateways stop at telling you what happened. You get request metrics, logs, cost breakdowns, and a dashboard to query. That's genuinely useful for incident response and budget conversations, and for a lot of teams it's enough.
But the gateway is already holding the most valuable evaluation data you have: real inputs, real outputs, real latency and cost, at production scale. If that data can't move into datasets, scorers, and pre-deploy checks, you end up rebuilding a second pipeline to get it there — or, more commonly, you don't, and you find out about quality regressions from users.
Braintrust is the option in this comparison built around closing that loop, which is the main reason it leads the list rather than any individual gateway feature. Whether that matters to you depends on whether prompt and model changes are a regular part of your release cycle. If they are, routing and evaluation living in the same system removes a real class of work. If your model layer is stable and your constraint is infrastructure policy, one of the others is a better fit.
What is an AI gateway?
The operating boundary between your application code and your model providers. It earns its place when LLM calls graduate from experiments to shipped features, because that's when you need consistent handling of access, usage, cost, logging, and debugging across more than one service.
What's the difference between an AI gateway and an LLM gateway?
Practically nothing — the terms are used interchangeably. Compare capabilities, not labels: does it support your providers, enforce the controls your organization requires, and give engineers a clear view into production behavior?
What's the difference between an AI gateway and a model router?
A router picks a destination for a request. A gateway runs the whole operating layer around it: credentials, caching, quotas, logs, cost visibility, policy enforcement. Routing is one function inside a gateway.
Do I need a gateway if I only use one provider?
Often yes. One provider still benefits once multiple services, teams, environments, or features share the same account. Centralizing traffic simplifies key management, spend control, and failure inspection, and it means a future provider change doesn't mean rewriting application logic.
Can a gateway and an evaluation platform be the same tool?
Yes, and it's a reasonable consolidation when production traffic should inform release decisions. A standalone gateway tells you what happened to a request; one wired into evaluation helps you decide whether that behavior is good enough to ship.