Key Takeaways
- AI capabilities change faster than enterprise systems can safely absorb, and the gap is permanent enough to design for rather than wait out.
- Traditional API gateways assume deterministic services, schema-level failures, and client-driven intent. Agentic AI systems violate all three, which is why a new architectural layer becomes necessary.
- The gateway concentrates the fastest-moving parts (guardrails, model routing, agent identity, action policy, audit) in one place, leaving the rest of the architecture stable.
- The pattern introduces real costs in latency, centralization, and operational overhead, and is not always right; single-team or single-LLM use cases may find in-application guardrails sufficient.
- Organizations with mature platform engineering can adopt proactively; others discover the need under incident pressure and pay a higher cost to build it then.
This article was written by participants of the
online InfoQ Certified Architect Program. It represents the capstone of their work, reflecting the cohort's collective learnings on the intersection of AI and modern software architecture.## Introduction
The integration layer for enterprise AI is now the fastest-evolving part of most systems, and no one owns it as a coherent layer. Teams shipping under pressure pick their own models, tools, and guardrails.
This is an evolutionary architecture problem before it is an AI problem. The pace of change in AI capabilities is unlikely to slow. New models, new protocols, and new failure modes will continue arriving on a cadence that enterprise systems were never designed to absorb. The useful question is not how to slow the change, but where in the architecture to put it, so that the rest of the system can stay where it is.
The AI gateway is one answer: an architectural seam where the fastest-moving parts of an AI system (guardrails, model routing, agent identity, action policy, and audit) can evolve at their own pace while the systems behind them stay stable. The pattern matters most where AI systems are acting autonomously rather than simply generating text. Services that consume LLMs face a bounded set of concerns; agentic systems, which perceive, decide, and act, introduce a substantially broader surface area. This article examines the pattern, its trade-offs, and the conditions under which it is worth the cost.
Figure 1: The AI gateway as the seam between stable enterprise systems and a fast-moving AI ecosystem, carrying four traffic types through one control plane: sync service to LLM, agent to LLM, agent to MCP, and agent to agent. Source: Original work by the authors.
The evolution mismatch #
One of the biggest challenges in enterprise AI today is not model performance, but the rate at which the ecosystem changes. Enterprise systems are usually built around stability. Core business platforms, integration layers, governance processes, and security controls often evolve over years, not months. The AI ecosystem operates very differently. Major model providers now release new models, updates, and capabilities several times a year. This is not a short-term challenge that will disappear as the technology matures. It comes from the fact that AI and enterprise systems evolve at fundamentally different speeds.
Figure 2: The rapid pace of change in AI, 2024–2026. Source: Original work by the authors.
The problem goes beyond the models themselves. New protocols and standards are also evolving at a rapid pace. The Model Context Protocol (MCP) has quickly gained traction as a way for models to connect with tools and external data sources, but the specification is still maturing. Features such as authentication, transport mechanisms, and tool discovery have changed as adoption has grown. At the same time, Agent-to-Agent (A2A) protocols are emerging to support communication between autonomous agents. While promising, there is still no clear industry standard, and different vendors are taking slightly different approaches. For architects and engineers, parts of the stack they integrate today may look different 12 months from now.
Security adds another layer of complexity. Every new model integration, tool connection, or agent workflow introduces new attack surfaces. Prompt injection attacks, data leakage, excessive tool permissions, and trust issues between agents are all concerns that architecture teams now have to consider. For example, a customer support agent connected to internal knowledge bases and ticketing systems could be manipulated through a malicious prompt hidden in a customer message, causing it to expose sensitive information or perform actions beyond its intended scope. Unlike traditional enterprise integrations, where interfaces may remain stable for years, AI integrations often change alongside the underlying models and protocols.
This is why the challenge should be viewed as an architectural problem rather than an AI problem, since AI is just another tool that must integrate with existing systems. Using concepts from evolutionary architecture such as fitness functions, architectural quanta, and evolutionary pressure, the question becomes clear: where do the fastest-changing components belong? If that boundary is designed poorly, change spreads across the entire system. If it is designed well, most of the enterprise platform can remain stable while the AI layer continues to evolve.
Why this is not just an API gateway problem #
When organizations begin integrating AI into their systems, the first instinct is often to treat models and agents like any other API consumer. The logic is understandable: enterprises already have API gateways that handle authentication, rate limiting, routing, observability, and policy enforcement. Why introduce another architectural component when a gateway already sits at the edge of the platform? The challenge is that agentic systems violate assumptions that traditional API gateways were designed around. This does not make existing gateways obsolete, but it does mean they are no longer sufficient on their own.
Figure 3: Traditional API gateway vs AI gateway. Source: Original work by the authors.
The first assumption is determinism. Traditional gateways sit in front of services whose behavior is generally predictable. Given the same input, an API is expected to produce the same output or execute the same operation. Agentic systems do not work that way. The same prompt can result in different responses, different reasoning paths, or even different tool selections. This makes testing, auditing, and incident investigation significantly harder. A gateway can tell you that a request was made, but it cannot explain why an agent invoked a tool or chose one action over another.
The second assumption is that failures occur at the schema level. API gateways are very good at detecting malformed requests, invalid tokens, missing parameters, or traffic that exceeds policy limits. Agent failures are often semantic rather than technical. A request can be perfectly valid from a protocol perspective and still represent a bad decision. Consider an agent that issues a refund to the wrong customer, retrieves sensitive data that was not required for the task, or executes a destructive operation based on manipulated context. The request itself may be correctly formatted and fully authenticated. The problem is that it should never have been made in the first place. This introduces entirely new attack surfaces, including prompt injection, context poisoning, and tool misuse.
The final assumption is that actions are fully specified by the client before reaching the system. Traditional gateways assume that the client has already determined the action to be performed. The gateway validates the request but does not assess whether the action aligns with the user's objective. Agentic systems introduce an additional layer of decision-making. Agents interpret goals, reason about possible actions, and decide which tools to invoke. As a result, the relationship between user intent and system behavior becomes indirect. Understanding why an action was taken, whether it was justified, and whether it complies with policy becomes an architectural concern.
The pattern: guardrails as the evolutionary seam #
Concerns like these need a place to live. That layer is the AI gateway: a single control plane for model routing, identity, action policy, content guarding, and audit. Each changes faster than the services behind it, so each needs one enforcement seam, not many.
Figure 4: AI gateway control planes. Each control gets revised less often than what it governs, so applications behind the gateway stay stable. References: CNCF, NIST SP 800-207, CSA ATF, OWASP LLM01/02, OTel GenAI, EU AI Act. Source: Original work by the authors.
Model routing, provider abstractions #
Models are the fastest-moving piece of the stack. Capability, pricing, and rankings shift continuously. Most systems need a mix of top-level, cheap, fine-tuned, or self-hosted models. A gateway isolates that churn, enabling model swaps, cost optimization, and provider failover without changing the application.
Security #
Identity and authorization are mature for humans and known services, but not for agents acting on one another's behalf. Delegated authority is the unsettled piece, and protocols revise their standards fast (Figure 2). A gateway carrying the delegation chain rather than breaking it makes the fast-moving standard cheap to follow.
Action policy applies zero trust to agent actions: every action is checked against policy every time, enforcing least privilege per operation. A token issued before a prompt-injection hijack still validates, so per-action checks can't stop the hijack, but they can stop the hijacked action from being authorized, with the gateway as a policy enforcement point, per NIST SP 800-207.
Segmentation governs not who an agent is or what it may do but what it may reach, which agents, tools, and whether the internet at all, so a compromised agent can't pivot to other systems.
Content guarding contains rather than eliminates injection and disclosure. It runs both ways: inspecting input for injection and output for leakage or offensive content. Evasion techniques evolve constantly, so dispersing checks across applications decreases reaction time and increases the operational complexity of updating them. Centralizing at the gateway isn't just efficient but security-critical: one place to update, as fast as attacks evolve.
Observability and audit #
Monitoring and keeping the record isn't a nice-to-have; for autonomous, non-deterministic agents, it's foundational. The two jobs differ: observability is operational, catching drift and cost spikes as the system runs; audit is evidentiary, answering to regulators later and demanding retention, and the EU AI Act will raise the bar. Plain logs record that a call happened; they can't explain why an agent acted. What's needed is semantic logging: Request → Decision → Action.
Figure 5: Illustrative example of semantic logging. Source: Original work by the authors.
Centralizing this record at the gateway is the evolutionary-architecture move: every call already passes through it, so when requirements change, a new article, a new provider, a longer retention period, you change one plane instead of N applications. It's also the natural single point to redact or hash PII before it's stored, storing an input hash rather than raw prompt text, since retaining raw prompts is itself AI Act compliance exposure.
Enforcement in practice #
This isn't hypothetical: kgateway's Agent Gateway, Portkey, and LiteLLM already implement subsets of these responsibilities. The emerging pattern is policy-as-config: guardrail, segmentation, and routing rules declared as version-controlled configuration, reviewed in a PR, deployed independently of the applications behind the gateway (the declarative model OPA uses). It splits ownership cleanly: security owns the policies, a platform team owns the gateway. The CSA ATF recommends exactly this move to policy-as-code.
agent: code-review-agent # identity
authorization:
allow:
- repo.read
- pr.comment
- static_scan.run # tool action shown in trace
deny:
- repo.push
- pr.merge
- secrets.read
tools: # external calls the agent may make
- github-mcp # MCP server: get_pr_diff
- static_scan
guardrails: # prompt & content guarding
input:
- prompt_injection_filter
output:
- secret_redaction
routing: # model routing / provider
default: gpt-4o
fallback: self-hosted-slm
observability: # the record → Request / Decision / Action
logging: semantic
Figure 6: Simplified gateway policy for an agent, illustrating the policy-as-config pattern. Source: Original work by the authors.
The gateway gives all of them a single place to change. What that concentration costs is the next question.
Trade-offs and when the pattern does not fit #
When adopting any solution, there should be consideration of what possible implications and trade-offs could be.
Figure 7: When an AI gateway may not fit. Source: Original work by the authors.
Latency and throughput #
Each AI gateway feature added could impact latency and throughput. Some LLM functionality may be offered via translation to provide an agnostic API. Connections from an agent to data sources, tools, or other agents will go through the AI gateway. Vendors may set throughput limits according to the plans they offer.
Perform benchmarking to determine the added latency and throughput limitations. Latency-sensitive workloads may not be suitable or may require a self-hosted, dedicated, or specialized AI gateway. If latency and throughput requirements can’t be met within the available budget, an AI gateway may not be an option.
Centralization #
Each domain may have its own API gateway. However, a centrally managed AI gateway may simplify governance, especially as the landscape changes. A team will need to take ownership of AI gateways, making them responsible for the operational and maintenance concerns. Depending on how teams are organized, there may be other teams involved in managing cost controls, guardrails, and what is provided, like data sources, models, and tools. Teams may manage these for certain workloads. Making it clear who owns each of these helps as changes are applied.
There may be organizational obstacles that need to be identified and overcome before an AI gateway could be considered.
Probabilistic protection #
Guardrails mitigate certain risks, but don’t eliminate them. False positives could reduce the effectiveness of the application or create frustration, like misidentifying a person’s name as profanity (the Scunthorpe problem). False negatives could result in incidents or legal issues, like a machine-learning-based guardrail failing to prevent a malicious user from jailbreaking a chatbot. The benefits of guardrails would need to be considered in balance with the potential impact on the workload and customer satisfaction.
Operational cost #
The improved observability will have a financial cost. Maintenance will be required by the team that manages the AI gateway and those connecting through it, as APIs change and features are added. When self-hosting, there will be resource costs. These costs may affect adoption, but should be considered against the impact of incidents and over-budgeted usage that could result.
Provider native features #
AI gateways can provide a standardized API for accessing LLM functionality. This makes it easier to switch between providers or enable different providers to be conditionally used through the same endpoint. However, when providers expose new functionality, there may be a delay between when this functionality is available, if it will become available. If the AI gateway enables custom functionality to be added, like LLMs or guardrail solutions, that reduces the potential impact.
What if it's not the right solution? #
Any functionality provided by an AI gateway could be implemented in the application. This reduces latency, but requires design to not tightly couple the implementation and reduce observability. It may also provide a path towards future adoption by making use of observability and cost controls to make room in the budget.
The benefits will need to be considered against the costs. If there are only a few teams, LLMs or guardrails in use, it may not be worth it for production workloads. However, it may have value for development, where overuse could be expensive.
An evolution story #
The trajectory toward consolidation is visible across the industry, even if individual organizations are at different points along it. The team behind this article spans Stages 1 and 2, with practitioner experience of mature gateway patterns in adjacent domains. Drawing on documented incidents and industry-wide patterns, here are the four stages most enterprises pass through. Each stage has a different shape depending on whether the AI systems involved are LLM-consuming services or autonomous agents, with the latter introducing a step-change in operational risk.
Stage 1: Single team, single provider #
Most enterprises that have started integrating AI are here or close to it. A single team adopts a single model provider, often for internal use. Code assistants and PR review tooling are common starting points, alongside narrow customer-facing pilots. The architecture is simple, the integration is direct, and the operational risk is limited. Some of our organizations are at or near this stage, with AI deployed for development tooling and tightly scoped use cases.
Stage 2: Adoption spreads #
Once the value of the first capability is clear, demand for more grows quickly. Teams adopt their own preferred models, often picking different providers. Each makes its own choices about authentication, prompt handling, and guardrails. EY’s AI platform publicly reports operating over 50,000 AI agents. Zapier reports more than 800 agents deployed company-wide across its 360-person workforce, meaning the company runs more AI agents than employees. Some organizations in our group operate at this stage today, with LLM-consuming services running at production scale across multiple product lines through a mix of integration paths, and with mature gateway practice in the standard API space alongside a recognized gap in AI-specific controls (PII handling, prompt injection detection). The deliberate decision to put those foundations in place before deploying autonomous agents is itself part of the architectural response. The AI Incident Database logged 362 incidents in 2025, up from 233 the year before, many traceable to missing controls rather than model failure. Nothing is on fire, but no one has a consolidated view of the surface area either.
Stage 3: A forcing function arrives #
Stage 3 is the moment when fragmentation stops being tolerable. An incident, a regulatory question, or a cost surprise makes the current state untenable, and the organization is forced to consolidate under pressure rather than by design. Stage 3 looks different in each organization, though the lesson is consistent. The most consequential incidents published in the past 18 months involve autonomous agents. In July 2025, a Replit AI agent wiped a production database during an explicit code-and-action freeze, affecting records for 1,196 companies and 1,206 executives, and fabricating thousands of fake users in the process. Separately, a report described an organization that accidentally spent $500 million on Claude in a single month after failing to set usage limits on employee licenses. Both incidents are the kind of failure mode a gateway is designed to contain, and both illustrate consequences that extend beyond the technical event. Other incidents in the same window, including EchoLeak (the first weaponized prompt-injection CVE) and the Vercel breach involving OAuth permissions to a compromised AI vendor, show the same pattern in different configurations.
The cost of Stage 3 is rarely just the incident itself. A serious incident typically produces some combination of customer-data exposure, regulatory escalation, brand damage, and a quarter or more of remediation work that displaces feature development. Smaller organizations may not have the runway to absorb both the immediate consequences and the consolidation cost that follows. Regulated industries can face license implications. Customer-facing brands can lose trust that does not return. Once consolidation becomes mandatory rather than optional, the cost of building the gateway under pressure is substantially higher than the cost of building it ahead of need.
Stage 4: Consolidation onto a shared layer #
Organizations at Stage 4 describe the gateway in similar terms: a control layer where security, cost, and routing decisions live in one place. This is usually retrofit, not greenfield. Decisions surface about what gets centralized, what stays in the application, and where the team disagrees. Once the gateway exists, the compliance picture sharpens, and so does the cost picture. The architectural pressure that drove consolidation becomes the criterion for everything else the gateway is asked to do.
Stages are a model, not a sequence #
Some organizations move directly from Stage 1 to Stage 4 because their existing platform practice gives them the foundation to do so. Others stall at Stage 2 indefinitely, accepting the risk because the cost of consolidation feels higher than the cost of fragmentation, until an incident makes the calculation for them. The choice between proactive and reactive adoption is partly a choice about how the organization's architectural maturity in adjacent domains transfers to the AI surface. Organizations with mature API governance, observability, and platform engineering have the option to build foundations ahead of need. Organizations without that maturity often discover the pattern under pressure and pay the difference.
Figure 8: A view of the four positions organizations occupy on the AI gateway trajectory. The proactive path (green dashed arrow) and the Stage 3 forcing-function path (red dashed arrow) show the two routes to consolidated maturity. Source: Original work by the authors.
Open questions, and what comes next #
The pattern this article describes is still being worked out across the organizations represented here, and two questions in particular remain unresolved. The first is where the line falls between policy enforced at the gateway and policy enforced inside the application. The gateway absorbs change well when the policy in question changes faster than the application logic, but applications often have context the gateway does not, and the line moves as both sides mature. The second is operational: running guardrails involves tuning, false positive triage, and on-call burden. The team that owns the gateway becomes the team that arbitrates safety, which is useful when the arbitration is principled and corrosive when it becomes the team that says no.
The deeper point sits outside AI. Wherever a subsystem evolves faster than the systems it touches, an explicit architectural seam becomes the design. AI agents are the current instance of this problem. The next instance is already being built somewhere.