{"slug": "transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic", "title": "Transforming Kong into an AI Gateway on GCP: Managing LLM Tokens, MCP, and Agentic Traffic", "summary": "Kong's AI Gateway, running on GCP, extends traditional API gateways with token-aware traffic management, Model Context Protocol (MCP) support, and governance for agentic traffic. The solution addresses cost unpredictability, non-interchangeable backends, and machine-driven client behavior by using plugins like ai-proxy, ai-rate-limiting-advanced, and ai-mcp-proxy on GKE with Vertex AI, Cloud Load Balancing, and Memorystore.", "body_md": "Most platform teams didn't set out to build \"AI infrastructure.\" They already had Kong running in front of hundreds of REST services on GKE, handling auth, rate limiting, and observability the way an API gateway is supposed to. Then product teams started shipping LLM-backed features, then RAG pipelines, then autonomous agents that call tools on their own and suddenly the gateway layer that worked fine for CRUD traffic started showing cracks. A single chat request can cost 50x another one depending on the model and prompt length. A single \"user request\" might now fan out into a dozen tool calls made by an agent with no human in the loop. None of that maps cleanly onto request-count rate limits or static routing rules\n\nThis is the problem Kong's AI Gateway capabilities are built to solve, and GCP is a natural place to run it: GKE for the gateway itself, Cloud Load Balancing at the edge, Vertex AI and Model Garden as first-class model backends alongside OpenAI, Anthropic, and others, and Memorystore for Redis backing distributed rate-limit state. Below is a practical walkthrough of turning an existing Kong deployment into a real AI gateway on GCP covering token-aware traffic management, Model Context Protocol (MCP) support, and the harder problem of governing agentic traffic that doesn't behave like normal API calls.\n\n**Why a Regular API Gateway Isn't Enough for LLM Traffic**\n\nTraditional gateways are built around a few assumptions that break down with generative AI workloads:\n\nCost is not proportional to request count. A request to a small model with a short prompt might cost fractions of a cent; a long-context request to a frontier model can cost dollars. Rate limiting on requests-per-minute doesn't protect your budget.\n\nBackends aren't interchangeable. Routing a request to GPT versus Gemini versus Claude isn't just a load-balancing decision ,it changes cost, latency, and even output quality.\n\nThe \"client\" is increasingly a machine, not a human. Agents chain multiple calls, retry autonomously, and can enter loops that a human would never trigger by clicking a UI.\n\nThe protocol itself is evolving. MCP now standardizes how models discover and call tools, and gateways need to speak it natively rather than treating it as an opaque HTTP payload.\n\nKong's AI Gateway is essentially the same Kong Gateway core, extended with a family of AI-specific plugins — ai-proxy, ai-proxy-advanced, ai-rate-limiting-advanced, ai-semantic-cache, ai-prompt-compressor, ai-mcp-proxy, and others — that make the gateway fluent in tokens, prompts, and tool calls instead of just headers and paths.\n\n**Step 1: Get Kong AI Gateway Running on GKE**\n\nIf you're already running Kong Gateway (OSS or Enterprise) on GKE, you don't need a new platform — you need the AI Gateway plugin set, which ships as part of Kong Gateway 3.6+ and is more fully featured from 3.12 onward (which added first-class MCP support).\n\nA typical GCP-native footprint looks like this:\n\nGKE hosts the Kong data plane nodes (and control plane, if self-managed rather than using Konnect).\n\nCloud Load Balancing (external HTTPS LB or Gateway API) terminates TLS and fronts the Kong proxy service.\n\nMemorystore for Redis backs distributed counters for the AI rate-limiting plugins, so token budgets are enforced consistently across replicas instead of per-pod.\n\nSecret Manager holds provider API keys (OpenAI, Anthropic, Vertex AI service account credentials) and is mounted into Kong via the CSI driver rather than baked into kong.yaml.\n\nCloud Monitoring / Managed Prometheus scrapes the metrics Kong's AI plugins emit — including the token-usage and MCP-specific metrics added via the Prometheus plugin extensions.\n\nDeployment itself is standard Kong-on-Kubernetes: the Kong Ingress Controller or KongClusterPlugin/KongPlugin CRDs manage plugin configuration declaratively, which plays well with GitOps if you're already running Config Sync or Argo CD on GKE.\n\n**Step 2: Route to Vertex AI and Other Providers with ai-proxy**\n\nThe ai-proxy plugin turns a Kong Route into an LLM endpoint. You define the provider, the model, and any transformation rules, and Kong normalizes the request/response shape so your applications can call a consistent internal API regardless of which upstream model actually serves it:\n\n```\nplugins:\n  - name: ai-proxy\n    config:\n      route_type: llm/v1/chat\n      auth:\n        header_name: Authorization\n        header_value: Bearer ${{ env \"VERTEX_AI_TOKEN\" }}\n      model:\n        provider: gemini\n        name: gemini-3.5-flash\n        options:\n          max_tokens: 1024\n          temperature: 0.7\n```\n\nFor multi-provider setups,routing cheap, latency-insensitive traffic to a Vertex AI model while sending premium requests to Claude or GPT ai-proxy-advanced adds weighted load balancing and semantic routing across multiple LLM targets behind a single Kong Route. This is also where cost-optimization patterns live: pairing ai-proxy-advanced with ai-semantic-cache (to skip redundant LLM calls entirely for near-duplicate prompts) and ai-prompt-compressor (which uses LLMLingua-style compression to shrink prompts before they hit the paid API) can meaningfully cut spend without touching application code.\n\n**Step 3: Make Token Usage a First-Class Rate-Limiting Dimension**\n\nThis is the core shift from \"API gateway\" to \"AI gateway\": rate limiting by request count is close to meaningless for LLM traffic. The ai-rate-limiting-advanced plugin reads the actual token usage returned by the LLM provider in each response and enforces budgets on that basis — prompt tokens, completion tokens, or total tokens, per consumer or consumer group, over a sliding window:\n\n```\nplugins:\n  - name: ai-rate-limiting-advanced\n    consumer_group: standard-tier\n    config:\n      policies:\n        - limits:\n            - limit: 20000\n              window_size: 60\n              window_type: sliding\n          identifier: consumer-group\n          tokens_count_strategy: total_tokens\n          strategy: redis\n          redis:\n            cluster_nodes:\n              - ip: ${MEMORYSTORE_HOST}\n                port: 6379\n      llm_format: openai\n```\n\nBacking this with Memorystore for Redis (rather than local in-memory counters) is what makes the limit hold across every GKE pod in the deployment, not just per-replica. Kong also supports cost-based rather than token-based budgets — assigning a dollar cost per input/output token for each model and capping consumer spend per hour, which is the more direct lever for finance and platform teams who think in terms of a budget line rather than a token count.\n\nA pattern worth calling out: tiering by consumer group lets you reject an unauthorized model choice — a standard-tier user requesting a premium model — at the gateway with a clean 400 before any upstream provider call is made, rather than silently downgrading the request or letting it through and eating the cost.\n\n**Step 4: Bring MCP Traffic Under Gateway Control**\n\nModel Context Protocol has quickly become the default way agents discover and invoke tools, but it introduces its own operational surface: persistent sessions, JSON-RPC method calls instead of simple REST verbs, and a mix of MCP clients, hosts, and servers that don't map onto a traditional request/response gateway model. Kong's ai-mcp-proxy plugin (introduced with Kong Gateway 3.12) is built specifically to sit in that path.\n\nThe plugin supports a few distinct modes:\n\nPassthrough, where Kong fronts an existing MCP server as-is, applying auth, rate limiting, and logging without altering the protocol.\n\nConversion, where Kong turns an existing REST API into an MCP-compatible tool by defining a tool schema in configuration — no code changes to the backend service required:\n\n```\nplugins:\n  - name: ai-mcp-proxy\n    config:\n      mode: conversion-listener\n      tools:\n        - description: Look up order status by ID\n          method: GET\n          path: /api/orders/{order_id}/status\n```\n\nOn top of the proxy itself, Kong adds the pieces enterprise MCP deployments actually need in production: an ai-mcp-oauth2 plugin implementing OAuth 2.1 so Kong acts as the OAuth Resource Server for MCP sessions (aligning with the MCP spec's June 2025 authorization update), Consumer and Consumer Group ACLs to restrict which tools a given caller can invoke, and MCP-specific logging that captures session IDs, JSON-RPC methods, payloads, and latencies — because \"who called what tool, with what arguments, and did it succeed\" is the audit trail security teams will ask for the moment an agent does something unexpected.\n\nRunning this on GCP, the MCP session-affinity requirements (MCP connections are typically longer-lived than a single HTTP request) are a reasonable fit for GKE's session affinity settings on the Kong Service, and Cloud Trace integration via OpenTelemetry gives you the span-level view of a tool call as it moves from agent → Kong → MCP server → downstream API.\n\n**Step 5: Govern Agentic Traffic, Not Just Individual Calls**\n\nThe hardest part of this transformation isn't any single plugin — it's that agentic traffic breaks the assumption that one inbound request equals one unit of work. An agent might make ten LLM calls and five tool calls to satisfy one user prompt, retry autonomously on failure, and occasionally loop. A few practical patterns help here:\n\nBudget at the session or workflow level, not just the request level. Tag agent-originated traffic with a consumer group and apply the token/cost budgets described above across the group's sliding window, so a runaway agent loop hits a 429 instead of a surprise bill.\n\nUse the AI Prompt Guard / Content Safety plugins as a policy boundary. Kong's AI Gateway includes plugins for prompt injection detection and integration with providers like Azure AI Content Safety to audit messages before they reach the upstream model — worth enabling specifically on agent-facing routes where inputs may originate from untrusted tool output rather than a human.\n\nScope MCP tool access tightly per agent identity. The Consumer/Consumer Group ACL model for MCP tools means a given agent credential can be restricted to exactly the tools it needs, which matters more for autonomous agents than for human-driven API clients, since there's no human in the loop to notice an overreaching call before it executes.\n\nTreat observability as the actual safety net. Because agent behavior is inherently less predictable than a fixed application flow, the OpenTelemetry span attributes Kong emits for token usage and tool-call metadata are what let you reconstruct, after the fact, exactly what an agent did and why — which is often the more realistic control than trying to prevent every bad outcome upfront.\n\n**Putting It Together**\n\nNone of these pieces are exotic individually rate limiting, OAuth, request logging, and reverse proxying are things Kong has done for years. What's changed is the unit of measurement: tokens instead of requests, tool calls instead of endpoints, and agent sessions instead of single client calls. Running this on GCP means the AI-specific Kong plugins sit on infrastructure that's already tuned for it Memorystore for the distributed counters that make token budgets actually hold, Vertex AI as a native backend alongside third-party model providers, and GKE plus Cloud Monitoring for the session handling and observability that MCP and agentic traffic both demand.\n\nThe teams that get the most value out of this tend to start narrow: put one high-traffic LLM route behind ai-proxy with token-based rate limiting first, prove out the Redis-backed budget enforcement, then layer in MCP proxying for a single tool integration before opening the gateway up to broader agentic workflows. The plugin architecture makes that incremental path realistic you're not re-platforming, you're extending a gateway you already trust to understand a new kind of traffic.", "url": "https://wpnews.pro/news/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic", "canonical_source": "https://dev.to/saurabhmi/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic-traffic-474e", "published_at": "2026-07-28 02:59:23+00:00", "updated_at": "2026-07-28 03:32:32.833110+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "ai-tools", "large-language-models", "developer-tools"], "entities": ["Kong", "Google Cloud Platform", "GKE", "Vertex AI", "Memorystore", "OpenAI", "Anthropic", "Cloud Load Balancing"], "alternates": {"html": "https://wpnews.pro/news/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic", "markdown": "https://wpnews.pro/news/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic.md", "text": "https://wpnews.pro/news/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic.txt", "jsonld": "https://wpnews.pro/news/transforming-kong-into-an-ai-gateway-on-gcp-managing-llm-tokens-mcp-and-agentic.jsonld"}}