cd /news/ai-infrastructure/private-multi-llm-routing-on-gke-wit… · home › topics › ai-infrastructure › article
[ARTICLE · art-170866] src=agentgateway.dev ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

Private Multi-LLM Routing on GKE with Cluster Toolkit

Google Cloud Platform's Cluster Toolkit v1.109.0 includes an agentgateway community blueprint that provisions a regional, CPU-only GKE Standard cluster with an internal Application Load Balancer to route requests to multiple LLM providers behind one private entry point. The blueprint routes Gemini at /v1/chat/completions by default and lets teams add OpenAI and Anthropic paths, using Workload Identity for Gemini and Kubernetes Secrets for optional API keys. Routing is path-based only — the blueprint does not configure semantic model selection or automatic failover, and Anthropic uses its native Messages API, so the three paths do not share one request format.

read5 min views1 publishedNov 16, 2026

Platform teams often need one private entry point for several LLM providers. Getting there on a new Kubernetes cluster means setting up networking, identity, ingress, and readiness alongside the gateway itself.

The agentgateway community blueprint for Google Cloud Cluster Toolkit, included in v1.109.0, provisions those pieces together. It starts with Gemini, then lets you add OpenAI and Anthropic behind the same internal load balancer. Agentgateway routes requests by provider path. This walkthrough deploys the default Gemini route, sends a request through it, and shows how to enable a second provider.

Component Blueprint default
Cluster and ingress Regional, CPU-only GKE Standard cluster and an internal Application Load Balancer
LLM routing Gemini at /v1/chat/completions ; optional OpenAI and Anthropic paths
Provider identity Workload Identity for Gemini; Kubernetes Secrets for optional API keys
Validation Staged readiness checks and manual requests from a temporary in-cluster curl Pod

This is path-based multi-LLM routing: clients choose the provider endpoint. The blueprint does not configure semantic model selection or automatic failover. Anthropic uses its native Messages API, so the three paths do not share one request format.

The diagram shows how the two Gateway controllers and provider routes fit together:

aiplatform.endpoints.predict permission; no Google service-account key is created. The blueprint deploys infrastructure, the controller, gateway configuration, and ingress in stages, checking readiness at each step. This keeps the cloud ingress and model-provider routing in their respective controllers.

Use a Google Cloud project where you can create the resources in the blueprint. Enable the Gemini Enterprise Agent Platform API, confirm access to the selected model, and check its quota. Follow the Cluster Toolkit setup guide for the required tools and project access. The released deployment guide has the example-specific prerequisites.

You need two kinds of access. The blueprint’s authorized_cidr variable sets a GKE authorized network for the Kubernetes API. Set it in the companion deployment file to the deployment runner’s public egress IP with a CIDR prefix, such as YOUR_IP/32, so that machine can reach the API.

Requests to the gateway follow a different path. The client must reach the internal load balancer from the VPC or a connected network. Allowing your public IP to reach the Kubernetes API does not give your laptop access to the private gateway. The in-cluster test below sends a request from inside GKE, so it works without a separate VPC connection from your laptop.

The example serves HTTP inside the VPC and does not authenticate downstream clients, so restrict network access to the listener. Increase total_max_nodes in the blueprint if you need more than two nodes in the system node pool.

The blueprint is included in the Cluster Toolkit v1.109.0 release bundles. Download the .tgz or .zip asset for your OS and architecture, then follow the prebuilt bundle installation steps to extract it. The bundle includes gcluster and the community examples. Run the remaining commands from the extracted cluster-toolkit directory.

Edit community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml. Set project_id, choose a unique deployment_name, and set authorized_cidr to your deployment machine’s public egress IP with a CIDR prefix, such as YOUR_IP/32. The default deployment name is agw-demo, the default region is us-central1, and the default model is gemini-2.5-flash. Keep provider credentials out of this file.

Create and deploy the example:

./gcluster create community/examples/agentgateway-gke/agentgateway-gke.yaml \
 -d community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml
./gcluster deploy agw-demo

Replace agw-demo in subsequent commands if you chose another name. The blueprint installs the agentgateway CRD and controller charts at the same pinned version, creates the proxy configuration, and attaches the internal ingress. Its readiness checks wait for the Gateway, route, backend, proxy, and ingress conditions before deployment completes.

Fetch cluster credentials:

gcloud container clusters get-credentials agw-demo \
 --region us-central1 --project YOUR_PROJECT

To see the request and response, run a temporary curl Pod through the internal load balancer:

GATEWAY_IP=$(kubectl -n agentgateway-system get gateway agentgateway-internal \
 -o jsonpath='{.status.addresses[0].value}')

kubectl -n agentgateway-system run agw-curl \
 --image=curlimages/curl:8.12.1 \
 --restart=Never --rm -i \
 --command -- curl \
 --fail-with-body --silent --show-error \
 --connect-timeout 10 --max-time 300 \
 "http://${GATEWAY_IP}/v1/chat/completions" \
 -H 'Content-Type: application/json' \
 -d '{
 "model": "gemini-2.5-flash",
 "messages": [{"role": "user", "content": "Count from one to ten."}],
 "max_tokens": 128,
 "thinking_config": {"thinkingBudget": 0},
 "stream": false
 }'

The response is an OpenAI-compatible chat completion. Its choices[0].message.content contains the answer; for example, an excerpt might look like this (the wording and other fields vary):

{"choices":[{"message":{"role":"assistant","content":"1, 2, 3, ..."}}]}

The configured vertex_model controls which Gemini model the gateway calls, even if a different model appears in the request. Refer to the released guide for the full manual-request notes.

The default deployment needs no external-provider API key. To add OpenAI, read your key from a protected local file into a Kubernetes Secret. Do not place the key in the blueprint or your shell history:

kubectl -n agentgateway-system create secret generic openai-credentials \
 --from-file=Authorization=/secure/path/openai-key

Uncomment or add openai_secret_name: openai-credentials under vars in agentgateway-gke-deployment.yaml. Then regenerate the same deployment and apply the update:

./gcluster create community/examples/agentgateway-gke/agentgateway-gke.yaml \
 -d community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml \
 --overwrite-deployment
./gcluster deploy agw-demo

The key stays out of Terraform; only the Secret name enters its state. To test an OpenAI model available to your account, send a request from a temporary curl Pod through the internal load balancer:

GATEWAY_IP=$(kubectl -n agentgateway-system get gateway agentgateway-internal \
 -o jsonpath='{.status.addresses[0].value}')

kubectl -n agentgateway-system run agw-curl \
 --image=curlimages/curl:8.12.1 \
 --restart=Never --rm -i \
 --command -- curl \
 --fail-with-body --silent --show-error \
 --connect-timeout 10 --max-time 300 \
 "http://${GATEWAY_IP}/openai/v1/chat/completions" \
 -H 'Content-Type: application/json' \
 -d '{
 "model": "OPENAI_MODEL_ID",
 "messages": [{"role": "user", "content": "Count from one to ten."}],
 "stream": false
 }'

Replace OPENAI_MODEL_ID with a model supported by your account. The response should contain an OpenAI chat completion. The Pod is removed when the request finishes.

Provider Gateway path Request format
Gemini /v1/chat/completions OpenAI-compatible Chat Completions
OpenAI /openai/v1/chat/completions OpenAI Chat Completions
Anthropic /v1/messages Native Anthropic Messages

Anthropic follows the same Secret and deployment pattern, but uses native Messages requests at /v1/messages. The released provider guide has the Anthropic request format and model-specific token settings.

After testing the gateway, explore more of agentgateway:

When you finish exploring, remove the deployment:

./gcluster destroy agw-demo

The GKE installation page points to the released blueprint. Its v1.109.0 README remains the source of truth for prerequisites, configuration, troubleshooting, and cleanup.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @google cloud platform 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/private-multi-llm-ro…] indexed:0 read:5min 2026-11-16 · —