{"slug": "private-multi-llm-routing-on-gke-with-cluster-toolkit", "title": "Private Multi-LLM Routing on GKE with Cluster Toolkit", "summary": "Google Cloud Platform's Cluster Toolkit v1.109.0 includes an agentgateway community blueprint that provisions a regional, CPU-only GKE Standard cluster with an internal Application Load Balancer to route requests to multiple LLM providers behind one private entry point. The blueprint routes Gemini at /v1/chat/completions by default and lets teams add OpenAI and Anthropic paths, using Workload Identity for Gemini and Kubernetes Secrets for optional API keys. Routing is path-based only — the blueprint does not configure semantic model selection or automatic failover, and Anthropic uses its native Messages API, so the three paths do not share one request format.", "body_md": "Platform teams often need one private entry point for several LLM providers. Getting there on a new Kubernetes cluster means setting up networking, identity, ingress, and readiness alongside the gateway itself.\n\nThe [agentgateway community blueprint for Google Cloud Cluster Toolkit](https://github.com/GoogleCloudPlatform/cluster-toolkit/tree/v1.109.0/community/examples/agentgateway-gke), included in [v1.109.0](https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/tag/v1.109.0), provisions those pieces together. It starts with Gemini, then lets you add OpenAI and Anthropic behind the same internal load balancer. Agentgateway routes requests by provider path. This walkthrough deploys the default Gemini route, sends a request through it, and shows how to enable a second provider.\n\n| Component | Blueprint default | \n|---|---|\n| Cluster and ingress | Regional, CPU-only GKE Standard cluster and an internal Application Load Balancer | \n| LLM routing | Gemini at `/v1/chat/completions` ; optional OpenAI and Anthropic paths | \n| Provider identity | Workload Identity for Gemini; Kubernetes Secrets for optional API keys | \n| Validation | Staged readiness checks and manual requests from a temporary in-cluster curl Pod | \n\nThis is path-based multi-LLM routing: clients choose the provider endpoint. The blueprint does not configure semantic model selection or automatic failover. Anthropic uses its native Messages API, so the three paths do not share one request format.\n\nThe diagram shows how the two Gateway controllers and provider routes fit together:\n\n`aiplatform.endpoints.predict` permission; no Google service-account key is created.\nThe blueprint deploys infrastructure, the controller, gateway configuration, and ingress in stages, checking readiness at each step. This keeps the cloud ingress and model-provider routing in their respective controllers.\n\nUse a Google Cloud project where you can create the resources in the blueprint. Enable the Gemini Enterprise Agent Platform API, confirm access to the selected model, and check its quota. Follow the [Cluster Toolkit setup guide](https://docs.cloud.google.com/cluster-toolkit/docs/setup/configure-environment) for the required tools and project access. The [released deployment guide](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/README.md#before-you-deploy) has the example-specific prerequisites.\n\nYou need two kinds of access. The [blueprint’s `authorized_cidr` variable](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/agentgateway-gke.yaml#L29-L30) sets a GKE authorized network for the Kubernetes API. Set it in the companion deployment file to the deployment runner’s public egress IP with a CIDR prefix, such as `YOUR_IP/32`, so that machine can reach the API.\n\nRequests to the gateway follow a different path. The client must reach the internal load balancer from the VPC or a connected network. Allowing your public IP to reach the Kubernetes API does not give your laptop access to the private gateway. The in-cluster test below sends a request from inside GKE, so it works without a separate VPC connection from your laptop.\n\nThe example serves HTTP inside the VPC and does not authenticate downstream clients, so restrict network access to the listener. Increase [`total_max_nodes`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/agentgateway-gke.yaml#L132) in the blueprint if you need more than two nodes in the system node pool.\n\nThe blueprint is included in the [Cluster Toolkit v1.109.0 release bundles](https://github.com/GoogleCloudPlatform/cluster-toolkit/releases/tag/v1.109.0). Download the `.tgz` or `.zip` asset for your OS and architecture, then follow the [prebuilt bundle installation steps](https://docs.cloud.google.com/cluster-toolkit/docs/setup/configure-environment#install-the-prebuilt-bundle) to extract it. The bundle includes `gcluster` and the community examples. Run the remaining commands from the extracted `cluster-toolkit` directory.\n\nEdit `community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml`. Set `project_id`, choose a unique `deployment_name`, and set `authorized_cidr` to your deployment machine’s public egress IP with a CIDR prefix, such as `YOUR_IP/32`. The default deployment name is `agw-demo`, the default region is `us-central1`, and the default model is `gemini-2.5-flash`. Keep provider credentials out of this file.\n\nCreate and deploy the example:\n\n```\n./gcluster create community/examples/agentgateway-gke/agentgateway-gke.yaml \\\n -d community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml\n./gcluster deploy agw-demo\n```\n\nReplace `agw-demo` in subsequent commands if you chose another name. The blueprint installs the agentgateway CRD and controller charts at the same pinned version, creates the proxy configuration, and attaches the internal ingress. Its readiness checks wait for the Gateway, route, backend, proxy, and ingress conditions before deployment completes.\n\nFetch cluster credentials:\n\n```\ngcloud container clusters get-credentials agw-demo \\\n --region us-central1 --project YOUR_PROJECT\n```\n\nTo see the request and response, run a temporary curl Pod through the internal load balancer:\n\n```\nGATEWAY_IP=$(kubectl -n agentgateway-system get gateway agentgateway-internal \\\n -o jsonpath='{.status.addresses[0].value}')\n\nkubectl -n agentgateway-system run agw-curl \\\n --image=curlimages/curl:8.12.1 \\\n --restart=Never --rm -i \\\n --command -- curl \\\n --fail-with-body --silent --show-error \\\n --connect-timeout 10 --max-time 300 \\\n \"http://${GATEWAY_IP}/v1/chat/completions\" \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"gemini-2.5-flash\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Count from one to ten.\"}],\n \"max_tokens\": 128,\n \"thinking_config\": {\"thinkingBudget\": 0},\n \"stream\": false\n }'\n```\n\nThe response is an OpenAI-compatible chat completion. Its `choices[0].message.content` contains the answer; for example, an excerpt might look like this (the wording and other fields vary):\n\n```\n{\"choices\":[{\"message\":{\"role\":\"assistant\",\"content\":\"1, 2, 3, ...\"}}]}\n```\n\nThe configured [`vertex_model`](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/agentgateway-gke.yaml#L41-L42) controls which Gemini model the gateway calls, even if a different model appears in the request. Refer to the [released guide](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/README.md#send-a-request-with-curl) for the full manual-request notes.\n\nThe default deployment needs no external-provider API key. To add OpenAI, read your key from a protected local file into a Kubernetes Secret. Do not place the key in the blueprint or your shell history:\n\n```\nkubectl -n agentgateway-system create secret generic openai-credentials \\\n --from-file=Authorization=/secure/path/openai-key\n```\n\nUncomment or add `openai_secret_name: openai-credentials` under `vars` in `agentgateway-gke-deployment.yaml`. Then regenerate the same deployment and apply the update:\n\n```\n./gcluster create community/examples/agentgateway-gke/agentgateway-gke.yaml \\\n -d community/examples/agentgateway-gke/agentgateway-gke-deployment.yaml \\\n --overwrite-deployment\n./gcluster deploy agw-demo\n```\n\nThe key stays out of Terraform; only the Secret name enters its state. To test an OpenAI model available to your account, send a request from a temporary curl Pod through the internal load balancer:\n\n```\nGATEWAY_IP=$(kubectl -n agentgateway-system get gateway agentgateway-internal \\\n -o jsonpath='{.status.addresses[0].value}')\n\nkubectl -n agentgateway-system run agw-curl \\\n --image=curlimages/curl:8.12.1 \\\n --restart=Never --rm -i \\\n --command -- curl \\\n --fail-with-body --silent --show-error \\\n --connect-timeout 10 --max-time 300 \\\n \"http://${GATEWAY_IP}/openai/v1/chat/completions\" \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"model\": \"OPENAI_MODEL_ID\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Count from one to ten.\"}],\n \"stream\": false\n }'\n```\n\nReplace `OPENAI_MODEL_ID` with a model supported by your account. The response should contain an OpenAI chat completion. The Pod is removed when the request finishes.\n\n| Provider | Gateway path | Request format | \n|---|---|---|\n| Gemini | `/v1/chat/completions` | OpenAI-compatible Chat Completions | \n| OpenAI | `/openai/v1/chat/completions` | OpenAI Chat Completions | \n| Anthropic | `/v1/messages` | Native Anthropic Messages | \n\nAnthropic follows the same Secret and deployment pattern, but uses native Messages requests at `/v1/messages`. The [released provider guide](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/README.md#optional-external-providers) has the Anthropic request format and model-specific token settings.\n\nAfter testing the gateway, explore more of agentgateway:\n\nWhen you finish exploring, remove the deployment:\n\n```\n./gcluster destroy agw-demo\n```\n\nThe [GKE installation page](https://agentgateway.dev/docs/kubernetes/latest/documentation/install/gke/) points to the released blueprint. Its [v1.109.0 README](https://github.com/GoogleCloudPlatform/cluster-toolkit/blob/v1.109.0/community/examples/agentgateway-gke/README.md) remains the source of truth for prerequisites, configuration, troubleshooting, and cleanup.", "url": "https://wpnews.pro/news/private-multi-llm-routing-on-gke-with-cluster-toolkit", "canonical_source": "https://agentgateway.dev/blog/2026-11-02-private-ai-gateway-gke-cluster-toolkit/", "published_at": "2026-11-16 00:00:00+00:00", "updated_at": "2026-11-16 19:52:25.349311+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "developer-tools"], "entities": ["Google Cloud Platform", "Cluster Toolkit", "agentgateway", "Gemini", "OpenAI", "Anthropic", "GKE"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/private-multi-llm-routing-on-gke-with-cluster-toolkit", "markdown": "https://wpnews.pro/news/private-multi-llm-routing-on-gke-with-cluster-toolkit.md", "text": "https://wpnews.pro/news/private-multi-llm-routing-on-gke-with-cluster-toolkit.txt", "jsonld": "https://wpnews.pro/news/private-multi-llm-routing-on-gke-with-cluster-toolkit.jsonld"}}