Automated incident triage on GKE with Gemini 4 Argon and Model Context Protocol: Correlating cluster events and VPA logs

A developer has outlined an architecture for a read-only incident triage agent on Google Kubernetes Engine that uses Gemini 4 Argon and the open-source GKE MCP server to correlate Pod events, Cloud Logging stack traces, and KCP_VPA vertical autoscaling decision logs. The agent runs under least-privilege RBAC and Workload Identity Federation, with no direct cluster write access, and proposes remediation as a structured report and YAML diff requiring human or GitOps approval. The approach relies on enabling VPA decision logging, available in Public Preview on GKE 1.36.0-gke.1601000 and later.

When an on-call engineer gets paged at 2 AM for a degraded payment service, the hardest part of incident triage is rarely fixing the YAML manifest. Instead, the bottleneck is correlating telemetry across three different control loops: transient Kubernetes Pod events that expire after an hour, application stack traces in Cloud Logging https://cloud.google.com/logging/docs , and vertical autoscaling actions emitted by the control plane. Consider a common failure mode on Google Kubernetes Engine GKE https://cloud.google.com/kubernetes-engine : a latency spike triggers an out-of-memory OOMKilled termination while the Vertical Pod Autoscaler VPA attempts an in-place resize InPlaceOrRecreate that gets capped by node capacity or GKE Autopilot compute ratio constraints. Diagnosing that sequence manually requires stitching together kubectl describe pod , container exit codes, and container.googleapis.com/vpa-controller decision logs. With the release of Gemini 4 Argon https://cloud.google.com/gemini-enterprise-agent-platform/models/gemini/4-argon and the open-source Google Kubernetes Engine MCP server https://github.com/GoogleCloudPlatform/gke-mcp , platform teams can deploy an in-cluster incident triage agent that autonomously queries Kubernetes state, inspects KCP VPA control plane logs, identifies the root cause, and generates a verified remediation patch. In this article, I will walk through the architecture of a read-only Gemini 4 Argon incident triage agent on GKE, show how to configure least-privilege RBAC and Workload Identity Federation for GKE https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity , and demonstrate how the agent correlates live cluster telemetry through the Model Context Protocol MCP . Single-turn log summarization falls short during real Kubernetes incidents because the first symptom is almost never the root cause. A CrashLoopBackOff status on a Pod tells you only that the kubelet is delaying restarts; it does not tell you whether the failure stems from an expired secret, a failed readiness probe during JVM initialization, or a memory limit that VPA could not resize in place. Gemini 4 Argon's long-horizon tool calling allows an agent to execute a structured diagnostic workflow across multiple MCP tools before forming a hypothesis: list k8s events and get k8s resource through the GKE MCP server to pull active events, Pod termination reasons, and deployment spec history. get k8s logs to capture application stack traces and unhandled exceptions. query vpa decision logs against Cloud Logging to evaluate KCP VPA decisions, checking whether autoscaling recommendations were capped or throttled. To keep the cluster safe from unintended mutations during an active outage, the triage agent operates under a strict separation of duties: | Operational phase | Access level | Mechanism | Security boundary | |---|---|---|---| | Telemetry discovery | Read-only get , list , watch | GKE MCP server + Kubernetes ClusterRole | Denies create , patch , delete , and access to Secret payloads | | Control plane log correlation | Read-only roles/logging.viewer | Cloud Logging API via Workload Identity | Scoped to k8s container , k8s pod , and vpa-controller logs | | Remediation proposal | Zero direct cluster write access | Structured Markdown report and YAML diff | Requires human approval or GitOps pipeline validation before apply | For the triage agent to inspect vertical autoscaling decisions alongside standard workload logs, enable VPA decision logging KCP VPA , available in Public Preview on GKE 1.36.0-gke.1601000 and later on your GKE Autopilot or GKE Standard cluster. On GKE Autopilot clusters, Vertical Pod Autoscaling is enabled by default; on GKE Standard clusters, ensure --enable-vertical-pod-autoscaling is also enabled. When updating --logging on an existing cluster, always include your existing logging components such as SYSTEM and WORKLOAD so you do not overwrite active settings: export PROJECT ID="production-ai-platform" export PROJECT NUMBER=$ gcloud projects describe "${PROJECT ID}" --format="value projectNumber " export LOCATION="us-central1" export CLUSTER NAME="commerce-production-cluster" Check currently enabled logging components first gcloud container clusters describe "${CLUSTER NAME}" \ --location="${LOCATION}" \ --project="${PROJECT ID}" \ --format="value loggingConfig.componentConfig.enableComponents " Enable SYSTEM, WORKLOAD, and KCP VPA logs gcloud container clusters update "${CLUSTER NAME}" \ --location="${LOCATION}" \ --project="${PROJECT ID}" \ --logging=SYSTEM,WORKLOAD,KCP VPA Never grant an AI agent cluster-admin permissions or read access to Kubernetes Secret objects. Create a dedicated sre-triage namespace and a ClusterRole that grants read-only visibility into workloads, events, HPA/VPA policies, and custom compute classes while explicitly excluding secrets. Save the following manifest as triage-agent-rbac.yaml : apiVersion: v1 kind: Namespace metadata: name: sre-triage --- apiVersion: v1 kind: ServiceAccount metadata: name: gemini-triage-sa namespace: sre-triage --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: gemini-readonly-triage-role rules: - apiGroups: "" resources: "pods", "pods/log", "events", "services", "nodes", "namespaces", "configmaps" verbs: "get", "list", "watch" - apiGroups: "events.k8s.io" resources: "events" verbs: "get", "list", "watch" - apiGroups: "apps" resources: "deployments", "statefulsets", "daemonsets", "replicasets" verbs: "get", "list", "watch" - apiGroups: "autoscaling" resources: "horizontalpodautoscalers" verbs: "get", "list", "watch" - apiGroups: "autoscaling.k8s.io" resources: "verticalpodautoscalers" verbs: "get", "list", "watch" - apiGroups: "cloud.google.com" resources: "computeclasses" verbs: "get", "list", "watch" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: gemini-readonly-triage-binding subjects: - kind: ServiceAccount name: gemini-triage-sa namespace: sre-triage roleRef: kind: ClusterRole name: gemini-readonly-triage-role apiGroup: rbac.authorization.k8s.io Apply the RBAC manifest and bind the gemini-triage-sa ServiceAccount to the Gemini platform, GKE Viewer, and Cloud Logging Viewer IAM roles via Workload Identity Federation: kubectl apply -f triage-agent-rbac.yaml for ROLE in roles/aiplatform.user roles/container.viewer roles/logging.viewer; do gcloud projects add-iam-policy-binding "${PROJECT ID}" \ --member="principal://iam.googleapis.com/projects/${PROJECT NUMBER}/locations/global/workloadIdentityPools/${PROJECT ID}.svc.id.goog/subject/ns/sre-triage/sa/gemini-triage-sa" \ --role="${ROLE}" \ --condition=None done The triage agent connects Gemini 4 Argon gemini-4-argon to an in-process gke-mcp session alongside a dedicated tool for querying container.googleapis.com/vpa-controller decision logs in Cloud Logging. Here is the core implementation triage agent.py : python import asyncio import os from google import genai from google.genai import types from google.cloud import logging as cloud logging from mcp import ClientSession, StdioServerParameters from mcp.client.stdio import stdio client PROJECT ID = os.environ "GOOGLE CLOUD PROJECT" LOCATION = os.environ.get "VERTEX LOCATION", "us-central1" MODEL ID = os.environ.get "GEMINI MODEL ID", "gemini-4-argon" genai client = genai.Client vertexai=True, project=PROJECT ID, location=LOCATION logging client = cloud logging.Client project=PROJECT ID def query vpa decision logs cluster name: str, namespace: str, workload name: str, limit: int = 10 - list dict : """Queries GKE KCP VPA control plane logs for a specific workload to inspect resize or eviction decisions. Args: cluster name: Name of the GKE cluster. namespace: Kubernetes namespace of the target workload. workload name: Name of the target Deployment or workload. limit: Maximum number of log entries to return. """ log filter = 'resource.type="k8s control plane component" ' f'AND resource.labels.cluster name="{cluster name}" ' f'AND logName="projects/{PROJECT ID}/logs/container.googleapis.com%2Fvpa-controller" ' f'AND jsonPayload.target.namespace="{namespace}" ' f'AND jsonPayload.target.name="{workload name}"' entries = logging client.list entries filter =log filter, order by=cloud logging.DESCENDING, max results=limit, results = for entry in entries: payload = entry.payload if isinstance entry.payload, dict else {} results.append { "timestamp": str entry.timestamp , "operation": payload.get "operation" , "state": payload.get "state" , "reason": payload.get "reason" , "confidence": payload.get "confidence" , } return results async def investigate workload incident cluster name: str, namespace: str, workload name: str - str: """Connects Gemini to the GKE MCP server and Cloud Logging to diagnose a failing workload.""" server params = StdioServerParameters command="gke-mcp", args= , env=os.environ.copy , async with stdio client server params as read stream, write stream : async with ClientSession read stream, write stream as mcp session: await mcp session.initialize prompt = f"Investigate the degraded Deployment '{workload name}' in namespace '{namespace}' " f"on GKE cluster '{cluster name}'. " "1. List recent Kubernetes events and inspect the Deployment and Pod statuses. " "2. Check the container termination exit codes and recent Pod logs. " "3. Query VPA decision logs using query vpa decision logs to see if an in-place resize " "or eviction occurred or was skipped. " "4. Provide a concise root-cause timeline and a git-ready YAML patch." config = types.GenerateContentConfig system instruction= "You are a senior GKE Site Reliability Engineer. Always verify Pod status, exit codes, " "Kubernetes events, and VPA decision logs before proposing a fix. Never guess when " "telemetry can be queried via your tools." , tools= mcp session, query vpa decision logs , automatic function calling=types.AutomaticFunctionCallingConfig maximum remote calls=8 , temperature=0.1, response = await genai client.aio.models.generate content model=MODEL ID, contents=prompt, config=config, return response.text By passing both mcp session and query vpa decision logs into tools , the google-genai SDK automatically discovers the GKE MCP server tools list k8s events , get k8s resource , get k8s logs and orchestrates the multi-step investigation up to the configured maximum remote calls limit. Passing env=os.environ.copy to StdioServerParameters ensures the gke-mcp subprocess inherits in-cluster Kubernetes service environment variables KUBERNETES SERVICE HOST and KUBERNETES SERVICE PORT and GOOGLE CLOUD PROJECT . To see what the triage agent captures in practice, suppose the checkout-service Deployment in the payments namespace starts restarting with exit code 137 OOMKilled while managed by a VerticalPodAutoscaler in InPlaceOrRecreate mode. When triggered, Gemini 4 Argon executes four tool calls in sequence and produces a structured diagnostic report: Root-cause summary for Deployment payments/checkout-service: 1. Pod status & exit code: Pod checkout-service-7c9d8b6f4-x82lp terminated with reason OOMKilled exit code 137 at 06:14:22 UTC after memory usage reached its 1Gi limit. 2. Kubernetes events: BackOff restarting failed container; InPlaceResizeFailed event reported on node gke-commerce-pool-4a91f. 3. VPA control plane logs KCP VPA : - Operation: APPLY RECOMMENDATION IN PLACE | State: FAILED | Confidence: HIGH - Reason: Recommended memory target 2Gi exceeded maxAllowed policy bound 1Gi defined in VerticalPodAutoscaler checkout-service-vpa. 4. Recommended remediation: Raise resourcePolicy.containerPolicies 0 .maxAllowed.memory in checkout-service-vpa from 1Gi to 3Gi so VPA can apply high-confidence in-place memory resizes without triggering OOM terminations. You can independently verify the agent's finding using kubectl and gcloud : Verify the VPA resource policy cap on the cluster kubectl get vpa checkout-service-vpa -n payments -o yaml Inspect the corresponding KCP VPA log entry in Cloud Logging gcloud logging read "resource.type=\"k8s control plane component\" AND resource.labels.cluster name=\"${CLUSTER NAME}\" AND logName=\"projects/${PROJECT ID}/logs/container.googleapis.com%2Fvpa-controller\" AND jsonPayload.target.name=\"checkout-service\"" \ --project="${PROJECT ID}" \ --limit=3 \ --format=json When running an AI-assisted triage agent inside your production GKE environment, apply these operational safeguards: kubectl apply or mutate resource limits during an active incident. Have the agent open a pull request against your GitOps repository managed by Config Sync or Argo CD so human operators can review the diff. pods/log output to the model API, and ensure your Gemini platform data governance policies disable prompt logging where required. Deployment or StatefulSet and cap each investigation to a maximum of 8 tool-calling turns. Combining Gemini 4 Argon with the GKE Model Context Protocol server and structured control plane logs KCP VPA turns multi-system incident correlation from a 30-minute manual scavenger hunt into an automated, auditable workflow. To deploy this pattern on your clusters, explore the open-source Google Kubernetes Engine MCP server repository https://github.com/GoogleCloudPlatform/gke-mcp , read the GKE VPA decision logs documentation https://cloud.google.com/kubernetes-engine/docs/how-to/view-vertical-pod-autoscaling-events , and review Workload Identity Federation for GKE https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity for zero-key IAM authentication.