{"slug": "automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol", "title": "Automated incident triage on GKE with Gemini 4 Argon and Model Context Protocol: Correlating cluster events and VPA logs", "summary": "A developer has outlined an architecture for a read-only incident triage agent on Google Kubernetes Engine that uses Gemini 4 Argon and the open-source GKE MCP server to correlate Pod events, Cloud Logging stack traces, and KCP_VPA vertical autoscaling decision logs. The agent runs under least-privilege RBAC and Workload Identity Federation, with no direct cluster write access, and proposes remediation as a structured report and YAML diff requiring human or GitOps approval. The approach relies on enabling VPA decision logging, available in Public Preview on GKE 1.36.0-gke.1601000 and later.", "body_md": "When an on-call engineer gets paged at 2 AM for a degraded payment service, the hardest part of incident triage is rarely fixing the YAML manifest. Instead, the bottleneck is correlating telemetry across three different control loops: transient Kubernetes Pod events that expire after an hour, application stack traces in [Cloud Logging](https://cloud.google.com/logging/docs), and vertical autoscaling actions emitted by the control plane.\n\nConsider a common failure mode on [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine): a latency spike triggers an out-of-memory (`OOMKilled`) termination while the Vertical Pod Autoscaler (VPA) attempts an in-place resize (`InPlaceOrRecreate`) that gets capped by node capacity or GKE Autopilot compute ratio constraints. Diagnosing that sequence manually requires stitching together `kubectl describe pod`, container exit codes, and `container.googleapis.com/vpa-controller` decision logs.\n\nWith the release of [Gemini 4 Argon](https://cloud.google.com/gemini-enterprise-agent-platform/models/gemini/4-argon) and the open-source [Google Kubernetes Engine MCP server](https://github.com/GoogleCloudPlatform/gke-mcp), platform teams can deploy an in-cluster incident triage agent that autonomously queries Kubernetes state, inspects `KCP_VPA` control plane logs, identifies the root cause, and generates a verified remediation patch.\n\nIn this article, I will walk through the architecture of a read-only Gemini 4 Argon incident triage agent on GKE, show how to configure least-privilege RBAC and [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity), and demonstrate how the agent correlates live cluster telemetry through the Model Context Protocol (MCP).\n\nSingle-turn log summarization falls short during real Kubernetes incidents because the first symptom is almost never the root cause. A `CrashLoopBackOff` status on a Pod tells you only that the kubelet is delaying restarts; it does not tell you whether the failure stems from an expired secret, a failed readiness probe during JVM initialization, or a memory limit that VPA could not resize in place.\n\nGemini 4 Argon's long-horizon tool calling allows an agent to execute a structured diagnostic workflow across multiple MCP tools before forming a hypothesis:\n\n`list_k8s_events` and `get_k8s_resource` through the GKE MCP server to pull active events, Pod termination reasons, and deployment spec history.`get_k8s_logs` to capture application stack traces and unhandled exceptions.`query_vpa_decision_logs` against Cloud Logging to evaluate `KCP_VPA` decisions, checking whether autoscaling recommendations were capped or throttled.\nTo keep the cluster safe from unintended mutations during an active outage, the triage agent operates under a strict separation of duties:\n\n| Operational phase | Access level | Mechanism | Security boundary | \n|---|---|---|---|\n| **Telemetry discovery** | Read-only ( `get` ,`list` ,`watch` ) | GKE MCP server + Kubernetes `ClusterRole` | Denies `create` ,`patch` ,`delete` , and access to`Secret` payloads | \n| **Control plane log correlation** | Read-only ( `roles/logging.viewer` ) | Cloud Logging API via Workload Identity | Scoped to `k8s_container` ,`k8s_pod` , and`vpa-controller` logs | \n| **Remediation proposal** | Zero direct cluster write access | Structured Markdown report and YAML diff | Requires human approval or GitOps pipeline validation before apply | \n\nFor the triage agent to inspect vertical autoscaling decisions alongside standard workload logs, enable VPA decision logging (`KCP_VPA`, available in Public Preview on GKE `1.36.0-gke.1601000` and later) on your GKE Autopilot or GKE Standard cluster. On GKE Autopilot clusters, Vertical Pod Autoscaling is enabled by default; on GKE Standard clusters, ensure `--enable-vertical-pod-autoscaling` is also enabled.\n\nWhen updating `--logging` on an existing cluster, always include your existing logging components (such as `SYSTEM` and `WORKLOAD`) so you do not overwrite active settings:\n\n```\nexport PROJECT_ID=\"production-ai-platform\"\nexport PROJECT_NUMBER=$(gcloud projects describe \"${PROJECT_ID}\" --format=\"value(projectNumber)\")\nexport LOCATION=\"us-central1\"\nexport CLUSTER_NAME=\"commerce-production-cluster\"\n\n# Check currently enabled logging components first\ngcloud container clusters describe \"${CLUSTER_NAME}\" \\\n    --location=\"${LOCATION}\" \\\n    --project=\"${PROJECT_ID}\" \\\n    --format=\"value(loggingConfig.componentConfig.enableComponents)\"\n\n# Enable SYSTEM, WORKLOAD, and KCP_VPA logs\ngcloud container clusters update \"${CLUSTER_NAME}\" \\\n    --location=\"${LOCATION}\" \\\n    --project=\"${PROJECT_ID}\" \\\n    --logging=SYSTEM,WORKLOAD,KCP_VPA\n```\n\nNever grant an AI agent `cluster-admin` permissions or read access to Kubernetes `Secret` objects. Create a dedicated `sre-triage` namespace and a `ClusterRole` that grants read-only visibility into workloads, events, HPA/VPA policies, and custom compute classes while explicitly excluding secrets.\n\nSave the following manifest as `triage-agent-rbac.yaml`:\n\n```\napiVersion: v1\nkind: Namespace\nmetadata:\n  name: sre-triage\n---\napiVersion: v1\nkind: ServiceAccount\nmetadata:\n  name: gemini-triage-sa\n  namespace: sre-triage\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRole\nmetadata:\n  name: gemini-readonly-triage-role\nrules:\n- apiGroups: [\"\"]\n  resources: [\"pods\", \"pods/log\", \"events\", \"services\", \"nodes\", \"namespaces\", \"configmaps\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups: [\"events.k8s.io\"]\n  resources: [\"events\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups: [\"apps\"]\n  resources: [\"deployments\", \"statefulsets\", \"daemonsets\", \"replicasets\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups: [\"autoscaling\"]\n  resources: [\"horizontalpodautoscalers\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups: [\"autoscaling.k8s.io\"]\n  resources: [\"verticalpodautoscalers\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n- apiGroups: [\"cloud.google.com\"]\n  resources: [\"computeclasses\"]\n  verbs: [\"get\", \"list\", \"watch\"]\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: ClusterRoleBinding\nmetadata:\n  name: gemini-readonly-triage-binding\nsubjects:\n- kind: ServiceAccount\n  name: gemini-triage-sa\n  namespace: sre-triage\nroleRef:\n  kind: ClusterRole\n  name: gemini-readonly-triage-role\n  apiGroup: rbac.authorization.k8s.io\n```\n\nApply the RBAC manifest and bind the `gemini-triage-sa` ServiceAccount to the Gemini platform, GKE Viewer, and Cloud Logging Viewer IAM roles via Workload Identity Federation:\n\n```\nkubectl apply -f triage-agent-rbac.yaml\n\nfor ROLE in roles/aiplatform.user roles/container.viewer roles/logging.viewer; do\n  gcloud projects add-iam-policy-binding \"${PROJECT_ID}\" \\\n      --member=\"principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${PROJECT_ID}.svc.id.goog/subject/ns/sre-triage/sa/gemini-triage-sa\" \\\n      --role=\"${ROLE}\" \\\n      --condition=None\ndone\n```\n\nThe triage agent connects Gemini 4 Argon (`gemini-4-argon`) to an in-process `gke-mcp` session alongside a dedicated tool for querying `container.googleapis.com/vpa-controller` decision logs in Cloud Logging.\n\nHere is the core implementation (`triage_agent.py`):\n\n``` python\nimport asyncio\nimport os\nfrom google import genai\nfrom google.genai import types\nfrom google.cloud import logging as cloud_logging\nfrom mcp import ClientSession, StdioServerParameters\nfrom mcp.client.stdio import stdio_client\n\nPROJECT_ID = os.environ[\"GOOGLE_CLOUD_PROJECT\"]\nLOCATION = os.environ.get(\"VERTEX_LOCATION\", \"us-central1\")\nMODEL_ID = os.environ.get(\"GEMINI_MODEL_ID\", \"gemini-4-argon\")\n\ngenai_client = genai.Client(vertexai=True, project=PROJECT_ID, location=LOCATION)\nlogging_client = cloud_logging.Client(project=PROJECT_ID)\n\ndef query_vpa_decision_logs(\n    cluster_name: str, namespace: str, workload_name: str, limit: int = 10\n) -> list[dict]:\n    \"\"\"Queries GKE KCP_VPA control plane logs for a specific workload to inspect resize or eviction decisions.\n\n    Args:\n        cluster_name: Name of the GKE cluster.\n        namespace: Kubernetes namespace of the target workload.\n        workload_name: Name of the target Deployment or workload.\n        limit: Maximum number of log entries to return.\n    \"\"\"\n    log_filter = (\n        'resource.type=\"k8s_control_plane_component\" '\n        f'AND resource.labels.cluster_name=\"{cluster_name}\" '\n        f'AND logName=\"projects/{PROJECT_ID}/logs/container.googleapis.com%2Fvpa-controller\" '\n        f'AND jsonPayload.target.namespace=\"{namespace}\" '\n        f'AND jsonPayload.target.name=\"{workload_name}\"'\n    )\n    entries = logging_client.list_entries(\n        filter_=log_filter,\n        order_by=cloud_logging.DESCENDING,\n        max_results=limit,\n    )\n    results = []\n    for entry in entries:\n        payload = entry.payload if isinstance(entry.payload, dict) else {}\n        results.append({\n            \"timestamp\": str(entry.timestamp),\n            \"operation\": payload.get(\"operation\"),\n            \"state\": payload.get(\"state\"),\n            \"reason\": payload.get(\"reason\"),\n            \"confidence\": payload.get(\"confidence\"),\n        })\n    return results\n\nasync def investigate_workload_incident(cluster_name: str, namespace: str, workload_name: str) -> str:\n    \"\"\"Connects Gemini to the GKE MCP server and Cloud Logging to diagnose a failing workload.\"\"\"\n    server_params = StdioServerParameters(\n        command=\"gke-mcp\",\n        args=[],\n        env=os.environ.copy(),\n    )\n\n    async with stdio_client(server_params) as (read_stream, write_stream):\n        async with ClientSession(read_stream, write_stream) as mcp_session:\n            await mcp_session.initialize()\n\n            prompt = (\n                f\"Investigate the degraded Deployment '{workload_name}' in namespace '{namespace}' \"\n                f\"on GKE cluster '{cluster_name}'. \"\n                \"1. List recent Kubernetes events and inspect the Deployment and Pod statuses. \"\n                \"2. Check the container termination exit codes and recent Pod logs. \"\n                \"3. Query VPA decision logs using query_vpa_decision_logs to see if an in-place resize \"\n                \"or eviction occurred or was skipped. \"\n                \"4. Provide a concise root-cause timeline and a git-ready YAML patch.\"\n            )\n\n            config = types.GenerateContentConfig(\n                system_instruction=(\n                    \"You are a senior GKE Site Reliability Engineer. Always verify Pod status, exit codes, \"\n                    \"Kubernetes events, and VPA decision logs before proposing a fix. Never guess when \"\n                    \"telemetry can be queried via your tools.\"\n                ),\n                tools=[mcp_session, query_vpa_decision_logs],\n                automatic_function_calling=types.AutomaticFunctionCallingConfig(\n                    maximum_remote_calls=8\n                ),\n                temperature=0.1,\n            )\n\n            response = await genai_client.aio.models.generate_content(\n                model=MODEL_ID,\n                contents=prompt,\n                config=config,\n            )\n            return response.text\n```\n\nBy passing both `mcp_session` and `query_vpa_decision_logs` into `tools`, the `google-genai` SDK automatically discovers the GKE MCP server tools (`list_k8s_events`, `get_k8s_resource`, `get_k8s_logs`) and orchestrates the multi-step investigation up to the configured `maximum_remote_calls` limit. Passing `env=os.environ.copy()` to `StdioServerParameters` ensures the `gke-mcp` subprocess inherits in-cluster Kubernetes service environment variables (`KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT`) and `GOOGLE_CLOUD_PROJECT`.\n\nTo see what the triage agent captures in practice, suppose the `checkout-service` Deployment in the `payments` namespace starts restarting with exit code `137` (`OOMKilled`) while managed by a `VerticalPodAutoscaler` in `InPlaceOrRecreate` mode.\n\nWhen triggered, Gemini 4 Argon executes four tool calls in sequence and produces a structured diagnostic report:\n\n```\nRoot-cause summary for Deployment payments/checkout-service:\n1. Pod status & exit code: Pod checkout-service-7c9d8b6f4-x82lp terminated with reason OOMKilled (exit code 137) at 06:14:22 UTC after memory usage reached its 1Gi limit.\n2. Kubernetes events: BackOff restarting failed container; InPlaceResizeFailed event reported on node gke-commerce-pool-4a91f.\n3. VPA control plane logs (KCP_VPA):\n   - Operation: APPLY_RECOMMENDATION_IN_PLACE | State: FAILED | Confidence: HIGH\n   - Reason: Recommended memory target (2Gi) exceeded maxAllowed policy bound (1Gi) defined in VerticalPodAutoscaler checkout-service-vpa.\n4. Recommended remediation: Raise resourcePolicy.containerPolicies[0].maxAllowed.memory in checkout-service-vpa from 1Gi to 3Gi so VPA can apply high-confidence in-place memory resizes without triggering OOM terminations.\n```\n\nYou can independently verify the agent's finding using `kubectl` and `gcloud`:\n\n```\n# Verify the VPA resource policy cap on the cluster\nkubectl get vpa checkout-service-vpa -n payments -o yaml\n\n# Inspect the corresponding KCP_VPA log entry in Cloud Logging\ngcloud logging read \"resource.type=\\\"k8s_control_plane_component\\\" AND resource.labels.cluster_name=\\\"${CLUSTER_NAME}\\\" AND logName=\\\"projects/${PROJECT_ID}/logs/container.googleapis.com%2Fvpa-controller\\\" AND jsonPayload.target.name=\\\"checkout-service\\\"\" \\\n    --project=\"${PROJECT_ID}\" \\\n    --limit=3 \\\n    --format=json\n```\n\nWhen running an AI-assisted triage agent inside your production GKE environment, apply these operational safeguards:\n\n`kubectl apply` or mutate resource limits during an active incident. Have the agent open a pull request against your GitOps repository (managed by Config Sync or Argo CD) so human operators can review the diff.`pods/log` output to the model API, and ensure your Gemini platform data governance policies disable prompt logging where required.`Deployment` or `StatefulSet`) and cap each investigation to a maximum of 8 tool-calling turns.\nCombining Gemini 4 Argon with the GKE Model Context Protocol server and structured control plane logs (`KCP_VPA`) turns multi-system incident correlation from a 30-minute manual scavenger hunt into an automated, auditable workflow.\n\nTo deploy this pattern on your clusters, explore the open-source [Google Kubernetes Engine MCP server repository](https://github.com/GoogleCloudPlatform/gke-mcp), read the [GKE VPA decision logs documentation](https://cloud.google.com/kubernetes-engine/docs/how-to/view-vertical-pod-autoscaling-events), and review [Workload Identity Federation for GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity) for zero-key IAM authentication.", "url": "https://wpnews.pro/news/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol", "canonical_source": "https://dev.to/googlecloud/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol-correlating-24eb", "published_at": "2026-10-20 19:14:39+00:00", "updated_at": "2026-10-20 19:18:44.460125+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "ai-infrastructure", "agent-protocols"], "entities": ["Google", "Gemini 4 Argon", "Google Kubernetes Engine", "Model Context Protocol", "Vertical Pod Autoscaler", "Cloud Logging", "GKE Autopilot", "Workload Identity Federation"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol", "markdown": "https://wpnews.pro/news/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol.md", "text": "https://wpnews.pro/news/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol.txt", "jsonld": "https://wpnews.pro/news/automated-incident-triage-on-gke-with-gemini-4-argon-and-model-context-protocol.jsonld"}}