{"slug": "incident-response-is-becoming-an-agent-review-workflow", "title": "incident response is becoming an agent review workflow", "summary": "AWS's DevOps Agent walkthrough demonstrates how incident response is evolving into a review workflow, using specific evidence from Amazon EKS control-plane issues. The agent correlates metrics, logs, and audit trails to produce a reviewable argument rather than a vague summary, shifting the engineer's role from investigation to verification.", "body_md": "The least useful sentence during an incident is \"the AI says it found the problem.\"\n\nGreat.\n\nWhat did it look at?\n\nWhich logs?\n\nWhich metrics?\n\nWhich permissions did it use?\n\nWhat did it rule out?\n\nWhat action is it recommending, and who is taking responsibility for that action at 2 a.m. when everyone is a little too confident and a little too tired?\n\nThis is why I found AWS's recent walkthrough of AWS DevOps Agent diagnosing Amazon EKS control-plane performance issues more interesting than the usual \"AI for operations\" story. The example is very specific: API server throttling, API Priority and Fairness seat exhaustion, audit logs, CloudWatch metrics, and a recommendation path for remediation.\n\nThat specificity matters.\n\nThe story is not \"AI fixes Kubernetes.\"\n\nThe story is that incident response is becoming a review workflow.\n\nKubernetes is extremely good at turning one problem into five symptoms.\n\nA deployment slows down. The API server starts throttling. Controllers fall behind. Webhooks add latency. A noisy client hammers the cluster. Someone notices pods are not scheduling quickly. Someone else sees a dashboard that is red but not very helpful. The actual cause is somewhere between application behavior, cluster configuration, cloud limits, and a control plane doing exactly what it was designed to do under pressure.\n\nThis is not a place where \"summarize the incident\" is enough.\n\nThe hard part is collecting the right evidence in the right order.\n\nWhen an engineer investigates an EKS control-plane issue, they are usually stitching together several sources:\n\nThe value of an agent is not that it has magical intuition.\n\nThe value is that it can do the boring correlation work quickly, consistently, and with enough structure that another person can inspect it.\n\nThat last part is the difference between help and risk.\n\nI do not want an operations agent to produce a vibe.\n\nI want it to produce an argument.\n\nSomething like:\n\n\"API server requests increased sharply at this time. The increase came mostly from this identity. The requests were concentrated in this verb and resource. APF metrics show this priority level exhausting seats. Audit logs confirm repeated calls from this workload. No matching control-plane health event suggests a regional service issue. The likely cause is this client behavior. Recommended actions are these, with these tradeoffs.\"\n\nThat is reviewable.\n\nIt can be wrong, but it is wrong in a way a human can challenge.\n\nThis is the shape I like in the AWS example. The DevOps Agent is not just a chatbot sitting beside a dashboard. It is using cloud-native evidence: metrics, logs, audit trails, and cluster-specific context. It can point at the thing it saw.\n\nThat changes the human role.\n\nThe engineer is no longer starting from a blank page, trying to remember which query caught the same class of issue six months ago. The engineer is reviewing a proposed incident narrative.\n\nIs the evidence complete?\n\nDid the agent overfit to the first suspicious metric?\n\nDid it miss a deploy that happened five minutes earlier?\n\nIs the remediation safe during business hours?\n\nShould we reduce client pressure, tune APF, roll back an automation, or just stop one bad actor before changing cluster-wide policy?\n\nThis is a much better use of AI than pretending the human can disappear from the loop.\n\nThere is a security angle here that is easy to miss because the demo looks operational.\n\nAn agent that diagnoses production infrastructure needs access.\n\nMaybe not write access. Hopefully not broad write access by default. But it needs enough read access to inspect logs, metrics, events, identities, configurations, and sometimes sensitive operational details. In Kubernetes, even read-only access can reveal a lot about the system.\n\nSo the agent's identity matters.\n\nWhich IAM role or access entry did it use? Which clusters can it inspect? Which namespaces? Which CloudWatch log groups? Can it see audit logs? Can it query across accounts? Can it read secrets by accident because someone treated \"debugging\" as a reason to grant everything?\n\nThis is where agentic ops becomes ordinary platform engineering again.\n\nThe agent needs least privilege. It needs audit trails. It needs clear boundaries between \"observe,\" \"recommend,\" and \"act.\" It needs different permissions for development, staging, and production. It needs a way to explain which evidence it could not access, because missing evidence is itself important during an incident.\n\nIf the agent says \"I did not find a bad client,\" but it could not read the audit log group where the bad client would be visible, that conclusion is not useful.\n\nConfidence without access transparency is theater.\n\nThere is a dangerous version of operations AI where the system jumps from diagnosis to action too quickly.\n\nThe agent sees throttling. The agent recommends changing API Priority and Fairness settings. Someone clicks approve because the recommendation sounds plausible and the dashboard is red. The incident improves for ten minutes, then a more important workload is starved because the change moved pressure somewhere else.\n\nKubernetes has many knobs that are locally reasonable and globally surprising.\n\nThis is why I care about the distinction between recommendation and ownership.\n\nAn agent can recommend a remediation. It can explain expected effects. It can show blast radius. It can link to the metrics that should improve. It can draft a change. It can even prepare a rollback plan.\n\nBut the organization still needs to decide who owns the action.\n\nDuring an incident, that ownership should be boringly explicit:\n\nThat sounds bureaucratic until you have to write the postmortem.\n\nThen it sounds like kindness.\n\nThe best thing an incident agent can do may not be fixing the issue.\n\nIt may be preserving the investigation.\n\nMost incident timelines are reconstructed after the fact from Slack messages, dashboards, terminal history, deploy logs, memory, and a few timestamps everyone argues about later. The incident has already consumed attention. Now the team has to do archaeology.\n\nAn agent can help by making the timeline part of the workflow from the beginning.\n\nIt can record what evidence was checked, which hypotheses were considered, which ones were rejected, which query showed the turning point, which remediation was chosen, and what happened afterward. Not as a polished executive summary. As operational notes that can survive stress.\n\nThat matters because incident response is full of reasonable decisions that look weird out of context.\n\nWhy did the team restart that controller?\n\nWhy did they pause that automation?\n\nWhy did they choose not to change APF during the incident?\n\nWhy did they decide the noisy client was the cause and not merely another symptom?\n\nIf the agent keeps the evidence trail, the postmortem is less about memory and more about learning.\n\nThat is a very practical form of AI assistance.\n\nI am skeptical of any operations AI story that jumps too quickly to autonomy.\n\nAutonomous remediation will happen in narrow places. Some of it already does. Restart the known bad job. Scale the obvious queue worker. Revert the canary that just failed health checks. Block a clearly abusive client. Fine.\n\nBut most production incidents are not clean enough to deserve a big red \"let the model handle it\" button.\n\nThe better near-term product is more boring:\n\nThat is still a big deal.\n\nIt means on-call engineers spend less time remembering where the evidence lives and more time deciding what the evidence means. It means new team members can learn from actual incident reasoning, not just final summaries. It means platform teams can encode common diagnostic paths without pretending every incident is identical.\n\nMost importantly, it gives reviewers something concrete.\n\nNot \"the agent was helpful.\"\n\n\"Here is the investigation it performed, here is what it concluded, here is who accepted it, and here is what changed.\"\n\nThat is the kind of sentence production systems can live with.\n\nAWS DevOps Agent diagnosing EKS control-plane performance issues is a useful signal because it shows where ops AI is probably going.\n\nNot straight to magic remediation.\n\nToward incident response as a structured, reviewable workflow.\n\nKubernetes incidents need evidence. Cloud providers already hold much of that evidence in logs, metrics, audit trails, identities, and configuration APIs. Agents can pull those pieces together faster than a tired human jumping between consoles. But the useful output is not a confident answer. It is a diagnosis that can be inspected.\n\nWhat did the agent see?\n\nWhat could it access?\n\nWhat did it miss?\n\nWhat action did it recommend?\n\nWho decided to act?\n\nIf those questions are answered, an agent becomes a second reviewer in the incident process. Not the owner. Not the hero. Not the person carrying the pager.\n\nA reviewer.\n\nThat may sound less exciting than autonomous operations.\n\nGood.\n\nProduction needs fewer exciting surprises.\n\nTo test my projects, I use [Railway](https://railway.com?referralCode=G_jRmP). If you want $20 USD to get started, [use this link](https://railway.com?referralCode=G_jRmP).", "url": "https://wpnews.pro/news/incident-response-is-becoming-an-agent-review-workflow", "canonical_source": "https://dev.to/pvgomes/incident-response-is-becoming-an-agent-review-workflow-4ni", "published_at": "2026-07-07 00:02:01+00:00", "updated_at": "2026-07-07 00:32:18.244573+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety"], "entities": ["AWS", "Amazon EKS", "AWS DevOps Agent", "Kubernetes", "CloudWatch"], "alternates": {"html": "https://wpnews.pro/news/incident-response-is-becoming-an-agent-review-workflow", "markdown": "https://wpnews.pro/news/incident-response-is-becoming-an-agent-review-workflow.md", "text": "https://wpnews.pro/news/incident-response-is-becoming-an-agent-review-workflow.txt", "jsonld": "https://wpnews.pro/news/incident-response-is-becoming-an-agent-review-workflow.jsonld"}}