The most dreaded, most mechanical part of on-call is the first hour of root-cause analysis. Turns out that is exactly the kind of work a team of AI agents can own today. So we built one, pointed it at a Kubernetes cluster, and watched it argue with itself.
Something breaks. A dashboard turns red. Someone's evening ends.
And the first hour is always the same choreography. Which service. What changed. What do the logs say. Is this new, or the thing from Tuesday that we swore we fixed. Roll back or fix forward.
None of it is clever. It is a checklist a tired human runs under pressure, the same checklist, in the same order, on every incident, forever. Meanwhile the machine that could run it is sitting right there, idle, like a very expensive paperweight with a monthly subscription.
That first hour is not creative work. It is mechanical work done under stress. Which is the single best description of "hand this to a computer" ever written.
Here is where most "AI for ops" demos faceplant. They build a chatbot. You ask it what is wrong. It writes you a gorgeous, well-structured paragraph.
It is also, frequently, confidently, eloquently making the whole thing up, because it never actually looked at anything.
Root-cause analysis is not a question. It is an investigation. Gather evidence, correlate the deploy, form a hypothesis, doubt yourself, conclude. So we did not write one big prompt and cross our fingers. We built a team. An AI SRE, if you want the term the industry is settling on. Not one that does everything a human SRE does. Just the part everyone dreads, the first hour.
Four specialists, each investigating on its own with its own read-only tools. One looks at the code, one at the recent releases, one at the infrastructure, one at the metrics. A synthesizer merges them and referees the disagreements. And a skeptic, who gets only the conclusion and is told, essentially, "prove them wrong."
Confidence is not the model saying "95 percent sure" with the same cheerful energy whether it is right or hallucinating. Confidence is earned. How many independent specialists landed on the same cause, and whether the skeptic failed to break it.
An alert fires. A dispatcher spins up one ephemeral Kubernetes Job per incident, carrying the team. It investigates read-only, posts a confidence-scored verdict, and if there is a real code fix, opens a pull request.
The whole control plane is tiny and boring on purpose. No framework, no vector database, no twelve-service diagram we would be too embarrassed to show you. The investigator can only read. It physically cannot touch anything. The one thing in the entire system that writes is the fix agent, and the worst it can do is propose a change for a human to approve.
We gave a service a latent bug and flipped it on. The error rate climbs, the alert fires.
The dispatcher spins up an agent Job. This part is weirdly satisfying. You watch the investigator appear, do its thing, and vanish, like a contractor who actually leaves when the job is done.
A couple of minutes later, a verdict. Not a wall of logs. A diagnosis, a confidence meter, and the exact fix.
Here is what happens inside that ephemeral Job.
On our incident, the team split. Two agents said the problem was in the code. Two said it was a config change. Nobody was going to agree on anything, and the incident just sat there, unbothered, like it had all day. Which, being an incident, it did.
code: found the exact defect, an unguarded lookup that crashes on a missing record. Confident.
release: the last rollout flipped a setting. That is the change that lit the fire.
infra: the pod is perfectly healthy. No crashes, no restarts, no memory pressure. This is not us.
metrics: roughly a third of requests failing, fast and deterministic, not a slow timeout.
skeptic: tried to refute it. Could not. The same setup ran perfectly healthy yesterday. Confirmed.
A single model would have picked one, said it with its whole chest, and been wrong forty percent of the time with excellent grammar. Here the disagreement is the signal. Both sides are right. There is a genuine code defect, and a config change is what set it off. The skeptic could not knock it down, two specialists agreed, so the verdict came back HIGH confidence. Earned, not vibes.
The machine was technically right. The trigger was a config change. The most annoying kind of right.
But there was a real, fixable bug sitting right there in the open. So we made a call. If a concrete fix exists, propose it, no matter what label we put on the incident. The diagnosis and the fix are two different questions, and we had been letting one veto the other.
So it did. It worked out the smallest safe change, wrote it up, and opened a pull request for a human to review. A real one. That merges. I refreshed the page twice to make sure it was not going to take it back.
Two things bit us. Both were educational in the way stepping on a rake is educational.
It spammed the channel. The first version had opinions and it shared them. Every few minutes. In the channel. Forever. The culprit was a subtle one. A service spraying errors has a perfectly healthy pod. Nothing is crashing. It is serving failures with a smile, vitals green, thumbs up. So our "all clear" signal kept firing, which reset everything and re-ran the whole pipeline on a loop. The fix was to treat an incident as resolved only when the alert actually clears, not when the pod is feeling optimistic. One incident, one notification.
Then it posted the whole story, root cause, recommended fix, and the proposed change, into one clean card so the humans could argue about it in the right venue.
The reason an AI SRE like this is more than a party trick comes down to a few boring, load-bearing properties.
Cost scales with incidents, not services. The Job exists only while it is investigating and then it is gone. Ten thousand healthy services cost you exactly nothing.
It is read-only by default. The investigator cannot change anything. The only thing it ever produces is a suggestion a human signs off on.
And it ports. Swap the local monitoring for whatever your cloud already runs and the brain is identical. The trigger and the evidence change. The team does not.
There is one question that is genuinely, humblingly hard. Is this the same incident as last time, or a brand new problem that happens to look identical? Right now we match on the symptom, and two completely different bugs can throw the exact same alert.
Knowing it is really the same issue needs a fingerprint of the diagnosed cause, which only exists after you investigate. So the next build is memory. A record of past incidents, so a repeat short-circuits straight to "seen this, here is what worked, you are welcome." After that, correlating one cause across many alerts, and a learning loop where a fix that gets accepted raises confidence and one that gets rejected lowers it.
It was everything after. The judgment. The trade-offs. The "do we roll back or fix forward while the CEO watches the dashboard."
So give the machine the toil. The evidence gathering, the deploy correlation, the first honest draft of a hypothesis with a confidence you can actually trust. Keep the judgment for yourself. That was always the good part anyway.
If you are on-call tonight, here is my question. What would you actually want an agent to hand you when the pager screams? The evidence, the hypothesis, or the fix ready to review? Tell me in the comments. Especially if the answer is "a different job."