I gave an AI eight real DevOps jobs. Here's where it shipped — and where it quietly got it wrong.

A developer tested Claude Code on eight real DevOps tasks, including CI log triage and on-call fixes, and found it genuinely useful for reading logs and proposing fixes, but unreliable as a deterministic gate: the same AI code review on the same pull request produced different verdicts on two runs. The developer recommends using AI reviewers as advisory tools, never as blocking status checks, and emphasizes guardrails like edit-only tools, throwaway branches, and no merge rights.

For the last couple of months I've been handing an AI agent Claude Code real DevOps work — not toy prompts, but actual failing pipelines, real pull requests, a live Kubernetes cluster — and filming what happens. Everything ran locally at $0 no managed API bill , on real repos, with a human reviewing every change before it merged. The interesting part was never "look, AI did the thing." It was the seam between where it genuinely shipped and where it produced something that looked right and wasn't. That seam is the whole job. Here's what I found. The result that made me stop trusting it as a gate I ran the same AI code review on the same pull request, twice. I got two different verdicts — one run approved it, the other flagged a blocking issue. Same diff, same prompt, same model. That's not a bug, it's the nature of the tool: it's a sampler, not a linter. And it has a direct consequence for how you're allowed to wire it in. An AI reviewer cannot be a required status check that blocks merges — not while the same input can yield "approve" and "request changes" on two runs. It's a fantastic second pair of eyes that catches things humans skim past. It is not a deterministic gate. Treat it like a very sharp junior who's occasionally, confidently wrong. What I do now: - Use it as an advisory reviewer that comments, never one that blocks. - Keep temperature/settings pinned and prompts version-controlled so at least the inputs are stable. - Assume any single run can miss things; the value is in the aggregate signal, not the verdict. Full run: https://youtu.be/lox5bc7zCqM https://youtu.be/lox5bc7zCqM Where it genuinely earned its keep Reading CI logs. I pointed it at a red GitHub Actions pipeline with three real failures — a dependency conflict, a genuine off-by-one in a test, and a workflow pinned to the wrong Python. It read each failing step's log, named the root cause, and proposed the smallest fix for each. It was legitimately faster than I am at the boring triage. The honest caveat is in the title of that episode: "Almost." — on one of them it fixed the symptom, and a human still had to redirect it to the actual cause. https://youtu.be/pIkPsiBZQ1M https://youtu.be/pIkPsiBZQ1M On-call, with hard guardrails. The one I trust most is an on-call agent that wakes up on a broken build, finds the fix, and opens a PR explaining what broke — and then stops. It has no merge rights. A human reads the PR over coffee and clicks merge. The autonomy ends at the merge button, on purpose. https://youtu.be/UcOt5JrQVco https://youtu.be/UcOt5JrQVco The guardrails that made agentic DevOps safe enough to film Every "the AI did DevOps and it was fine" story is really a story about constraints. The set that worked for me: - Edit-only tools. No shell, no network, no installs. It can read and edit files and open PRs — that's it. - Throwaway branches only. It never commits to main . - PR-only, no merge rights. A human reviews every change. This is the load-bearing one. - One attempt per failure. No token-burning retry loops that "eventually" pass by luck. - Tests are the spec. It never silences or weakens a failing test to go green. Give an agent hands but not the keys, and most of the scary failure modes just… can't happen. Scaling review without paying for it Three episodes are about making review cheap and constant: - Instant review on every PR , free, via a GitHub Actions workflow — the review lands seconds after the PR opens. https://youtu.be/8xEto5pcoJY https://youtu.be/8xEto5pcoJY - Ephemeral self-hosted runners on Kubernetes Actions Runner Controller : a fresh pod per PR that runs the review and self-destructs. $0, no idle runners. https://youtu.be/PA6NoR7Vc6A https://youtu.be/PA6NoR7Vc6A - A reviewer with memory — a git-tracked file of your team's conventions that it learns and applies, so it stops re-flagging things you've already decided. https://youtu.be/-9g8sSO0fL0 https://youtu.be/-9g8sSO0fL0 The takeaways, if you skipped to the bottom - AI review is advisory, not a gate. Same PR can get opposite verdicts; never make it a required check. - Humans keep the merge button. Agents get hands read/edit/PR , never the keys merge/deploy . - Constraints are the product. Edit-only, throwaway branches, one attempt, tests-as-spec. - Verify every change. "Valid YAML" and "the tests pass" are not the same as "correct." - It's genuinely useful for the boring middle — log triage, first-pass review, drafting fixes — which is most of the toil. I'm filming the whole series honestly — including the episodes where it gets things wrong at DevOps Autopilot https://www.youtube.com/@DevOpsAutopilot . Happy to answer anything in the comments — especially if you've wired an AI reviewer into CI and have a different take on the determinism problem.