# I gave an AI eight real DevOps jobs. Here's where it shipped — and where it quietly got it wrong. > Source: > Published: 2026-08-31 10:35:51+00:00 For the last couple of months I've been handing an AI agent (Claude Code) real DevOps work — not toy prompts, but actual failing pipelines, real pull requests, a live Kubernetes cluster — and filming what happens. Everything ran locally at $0 (no managed API bill), on real repos, with a human reviewing every change before it merged. The interesting part was never "look, AI did the thing." It was the seam between *where it* genuinely shipped and *where it produced something that looked right and wasn't.* That seam is the whole job. Here's what I found. ## The result that made me stop trusting it as a gate I ran the same AI code review on the **same pull request, twice.** I got two different verdicts — one run approved it, the other flagged a blocking issue. Same diff, same prompt, same model. That's not a bug, it's the nature of the tool: it's a sampler, not a linter. And it has a direct consequence for how you're allowed to wire it in. **An AI reviewer cannot be a** required status check that blocks merges — not while the same input can yield "approve" and "request changes" on two runs. It's a fantastic *second pair of eyes* that catches things humans skim past. It is not a deterministic gate. Treat it like a very sharp junior who's occasionally, confidently wrong. What I do now: - Use it as an **advisory** reviewer that comments, never one that blocks. - Keep temperature/settings pinned and prompts version-controlled so at least the *inputs* are stable. - Assume any single run can miss things; the value is in the aggregate signal, not the verdict. (Full run: [https://youtu.be/lox5bc7zCqM](https://youtu.be/lox5bc7zCqM)) ## Where it genuinely earned its keep **Reading CI logs.** I pointed it at a red GitHub Actions pipeline with three real failures — a dependency conflict, a genuine off-by-one in a test, and a workflow pinned to the wrong Python. It read each failing step's log, named the root cause, and proposed the smallest fix for each. It was legitimately faster than I am at the boring triage. The honest caveat is in the title of that episode: *"Almost."* — on one of them it fixed the symptom, and a human still had to redirect it to the actual cause. ([https://youtu.be/pIkPsiBZQ1M](https://youtu.be/pIkPsiBZQ1M)) **On-call, with hard guardrails.** The one I trust most is an on-call agent that wakes up on a broken build, finds the fix, and opens a PR explaining what broke — and then *stops.* It has no merge rights. A human reads the PR over coffee and clicks merge. The autonomy ends at the merge button, on purpose. ([https://youtu.be/UcOt5JrQVco](https://youtu.be/UcOt5JrQVco)) ## The guardrails that made agentic DevOps safe enough to film Every "the AI did DevOps and it was fine" story is really a story about constraints. The set that worked for me: - **Edit-only tools.** No shell, no network, no installs. It can read and edit files and open PRs — that's it. - **Throwaway branches only.** It never commits to `main` . - **PR-only, no merge rights.** A human reviews every change. This is the load-bearing one. - **One attempt per failure.** No token-burning retry loops that "eventually" pass by luck. - **Tests are the spec.** It never silences or weakens a failing test to go green. Give an agent hands but not the keys, and most of the scary failure modes just… can't happen. ## Scaling review without paying for it Three episodes are about making review cheap and constant: - **Instant review on every PR**, free, via a GitHub Actions workflow — the review lands seconds after the PR opens. ([https://youtu.be/8xEto5pcoJY](https://youtu.be/8xEto5pcoJY)) - **Ephemeral self-hosted runners on Kubernetes** (Actions Runner Controller): a fresh pod per PR that runs the review and self-destructs. $0, no idle runners. ([https://youtu.be/PA6NoR7Vc6A](https://youtu.be/PA6NoR7Vc6A)) - **A reviewer with memory** — a git-tracked file of your team's conventions that it learns and applies, so it stops re-flagging things you've already decided. ([https://youtu.be/-9g8sSO0fL0](https://youtu.be/-9g8sSO0fL0)) ## The takeaways, if you skipped to the bottom - **AI review is advisory, not a gate.** Same PR can get opposite verdicts; never make it a required check. - **Humans keep the merge button.** Agents get hands (read/edit/PR), never the keys (merge/deploy). - **Constraints are the product.** Edit-only, throwaway branches, one attempt, tests-as-spec. - **Verify every change.** "Valid YAML" and "the tests pass" are not the same as "correct." - **It's genuinely useful for the boring middle** — log triage, first-pass review, drafting fixes — which is most of the toil. I'm filming the whole series (honestly — including the episodes where it gets things wrong) at [DevOps Autopilot](https://www.youtube.com/@DevOpsAutopilot). Happy to answer anything in the comments — especially if you've wired an AI reviewer into CI and have a different take on the determinism problem.