{"slug": "i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly", "title": "I gave an AI eight real DevOps jobs. Here's where it shipped — and where it quietly got it wrong.", "summary": "A developer tested Claude Code on eight real DevOps tasks, including CI log triage and on-call fixes, and found it genuinely useful for reading logs and proposing fixes, but unreliable as a deterministic gate: the same AI code review on the same pull request produced different verdicts on two runs. The developer recommends using AI reviewers as advisory tools, never as blocking status checks, and emphasizes guardrails like edit-only tools, throwaway branches, and no merge rights.", "body_md": "For the last couple of months I've been handing an AI agent (Claude Code) real DevOps work\n\n— not toy prompts, but actual failing pipelines, real pull requests, a live Kubernetes\n\ncluster — and filming what happens. Everything ran locally at $0 (no managed API bill), on\n\nreal repos, with a human reviewing every change before it merged.\n\nThe interesting part was never \"look, AI did the thing.\" It was the seam between *where it*\n\ngenuinely shipped and *where it produced something that looked right and wasn't.* That\n\nseam is the whole job. Here's what I found.\n\n##\nThe result that made me stop trusting it as a gate\n\nI ran the same AI code review on the **same pull request, twice.** I got two different\n\nverdicts — one run approved it, the other flagged a blocking issue. Same diff, same prompt,\n\nsame model.\n\nThat's not a bug, it's the nature of the tool: it's a sampler, not a linter. And it has a\n\ndirect consequence for how you're allowed to wire it in. **An AI reviewer cannot be a**\n\nrequired status check that blocks merges — not while the same input can yield \"approve\"\n\nand \"request changes\" on two runs. It's a fantastic *second pair of eyes* that catches\n\nthings humans skim past. It is not a deterministic gate. Treat it like a very sharp junior\n\nwho's occasionally, confidently wrong.\n\nWhat I do now:\n\n- Use it as an\n**advisory** reviewer that comments, never one that blocks.\n- Keep temperature/settings pinned and prompts version-controlled so at least the\n*inputs*\nare stable.\n- Assume any single run can miss things; the value is in the aggregate signal, not the verdict.\n\n(Full run: [https://youtu.be/lox5bc7zCqM](https://youtu.be/lox5bc7zCqM))\n\n##\nWhere it genuinely earned its keep\n\n**Reading CI logs.** I pointed it at a red GitHub Actions pipeline with three real failures\n\n— a dependency conflict, a genuine off-by-one in a test, and a workflow pinned to the wrong\n\nPython. It read each failing step's log, named the root cause, and proposed the smallest fix\n\nfor each. It was legitimately faster than I am at the boring triage. The honest caveat is in\n\nthe title of that episode: *\"Almost.\"* — on one of them it fixed the symptom, and a human\n\nstill had to redirect it to the actual cause. ([https://youtu.be/pIkPsiBZQ1M](https://youtu.be/pIkPsiBZQ1M))\n\n**On-call, with hard guardrails.** The one I trust most is an on-call agent that wakes up on\n\na broken build, finds the fix, and opens a PR explaining what broke — and then *stops.* It\n\nhas no merge rights. A human reads the PR over coffee and clicks merge. The autonomy ends at\n\nthe merge button, on purpose. ([https://youtu.be/UcOt5JrQVco](https://youtu.be/UcOt5JrQVco))\n\n##\nThe guardrails that made agentic DevOps safe enough to film\n\nEvery \"the AI did DevOps and it was fine\" story is really a story about constraints. The set\n\nthat worked for me:\n\n-\n**Edit-only tools.** No shell, no network, no installs. It can read and edit files and open\nPRs — that's it.\n-\n**Throwaway branches only.** It never commits to `main`\n\n.\n-\n**PR-only, no merge rights.** A human reviews every change. This is the load-bearing one.\n-\n**One attempt per failure.** No token-burning retry loops that \"eventually\" pass by luck.\n-\n**Tests are the spec.** It never silences or weakens a failing test to go green.\n\nGive an agent hands but not the keys, and most of the scary failure modes just… can't happen.\n\n##\nScaling review without paying for it\n\nThree episodes are about making review cheap and constant:\n\n-\n**Instant review on every PR**, free, via a GitHub Actions workflow — the review lands\nseconds after the PR opens. ([https://youtu.be/8xEto5pcoJY](https://youtu.be/8xEto5pcoJY))\n-\n**Ephemeral self-hosted runners on Kubernetes** (Actions Runner Controller): a fresh pod\nper PR that runs the review and self-destructs. $0, no idle runners.\n([https://youtu.be/PA6NoR7Vc6A](https://youtu.be/PA6NoR7Vc6A))\n-\n**A reviewer with memory** — a git-tracked file of your team's conventions that it learns\nand applies, so it stops re-flagging things you've already decided.\n([https://youtu.be/-9g8sSO0fL0](https://youtu.be/-9g8sSO0fL0))\n\n##\nThe takeaways, if you skipped to the bottom\n\n-\n**AI review is advisory, not a gate.** Same PR can get opposite verdicts; never make it a\nrequired check.\n-\n**Humans keep the merge button.** Agents get hands (read/edit/PR), never the keys (merge/deploy).\n-\n**Constraints are the product.** Edit-only, throwaway branches, one attempt, tests-as-spec.\n-\n**Verify every change.** \"Valid YAML\" and \"the tests pass\" are not the same as \"correct.\"\n-\n**It's genuinely useful for the boring middle** — log triage, first-pass review, drafting\nfixes — which is most of the toil.\n\nI'm filming the whole series (honestly — including the episodes where it gets things wrong)\n\nat [DevOps Autopilot](https://www.youtube.com/@DevOpsAutopilot). Happy to answer anything\n\nin the comments — especially if you've wired an AI reviewer into CI and have a different take\n\non the determinism problem.", "url": "https://wpnews.pro/news/i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly", "canonical_source": "https://dev.to/sambit_kumarmallick_bce8/i-gave-an-ai-eight-real-devops-jobs-heres-where-it-shipped-and-where-it-quietly-got-it-wrong-4m2i", "published_at": "2026-08-31 10:35:51+00:00", "updated_at": "2026-08-31 10:52:08.834811+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops", "ai-products"], "entities": ["Claude Code", "GitHub Actions", "Kubernetes", "Actions Runner Controller"], "alternates": {"html": "https://wpnews.pro/news/i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly", "markdown": "https://wpnews.pro/news/i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly.md", "text": "https://wpnews.pro/news/i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly.txt", "jsonld": "https://wpnews.pro/news/i-gave-an-ai-eight-real-devops-jobs-here-s-where-it-shipped-and-where-it-quietly.jsonld"}}