{"slug": "what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code", "title": "What 1,135 agent-written pull requests taught me about reviewing AI code", "summary": "An engineer who ran an autonomous software team with 26 agent roles inside a GitHub repository for five months reports that the system merged 1,135 pull requests. The engineer found that AI code review by the same model type is weaker on subtle defects, and that dry-run outputs can mislead reviewers, among other lessons learned.", "body_md": "For the last five months I've run an autonomous software team inside a GitHub\n\nrepo. 26 agent roles: a Discussion becomes a spec, a spec becomes a pull\n\nrequest, and nothing merges until code review, security review and acceptance\n\nall pass. It has merged 1,135 PRs.\n\nThis isn't a post about the tool. It's about the five things that turned out to\n\nbe true, most of which I got wrong first. They apply to anyone putting agents\n\nanywhere near a codebase, whether you use my thing or not.\n\nA model reviewing another model's code catches slips, conventions, and missing\n\ncases that were visible on the surface. It is much weaker on the defects that\n\nactually hurt — a race, an off-by-one under load, a guard on the wrong side of\n\na branch. Those read fine. They read fine to the author *and* to the reviewer,\n\nfor the same reasons.\n\nIf your pipeline is `generate → review → merge`\n\nand both steps are the same\n\nkind of mind, you have one opinion sampled twice, not two opinions.\n\nThis is the one that changed how I think.\n\nA cleanup script's `--dry-run`\n\nreported 116 removals. The real run, on an\n\nidentical population seconds later, removed zero. 190 directories before, 190\n\nafter.\n\nThe skip that protects git-tracked worktrees is gated on the run *not* being a\n\ndry run:\n\n```\nif [[ \"$dry_run\" == false && \"$enable_git_tracked_removal\" == false ]]; then\n  skip_git_tracked\nfi\n# a dry run never evaluates this, falls through, and reports \"would remove\"\n```\n\nNobody lost data — the divergence runs the safe way. What broke was subtler.\n\n**Two code reviews had already cited that dry-run output as their\nverification.** A reviewer asked \"did you check this against the live\n\nThe reviews weren't lazy. The evidence was lying.\n\nThe takeaway isn't \"distrust reviewers.\" It's that a verdict is only as good as\n\nthe artifact it cites, and almost nobody records *which mode produced the\nartifact*.\n\n`dry-run`\n\noutput and real output should not be the same shape ofAn agent role in this system is a Markdown behaviour spec plus a JSON policy\n\nrecord — timeouts, retries, token ceilings, concurrency caps. Adding a role is\n\nwriting a file. Changing what a reviewer cares about is editing prose.\n\nThe moment roles became data, three things got easier: diffing a behaviour\n\nchange, capping concurrency per role instead of globally, and testing\n\norchestration without spawning anything.\n\nEvery agent ends its final message with a structured envelope the orchestrator\n\nparses for routing. Not \"read the last paragraph and infer whether it passed.\"\n\nThis single decision removed most of the flakiness in the system. If you are\n\nregexing model prose to decide what happens next, that is where your\n\nintermittent failures live.\n\nRelated: add an `unknowns[]`\n\nfield. An agent that can say \"I could not tell\n\nwhether this path is reachable\" gets routed to a research step for a few\n\nthousand tokens, instead of producing a confident PR that burns a full review\n\ncycle discovering the same thing.\n\nProvisioning halts after its first phase. It installs labels, hooks, state and\n\ndependencies — then stops and asks for a backlog.\n\nI fought this for a while because a halt feels like an unfinished feature. It\n\nisn't. The next step needs to know what you actually want built and what's\n\nexplicitly out of scope, and a system that guessed would produce a hundred\n\nconfident pull requests solving the wrong problem. Fast, thorough, and wrong is\n\nworse than stopping.\n\nIf you're building agent tooling, find the place where yours is guessing at\n\nintent and make it stop there instead.\n\nMost of those 1,135 PRs are the system's own tooling. Spawn queues, review\n\ngates, the circuit breaker, the sandbox guard. For a long stretch its only\n\nproduct was itself.\n\nI've come round to thinking that's the interesting part rather than the\n\nembarrassing one — a team that can't improve its own process needs a human for\n\nevery improvement, forever — but you should decide that for yourself, which is\n\nwhy the queue is public.\n\nEverything above is from a real system you can inspect without installing\n\nanything:\n\nIt's AGPL-3.0 and self-hosted — your machine, your repo, your key, no service\n\nin the middle. Install is three lines in Claude Code:\n\n```\n/plugin marketplace add fulcrumaxe/fulcrumaxe\n/plugin install fulcrumaxe@fulcrumaxe\n/coldstart --path /path/to/your/repo --name your-project\n```\n\nRepo: [https://github.com/fulcrumaxe/fulcrumaxe](https://github.com/fulcrumaxe/fulcrumaxe)\n\nHappy to answer anything in the comments — including what it costs to run,\n\nwhich is the question I'd ask first.", "url": "https://wpnews.pro/news/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code", "canonical_source": "https://dev.to/john_problems_/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code-593j", "published_at": "2026-09-04 08:25:20+00:00", "updated_at": "2026-09-04 08:54:04.891164+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "mlops", "ai-research"], "entities": ["GitHub"], "alternates": {"html": "https://wpnews.pro/news/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code", "markdown": "https://wpnews.pro/news/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code.md", "text": "https://wpnews.pro/news/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code.txt", "jsonld": "https://wpnews.pro/news/what-1135-agent-written-pull-requests-taught-me-about-reviewing-ai-code.jsonld"}}