What 1,135 agent-written pull requests taught me about reviewing AI code

An engineer who ran an autonomous software team with 26 agent roles inside a GitHub repository for five months reports that the system merged 1,135 pull requests. The engineer found that AI code review by the same model type is weaker on subtle defects, and that dry-run outputs can mislead reviewers, among other lessons learned.

For the last five months I've run an autonomous software team inside a GitHub repo. 26 agent roles: a Discussion becomes a spec, a spec becomes a pull request, and nothing merges until code review, security review and acceptance all pass. It has merged 1,135 PRs. This isn't a post about the tool. It's about the five things that turned out to be true, most of which I got wrong first. They apply to anyone putting agents anywhere near a codebase, whether you use my thing or not. A model reviewing another model's code catches slips, conventions, and missing cases that were visible on the surface. It is much weaker on the defects that actually hurt — a race, an off-by-one under load, a guard on the wrong side of a branch. Those read fine. They read fine to the author and to the reviewer, for the same reasons. If your pipeline is generate → review → merge and both steps are the same kind of mind, you have one opinion sampled twice, not two opinions. This is the one that changed how I think. A cleanup script's --dry-run reported 116 removals. The real run, on an identical population seconds later, removed zero. 190 directories before, 190 after. The skip that protects git-tracked worktrees is gated on the run not being a dry run: if "$dry run" == false && "$enable git tracked removal" == false ; then skip git tracked fi a dry run never evaluates this, falls through, and reports "would remove" Nobody lost data — the divergence runs the safe way. What broke was subtler. Two code reviews had already cited that dry-run output as their verification. A reviewer asked "did you check this against the live The reviews weren't lazy. The evidence was lying. The takeaway isn't "distrust reviewers." It's that a verdict is only as good as the artifact it cites, and almost nobody records which mode produced the artifact . dry-run output and real output should not be the same shape ofAn agent role in this system is a Markdown behaviour spec plus a JSON policy record — timeouts, retries, token ceilings, concurrency caps. Adding a role is writing a file. Changing what a reviewer cares about is editing prose. The moment roles became data, three things got easier: diffing a behaviour change, capping concurrency per role instead of globally, and testing orchestration without spawning anything. Every agent ends its final message with a structured envelope the orchestrator parses for routing. Not "read the last paragraph and infer whether it passed." This single decision removed most of the flakiness in the system. If you are regexing model prose to decide what happens next, that is where your intermittent failures live. Related: add an unknowns field. An agent that can say "I could not tell whether this path is reachable" gets routed to a research step for a few thousand tokens, instead of producing a confident PR that burns a full review cycle discovering the same thing. Provisioning halts after its first phase. It installs labels, hooks, state and dependencies — then stops and asks for a backlog. I fought this for a while because a halt feels like an unfinished feature. It isn't. The next step needs to know what you actually want built and what's explicitly out of scope, and a system that guessed would produce a hundred confident pull requests solving the wrong problem. Fast, thorough, and wrong is worse than stopping. If you're building agent tooling, find the place where yours is guessing at intent and make it stop there instead. Most of those 1,135 PRs are the system's own tooling. Spawn queues, review gates, the circuit breaker, the sandbox guard. For a long stretch its only product was itself. I've come round to thinking that's the interesting part rather than the embarrassing one — a team that can't improve its own process needs a human for every improvement, forever — but you should decide that for yourself, which is why the queue is public. Everything above is from a real system you can inspect without installing anything: It's AGPL-3.0 and self-hosted — your machine, your repo, your key, no service in the middle. Install is three lines in Claude Code: /plugin marketplace add fulcrumaxe/fulcrumaxe /plugin install fulcrumaxe@fulcrumaxe /coldstart --path /path/to/your/repo --name your-project Repo: https://github.com/fulcrumaxe/fulcrumaxe https://github.com/fulcrumaxe/fulcrumaxe Happy to answer anything in the comments — including what it costs to run, which is the question I'd ask first.