# Moving fast with AI, without losing control

> Source: <https://dash0.com/blog/moving-fast-with-ai-without-losing-control>
> Published: 2026-11-17 07:49:31+00:00

Nobody knows what an engineering organization looks like once AI writes most of the code. We don't either, not yet at least. But our customers need a product that holds up in that future, so we had to build for it anyway. Over the past year we went all in on AI in our own engineering. At the same time, we grew the engineering team by 58%.

This post is about what broke along the way, and about the experiment that changed how we ship. We let pull requests approve themselves.

Spoiler alert: this story has a happy ending.

"Nine women can't make a baby in a month" is the classic argument that adding people to a team slows it down. Turns out, sometimes they can. But in between, things broke.

## [Everyone got faster, and more worried](#everyone-got-faster-and-more-worried)

On 24 November 2025 Claude Opus 4.5 came out, and coding agents went from experiment to everyday work. The promise was real:

Nothing before has hit with the magnitude of AI.Martin Fowler, Chief Scientist, Thoughtworks

Faros tracked 22,000 developers, and in their highest-AI quarters each developer completed 66% more epics and 34% more tasks ([Faros AI Engineering Report 2026](https://pages.faros.ai/hubfs/AI_Engineering_Report_2026_The_Acceleration_Whiplash_Faros.pdf)).

But then we started seeing the secondary effects:

Software has become a brittle mess, with 98% uptime becoming the norm.Mario Zechner, Creator of Pi

The same Faros report showed the other side of the AI coding coin. Median time in PR review went up 441%. And 72% of organizations have already had at least one production incident caused by AI-generated code ([Harness, State of AI in Software Engineering](https://www.harness.io/the-state-of-ai-in-software-engineering)).

**We were one of those organizations.**

Not a great place to be when you sell observability and customers trust you with the reliability of their software.

## [Four things broke](#four-things-broke)

Code review became the bottleneck. Quality got worse before it got better. Everyone built fast, and the product started to look like a patchwork. And more agents meant a bigger bill.

This post is mostly about reviews, with a look at quality. Coherence and cost deserve their own posts.

## [Then came loop engineering, and reviews exploded](#then-came-loop-engineering-and-reviews-exploded)

Agents write a pull request in minutes. Then every pull request waits in the same narrow queue for a human review. In a typical week, 56% of the pull requests we opened were still waiting for a review. The slowest 25% waited almost a day for a first look.

Code throughput went up, and so did cycle time. Engineers spent hours a day reviewing, and at some point they started rubber-stamping. People were frustrated, and they had a point: a review nobody reads doesn't catch anything.

## [What if we let go of reviews?](#what-if-we-let-go-of-reviews)

A human review for every change had been a rule for as long as any of us had worked in software. Questioning it felt like a lot. What if we stopped?

We didn't decide that up front. We wanted to find out what happens when pull requests can approve themselves, so we built [AutoMerge](https://dash0.com/darkplane-automerge) and ran it as an experiment. Every change to the policy started as a hypothesis, and we logged the result and dropped what didn't work. Humans were in control of the approval policy the whole time, auditing every change.

## [How a pull request gets approved](#how-a-pull-request-gets-approved)

Every pull request goes through the same policy, whether a human or an agent opened it. On our monorepo the policy scores four criteria: author track record, blast radius, review evidence, and change scope. Some are deterministic, like authorship and reverts. An AI evaluator judges the others, like blast radius and review evidence. AutoMerge weights the scores into one number and compares it with a threshold of 75, which we can tune too. If it scores above the threshold, the pull request is approved to merge. Below it, a human decides. A criterion can also have a floor, and a score below its floor sends the pull request to a human whatever the total. Protected paths always go to a human too.

Every verdict comes with a written reason, posted as a comment on the pull request. This one is real. [Agent0](https://dash0.com/agent0), our AI SRE, had found that our synthetic check emails still said FAILING, with a red bar, after the issue was already closed. It opened a small fix, and AutoMerge did not approve it. The combined score was 73.3, close to the threshold, but the total didn't decide it. Agent0 had almost no history in that part of the code, so its author track record fell below the floor. An engineer approved the fix instead, and it merged 26 minutes after Agent0 opened it.

Most of our experimenting happens in the criteria, because they are the part you craft and control. What counts as a safe change depends on your codebase, so you decide which criteria to score, how much each one weighs, and where the floors sit. For us, engineers gave feedback on every verdict, and we tuned the criteria until we trusted them.

## [Dry run first, autonomy later](#dry-run-first-autonomy-later)

Nothing touched a real merge before we had watched it in dry run, where AutoMerge approved nothing and only recorded what it would have done. For weeks on our monorepo we compared those verdicts with what humans did. Auto-approve became the default only in mid-September.

- Evaluated PRs still in dry run
- Merged PRs that were auto-approved

| Period | Evaluated PRs still in dry run | Merged PRs that were auto-approved | 
|---|---|---|
| Aug 31 |  |  | 
| Sep 7 |  |  | 
| Sep 14 |  |  | 
| Sep 21 |  |  | 
| Sep 28 |  |  | 
| Oct 5 |  |  | 

Two fences stayed in place. Protected paths are never auto-approved, whatever the score. And some pull requests that are auto-approved still get a human review, which is why the share of merges that were auto-approved stays below the share that passed the policy.

There was also a bonus side effect: pull requests got smaller, because engineers like a fast merge.

## [Under 1% wrongly approved, but still too cautious](#under-1percent-wrongly-approved-but-still-too-cautious)

Across all pull requests, we checked every verdict against what happened next. Fewer than 1% were incorrect approvals. 61% were unnecessary holds, where the policy sent a pull request to a human that could have merged on its own. That second number is the one we are tuning now. A policy this cautious still leaves a lot of review work on the table.

## [It doesn't always work](#it-doesnt-always-work)

Engineers let us know when the policy gets it wrong. Peter Froehlich, one of our principal platform engineers, posted this in Slack:

The score was 69.98 and the threshold was 70. The maths is right, and it still feels unfair. We'll take that kind of complaint any day, because the engineer could see exactly which criterion cost them the merge, and so could we. When a human reviewer waves a change through, there is no score to argue about.

It isn't only the policy that falls short. Costs went up across the board, and with review no longer the bottleneck, the bottleneck is moving down the line to CI, so developer experience is suddenly a topic again. Those deserve their own posts.

## [Our AI SRE opens pull requests too](#our-ai-sre-opens-pull-requests-too)

You met Agent0 earlier, when AutoMerge held back its email fix. That was one pull request of many. Between August and October, **two thirds of the pull requests it opened on our monorepo were merged**, most of them bug fixes. In one example, a rate-limited API returned a generic 500. Agent0 changed it to a proper 429 with a `Retry-After` header, and the fix was **merged 8 minutes after the pull request was opened**.

Agent0 doesn't resolve incidents on its own yet. It does the digging, and a human approves the fix. When the pager goes off at 5am, a full root cause analysis is already waiting, so the engineer on call starts from an answer instead of a blank dashboard.

## [More than twice the merges, faster than before](#more-than-twice-the-merges-faster-than-before)

We measured six weeks on the monorepo, from the end of August to mid-October. Merges per day went up 133%. The time the slowest 25% of pull requests waited for a first review went down 60%, and the median wait went down 55%. The queue of open pull requests stayed flat, while the number of pull requests opened per week grew 117%.

The median time to merge went down 49%. Reviewers now spend their time on the pull requests the policy holds back, instead of on every one.

## [What about quality and reliability?](#what-about-quality-and-reliability)

Merging twice as much is only good news if what you merge holds up in production. It held, and it improved. Bugs opened per week went down 56% between the first and the sixth week. Per merged pull request, bugs went down 81%. Incidents stayed stable.

### [Quality broke first, so we put an agent on it](#quality-broke-first-so-we-put-an-agent-on-it)

The drop in bugs has a back story. Quality was the first thing to break. In Q2, open issues from Customer Solutions grew fast. Agent0 connects to Linear and Slack, so we built a bot on it that assigns bugs to teams and nudges engineers every morning: what to pick up, and what is close to its SLA. The backlog of active bugs went down 45% from its peak.

It worked so well that the bottleneck moved again, this time to our customer solutions department. A ticket only closes when the customer confirms the fix, and that last step was human, so it waited. Engineering and Customer Solutions sat down together and closed what was already fixed. Then we changed the process, so fixed tickets now go back to the solutions architect who opened them to follow up with the customer.

## [Humans own the what and the why](#humans-own-the-what-and-the-why)

So where do humans still decide? This is where we have drawn the line for now:

The line moves both ways. While autonomy grew, the share of protected pull requests also went up, from 8% to 18%. The more we learnt about where the risk sits, the more paths we protected.

## [Decide where things are allowed to break](#decide-where-things-are-allowed-to-break)

When AI writes more of your code, humans read less of it. Correctness then depends on what the code does at runtime, not on what a reviewer saw in the diff, so we lean on [OpenTelemetry](https://opentelemetry.io/): one open standard that both humans and agents can query.

Things will break more often, so we decided where we can afford it. Where we can roll back in minutes, a broken change costs little, and we put our effort into recovery time instead of trying to prevent every failure. Where we can't afford a break, the path is protected and a human decides. In both cases we count on runtime signals to tell us something is wrong before a customer does.

## [Nobody knows the end state](#nobody-knows-the-end-state)

Looking back, most of the work was unlearning. Not long ago we would have told you that adding people slows a team down. Our headcount went up 58% and the features we shipped went up 83%. We would also have told you that every change needs a human review. Today 46% of our merges skip one, and we haven't seen a significant increase in production incidents. Even the goal changed. We used to aim for fewer incidents, and now we care more about how fast we recover from the ones we get.

We don't know what an AI-driven engineering organization looks like when it's done, and we doubt anyone does. We got this far one experiment at a time, with a dry run before every step and the policy and the telemetry in our own hands.

If you want to run the same experiment on your own repositories, AutoMerge starts in dry run, where it only comments on pull requests and changes nothing. [Read how AutoMerge works](https://dash0.com/docs/dash0/darkplane/automerge).
