{"slug": "building-a-software-factory", "title": "Building a Software Factory", "summary": "Cloud agents drove 38% of the pull requests merged in October, up from 0.1% in June, after the engineering team built an internal service that runs coding agents in the cloud and keeps each pull request until CI is green, according to the team's own account. Between September and mid-November the median wait for a first approval fell from over two hours to about five minutes and weekly merged pull requests more than doubled, while changes approved by an agent were reverted no more often than those approved by people. In a sample of 100 agent pull requests merged in the first half of November, 78 had no commits from a person, and in the core API repository 93% of the AI reviewer's comment threads were resolved or led to a code change, up from 84% in September.", "body_md": "Engineering\n\n# Building a Software Factory\n\nAI made writing code fast, but reviewing it became a bottleneck for our engineers. This is how we handed our delivery line to agents one step at a time, and what the numbers say.\n\nEvery month, agents write an increasing share of our code, and the queue of pull request reviews grows as a result. Review bots catch problems, but each finding sends the change back for another round of reviews, and in the end, the pull request still waits for a person to approve it.\n\nOver the summer, the share of human reviews in our core API repository spent on agent-written code more than tripled, to about one in five, and the number of pull requests awaiting human approval doubled.\n\nAnthropic engineer Felix Rieseberg’s “[How to be more ambitious](https://felixrieseberg.com/how-to-be-more-ambitious-a-tutorial-for-humans-with-models/)” inspired us to think ahead. Instead of asking agents for better changes, we tested if they could take over each stage of software delivery and decided how we would judge each handover before looking at its results. OpenAI [has described](https://openai.com/index/harness-engineering/) a similar shift on a product built from an empty repository.\n\nCloud agents drove 38% of the pull requests we merged in October, up from 0.1% in June. Between September and mid-November, the median wait for a first approval fell from over two hours to about five minutes and the number of pull requests we merged each week more than doubled. Changes approved by an agent were reverted no more often than those approved by people.\n\nWhere we began\n\n## Every Step Needed an Engineer\n\n1. 1\n2. 2\n3. 3\n4. 4\n5. 5\n6. 6\n7. 7\n\nAt the start of the year, an engineer picked a task, prompted an agent to write it and opened a pull request, then waited for CI and handed issues back to the agent. Then they found a colleague to review it. Once a change merged, main could break, and whoever noticed first had to find the cause and fix forward or roll it back. QA was also done by hand. Every handoff between these steps was a person’s job, so the more changes agents wrote, the more work was passed to engineers.\n\nStep 1 · Write and pass CI\n\n## Let Agents Get to Green\n\n1. 1\n2. 2\n3. 3\n4. 4\n5. 5\n6. 6\n7. 7\n\n**The problem:** An agent writing a change might save us an hour of typing, but it leaves an afternoon of cleanup fixing flaky tests, lint failures, merge conflicts and review comments.\n\n**What we built:** An internal service that runs coding agents in the cloud and stays with each pull request until it's green. You give it a task and approve the plan, and then the agent writes the change, opens the pull request, reruns CI, fixes tests, answers review comments and rebases.\n\n**How we knew it worked:** The service was handling 38% of our pull requests by its third month, and our agents also needed minimal intervention. In a sample of 100 of their pull requests merged in the first half of November, 78 had no commits from a person.\n\nWhere the bottleneck moved\n\nAgents produced review-ready pull requests faster than people could read them, so review became the next station to hand over.\n\nStep 2 · Review\n\n## Let Agents Review\n\n1. ✓\n2. ✓\n3. 3\n4. 4\n5. 5\n6. 6\n7. 7\n\n**The problem:** Our third-party review bot lacked context, so engineers learned to skim past it and still read every change themselves.\n\n**What we built:** An AI reviewer with the same context as the agents writing the code, including the plan, the diff and the code around it. Usually, the author is another agent, so a change goes back and forth between the coding agent and the AI reviewer until it’s clean.\n\n**How we knew it worked:** In our core API repository, 93% of the AI reviewer’s comment threads were resolved or led to a code change in the first half of November, up from 84% in September. A rebuild in late September cut the cost of each review to about a third, while weekly review volume increased.\n\nWhere the bottleneck moved\n\nPull requests came out of AI review ready to merge, but still waited hours for a person to click approve.\n\nStep 3 · Approve\n\n## Let Agents Approve\n\n1. ✓\n2. ✓\n3. ✓\n4. 4\n5. 5\n6. 6\n7. 7\n\n**The problem:** Letting the AI reviewer approve would clear the queue, but that meant code would merge without a person reading it. We needed proof that AI approvals matched the judgment of our engineers before taking this step.\n\n**What we built:** Risk-based approval. A separate agent rates each pull request low, medium or high risk, rounding up when unsure, and the rating decides who can approve it. At first we let AI approve low-risk changes on its own and medium-risk changes where a repository allows it. High-risk changes go to a person, and code owners still sign off on critical areas of our codebases like database migrations, secrets and authentication.\n\nWe started by having the reviewer approve alongside people and tracking how often a person disagreed. In mid-October, we let pull requests merge on AI approval alone. Across September and October, a person objected to only 15 of the 977 pull requests it approved, about 1.5%, at about the same rate before and after that date.\n\n**How we knew it worked:** The median wait for a first approval fell from over two hours in September to about five minutes in mid-November, and about seven in ten merges the AI reviews in our core API repository go through on its approval alone. Of all pull requests merged on AI approval alone from October to November, only one in a thousand was reverted, compared with about one in two hundred for those people approved. Given the majority of AI approvals were for low-risk changes less likely to be reverted, we compared across risk levels and found AI approvals hold comparable to that of humans.\n\nWhere the bottleneck moved\n\nIn our core API repository, the median time from ready for review to merge fell from about 11 hours to under two, but people still tested the product by hand.\n\nStep 4 · QA\n\n## Let Agents Run QA\n\n1. ✓\n2. ✓\n3. ✓\n4. ✓\n5. 5\n6. 6\n7. 7\n\n**The problem:** We launched a new product called [App Studio](https://runway.com/product/app-studio), and unit tests weren’t enough to build confidence. We needed something to use it like a person would, signing up and testing all features and changes.\n\n**What we built:** A QA loop that tests the product like a user. Agents drive browsers through the live product in parallel, flagging anything that doesn’t run as expected. Before reporting a bug, the agent rules out any issues with its own setup and finds the code that caused it. Confirmed bugs become a task for agents to fix, and the loop reruns the scenario to verify it.\n\n**How we knew it worked:** In its first four days, the QA loop found 86 confirmed bugs, and agents fixed all of them. Since early November, every change merged has to pass a verification run on its own preview environment first.\n\nWhere the bottleneck moved\n\nMore changes landing meant more chances to break main, and an engineer still had to uncover the break, find the underlying cause and undo it.\n\nStep 5 · Fix breaks\n\n## Let Agents Watch Main\n\n1. ✓\n2. ✓\n3. ✓\n4. ✓\n5. ✓\n6. 6\n7. 7\n\n**The problem:** At this volume, we can't wait for someone to spot or report issues. Every hour the main codebase stays broken, more changes pile on top of the break.\n\n**What we built:** When an alert fires, agents work through a set of steps: retry the checks, diagnose the failure, apply a known fix if one exists and call a person if none of that works. A coordinator, which we describe more below, runs these steps and holds its own merges until main is green. After each release, our release system monitors production and automatically undoes the release if a health check fails.\n\n**How we knew it worked:** In its first two weeks it saw about 140 alerts. All but one were fixed without the coordinator having to call anyone in, with a median of 23 minutes from red to green. Automatic rollback has fired in production.\n\nWhere the bottleneck moved\n\nEvery station now had an agent, but an engineer still decided what each one worked on and in what order.\n\nStep 6 · Run the line\n\n## Let Agents Run the Factory\n\n1. ✓\n2. ✓\n3. ✓\n4. ✓\n5. ✓\n6. ✓\n7. 7\n\n**The problem:** Engineers were the line’s coordinators, responsible for starting agents, ordering merges and babysitting the runs that stalled.\n\n**What we built:** A coordinator that runs the software factory. It reads the issue tracker, hands work to agents and orders merges. It brings in a person when our rules require one or when a step gets stuck.\n\n**How we knew it worked:** We earned trust in the coordinator the same way we did for AI approval. First it ran read-only, only tracking its decisions. Before production, a person judged every action it proposed in short test runs. In the first two weeks after the coordinator went live, it dispatched over 14,000 agent tasks, from diagnosis and planning to review and fixes, which led to about 1,900 pull requests.\n\nWhere the bottleneck moved\n\nWith the line running itself where our rules allow, more of the wait is now upstream, in deciding what to build and answering agents’ questions.\n\nWhere the line is now\n\n## Where Each Station Stands Today\n\nThe line is furthest along in the codebase behind App Studio. Nearly every pull request merged there in October and November was written by agents, and our agent service drove about seven in ten of them in October.\n\nBy late October, we saw about 2.5 times as many weekly merged pull requests across our main repositories compared to September. The median approval wait fell from over two hours to about five minutes.\n\n| Getting a change to green | Engineers did it by hand | Agents drive 30% to 40% of merges | \n| Review | People reviewed every pull request, about 200 a week in our core API repository | AI reviews every pull request, about 1,000 a week in our core API repository and 3,000 across six repositories. People still approve the high-risk ones | \n| Approval | The median pull request waited 2.3 hours for its first approval | Median wait of about five minutes; about seven in ten reviewed merges go through on AI approval alone | \n| Testing the product | People clicked through manually | Agents test like users, with a median of 4.6 hours to a merged fix | \n| After the merge | Someone noticed an alert or got paged | Agents react to all alerts within seconds and rollback production if needed | \n| Coordination | Engineers | 14,000+ agent tasks, about 1,900 pull requests opened and 442 merges ordered by the coordinator in its first two weeks | \n\nThe new job\n\n## The Engineer’s Role\n\nOur engineers now read a smaller share of the diffs. They decide what gets built, answer agents’ questions about architecture, set the rules for what ships and fix the places where agents stall. That last job has four parts:\n\n- **Find where agents waste effort:** We look for runs that stall and for the parts of the codebase where agents’ changes keep needing rework.\n- **Remove the blockers:** Many of our biggest wins have been access and infrastructure fixes. If an agent can’t reach a test environment, credential or preview deploy, this stalls everything behind it.\n- **Make the path agent-friendly:** We organize codebases so agents can find their way around them, write playbooks for failures we’ve already seen and set up better patterns and core systems so agents can use them safely.\n- **Keep the measures honest:** We decide how each station will be judged before we look at its results.\n\nTo find where agents waste effort, we built a tool that measures friction from the AI agent’s record on every pull request, including how long it takes to implement the change, how many problems the reviewer finds, how serious they are, whether a problem comes back after an agent’s fix and how long a change takes to come clean. It compares each area of a codebase with what’s typical for a pull request of the same size and draws the result as a map, so the areas where agents struggle most stand out.\n\nEach area on the map comes with a brief an agent can act on: the evidence, a likely cause, a bounded change and a test that keeps it fixed. In our first cleanup, 23 pull requests in one internal service reused existing helpers, removed dead code and added reuse checks to CI, and duplicated lines in the files they touched fell by 16%.\n\n## What's Next\n\nAutomating software delivery shifts the bottleneck upstream, to deciding what to build in the first place. Our long-term goal is a system that takes a description of an outcome and ships it, bringing in people at defined points where human judgment is required.\n\nIf you want to help build the next factory, [we're hiring](https://runway.com/careers).", "url": "https://wpnews.pro/news/building-a-software-factory", "canonical_source": "https://runway.com/news/engineering/building-a-software-factory", "published_at": "2026-11-20 17:33:02+00:00", "updated_at": "2026-11-20 17:46:37.743829+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["Felix Rieseberg", "Anthropic", "OpenAI"], "also_reported_by": [], "takeaway": "Handing each delivery stage to agents with shared context — an agent that stays on a pull request until CI is green and an AI reviewer that sees the plan, diff and surrounding code — cut median first-approval wait from over two hours to about five minutes and raised the AI reviewer's resolved-thread", "recommendation": {"advice": "Give your AI reviewer the same context as the coding agent — plan, diff and surrounding code — and let them iterate until clean; the team saw 93% of review threads resolved or driving a code change, up from 84%.", "playbook": "coding-agents", "playbook_url": "https://wpnews.pro/playbooks/coding-agents"}, "alternates": {"html": "https://wpnews.pro/news/building-a-software-factory", "markdown": "https://wpnews.pro/news/building-a-software-factory.md", "text": "https://wpnews.pro/news/building-a-software-factory.txt", "jsonld": "https://wpnews.pro/news/building-a-software-factory.jsonld"}}