How we stopped working for our agents: when software factory DOESN'T work

A software team in early 2026 found that automating implementation with AI agents shifted the bottleneck from writing code to reviewing it, increasing review time and lowering engineer happiness. They concluded that implementation was the wrong first thing to automate and pivoted to building a dedicated review environment that gives agents full workspace context, unlike off-the-shelf diff-based tools. The team's experience highlights the importance of matching agent capabilities to the right tasks and environment.

Like most teams in early 2026, we started with agents on our own machines. Each of us had Claude Code or Cursor open all day. You’d paste a Linear ticket, ask the agent to implement it, review its output in the same session, manually run the app to check nothing was broken, then push and open a PR. The agent was a local tool, like a compiler, and it ran when you told it to, on your laptop, in your terminal. This worked well enough that we started leaning on it for everything: implementation obviously, but also “review this PR for me,” “check if this change breaks the CLI,” “write the migration and verify it runs.” The agents were plenty capable, but we were still the glue between them. Every task required a human to start it, watch it, and decide what to do with the result. We were working for our agents as much as they were working for us. The first thing we tried to move off our laptops was implementation, which in hindsight was the naive choice. We deployed an agent that could take a Linear ticket end to end: boot a sandbox, read the issue, explore the codebase, write code, run tests, open a PR. The initial thing was impressive enough that we immediately started talking about what to automate next, but we hadn’t paused to notice what happened after the PR was opened. What happened was that we reviewed it carefully and we noticed that the reviews were taking longer than they used to. Agent-generated code looks plausible at a glance, but the design decisions are often wrong in ways that take real effort to catch. An agent will pick an approach, hit a wall, and then spend twenty tool calls forcing it to work instead of stepping back and choosing a different design. The result is code that passes tests but is built on a bad idea, and reviewing that is harder than reviewing a bug, because you have to understand the alternative the agent didn’t consider. We were spending more time reviewing and manually verifying agent output than we had spent writing the code ourselves. The bottleneck hadn’t disappeared; it moved from writing to verification. Time spent reviewing bad code went up, engineers’ happiness went down. We went back to square one and asked a question that felt counterintuitive at the time: what if implementation is the wrong first thing to automate? It’s all about the right environment We looked at off-the-shelf review tools first. A PR opens, an agent reads the diff, posts comments. But these are glorified diff checkers. They see the changed lines and nothing else. They’ll catch style nits and obvious single-file bugs, but they’ll never say “this is going to break the CLI” because they have no context beyond the diff. Consider a PR that changes the timeout field on sandbox creation from seconds to milliseconds, for consistency with another internal API. The backend code is updated, tests pass, the diff looks correct. But the CLI sends --timeout 30 meaning 30 seconds, which is now 30 milliseconds. The frontend sends timeout: 300 meaning 5 minutes, which becomes 300 milliseconds. Nothing crashes. Sandboxes just time out almost instantly and users get cryptic errors. The bug is a silent semantic shift across three repos, and no test in any of them catches it. A diff-based reviewer sees a clean backend change and moves on. We were already using agents to review PRs on our own machines, and they didn’t work like that at all. Given a proper workspace, the agent ran grep -r to find callers of the changed function, checked out related PRs in other repos, read the API docs, opened the SDK source to verify the contract still held. It was doing exactly what a good human reviewer does, except faster and more systematically, as long as it had access to the right files. The workflow was great. It was just manual, and it only ran when someone remembered to kick it off. So we needed to give the reviewer the same kind of environment it had on our laptops, but running on its own. Coding agents behave like developers, not like CI jobs: they install packages, start databases, modify system config, leave files around. Two agents on the same machine step on each other the same way two developers sharing a single laptop would. So each agent gets its own sandbox https://docs.islo.dev/cli/sandbox-commands : an isolated machine with its own filesystem, its own processes, full root access, and no way to interfere with anything else running in the system. The next problem is that provisioning a machine from scratch installing dependencies, cloning repos, compiling Rust binaries, running database migrations adds up to 20-30 minutes before the agent can do anything useful. So instead of setting up every time, we use snapshots https://docs.islo.dev/cli/snapshots to freeze a fully prepared sandbox’s disk state and restore it in seconds when a job starts. We maintain a code snapshot called islo-code with all repos cloned, dependencies installed, and toolchains ready, rebuilt on a schedule by CI so it stays current with main . The reviewer boots into a warm workspace with the full codebase already on disk. Sandboxes are also persistent across the lifecycle of a piece of work. If the agent gets feedback and needs to iterate, the same sandbox is reused with the same filesystem state, the same session history, everything intact. The review agent’s prompt opens like this: You are reviewing PR {{PR NUMBER}} in {{REPO}}. You are on the PR branch inside an isolated sandbox VM. You have full root access and can do whatever you need: install packages, start services, run databases, build and run the app. This is your sandbox, use it freely. Variables like {{PR NUMBER}} and {{REPO}} get substituted by the harness at runtime. The reviewer can grep -r across repos, start the API server and hit endpoints, check CI status with gh pr checks . When someone pushes fixes and the reviewer runs again on the same PR, it resumes its session and has its full history, so it knows which comments the author addressed, which issues were ignored, and doesn’t repeat itself. The prompt also tells it not to nitpick formatting CI handles that , and to check for open PRs in other repos before flagging a “missing endpoint” because the repos in the snapshot may be on main while active development is on branches . Verification as evidence, not opinion With review working, the next bottleneck became verification . CI ran, tests passed, the reviewer said it looked good, but nobody had actually run the product to see if the feature worked. This too was something we were already doing locally or on staging. Going back to the timeout example: cargo test runs the API’s unit tests, which were updated to use milliseconds. They pass. But nobody booted the CLI and ran islo use --timeout 30 to see if the sandbox actually stays up. Nobody opened the frontend and created a sandbox to see if it dies after 300 milliseconds. Those are different codepaths, in different repositories, all passing the wrong value. Tests verify the code that was modified - verification means running the entire product and checking that the thing actually works end-to-end. Building the verification environment The code snapshot islo-code that powers review and implementation has repos on disk and toolchains ready, but it can’t run the full product. The verifier needs the entire Islo platform compiled, configured, seeded with test data, and bootable with one command. That’s a separate, much heavier platform snapshot we call islo-platform : 8 vCPUs, 16 GB of memory, and about 50 GB of disk with everything pre-warmed. A CI workflow rebuilds this platform snapshot every two hours, creating a fresh sandbox with all six service repos cloned, then running a build script that installs and compiles everything: Rust services : the compute worker manages VM lifecycle on bare metal , the network gateway proxy, credential injection, routing , and the CLI, all compiled in release mode Python : the control-plane API with dependencies installed and database migrations pre-run Node : the frontend dashboard with dependencies installed, plus browser-use with Chromium for browser-driven testing PostgreSQL : seeded with a test tenant, user, API keys, and compute region configuration Auth : a real Descope project with a fixed test user and deterministic OTP, so the agent can log in through the actual auth UI without any mocking Toolchains : gh , the Islo CLI built to talk to the local API, and everything else an agent might need The build writes a manifest file recording the commit SHA of each repo at build time. When the snapshot is restored later, launch-platform compares those SHAs against origin/main to decide which services need rebuilding, so a snapshot that’s a few hours old doesn’t require a full recompile. After the build completes, CI saves the snapshot and tears down the build sandbox. The whole process takes about 15 minutes. Booting the stack When the verify job runs, its first step calls launch-platform , which checks out any PR branches, selectively rebuilds changed services, and then boots the full stack. The boot sequence is ordered by dependency: PostgreSQL starts, migrations run Control-plane API starts handles auth, tenants, sandbox routing Network gateway starts proxies requests between services, injects credentials Compute worker starts manages VM lifecycle, exposes the compute API Frontend dashboard starts, with dev server proxying API and compute requests to the local services Browser pre-authentication : a script uses browser-use to navigate the real Descope login UI, enter the test user’s credentials, and store the authenticated session so the verify agent can open the dashboard already logged in The browser session is pre-authenticated against real Descope. When the verify agent opens http://localhost:5173 with browser-use , it sees the dashboard as a logged-in user who went through the actual auth flow, session handling, and permission checks. When a change spans multiple repos, launch-platform accepts flags to check out specific branches or PRs per service: launch-platform --islo-frontend pr/244 --islo-web-api pr/448 It only rebuilds the services that changed. If only the frontend PR is being tested, the Rust services stay on their snapshot versions and don’t need recompilation. If both frontend and backend changed, both get rebuilt and the full stack restarts. How the agent verifies The verification prompt tells the agent to interact with the feature the way a user would. If the feature is a UI flow, the agent opens the browser with browser-use , fills out forms, clicks buttons, and takes screenshots at each step. If it’s a CLI command, the agent runs it. If it’s an API change, the agent calls it through the SDK. The point is to exercise the actual user-facing path, not test internal plumbing. Every claim in the verification report must have command output or a screenshot backing it up. The agent also has access to service logs for every component in the stack, psql for direct database inspection, and curl for hitting API endpoints. When something fails, the prompt tells the agent to check the relevant service log for the error before reporting. The goal is that the verification report contains enough evidence for a human to audit the result without needing to reproduce it. When verification passes, the agent posts a structured report on the PR and notifies the team on Slack. Each scenario gets a pass/fail verdict with screenshots showing exactly what the agent saw: When it fails, it requests changes on the PR with specific findings. The verify agent never modifies code and never pushes commits. It only observes and reports. Implementing again, with a safety net With review and verify working, we returned to implementation. This time the agent had a feedback loop: it writes code, review catches cross-repo issues, verification catches end-to-end failures, and change requests route feedback back to the implementer automatically. The code doesn’t need to be perfect on the first try because the agent iterates until it passes. The implementer reads a Linear issue, explores the codebase, writes the change, runs the repo’s own test suite and linters locally, then opens a PR. The PR is automatically picked up by the reviewer. If the reviewer approves, verification runs next. If either requests changes, the feedback goes back to the implementer, which resumes its session and pushes fixes. We added an iteration guard because agents in tight feedback loops will burn through tokens indefinitely if you let them. The agent’s configuration increments a counter file before each run and exits if it exceeds five, and the prompt itself tells the agent to count its own commits and stop after five, posting a comment on the PR and the source Linear issue saying a human should take over. Wiring it together with trigger rules At this point all four agents plus the delegator, which I’ll get to were working individually, but the orchestration between them were still manual. A human had to trigger the reviewer after a PR opened, kick off verification after review passed, route feedback back to the implementer when changes were requested. The agents were capable but the glue was still us. The orchestration layer uses Islo’s automations https://docs.islo.dev/concepts/automations features: each agent is defined as a durable job with a job.toml manifest that specifies its sandbox requirements, snapshot, compute size, and the commands to run. Incoming webhooks https://docs.islo.dev/concepts/incoming-webhooks receive events from GitHub and Linear, and declarative trigger rules match conditions against the webhook JSON payload to decide which job to start. For example, the review agent’s trigger for PR opens: rules rules.when op = "all" rules.when.conditions op = "equals" json path = "$.action" value = "opened" rules.actions action type = "trigger job" job name = "review" rules.actions.params.repo source = { source = "json path", path = "$.repository.full name" } rules.actions.params.pr number source = { source = "json path", path = "$.pull request.number" } Each agent owns its trigger rules in agents/<role /trigger-rules/<source .toml . A build script assembles all fragments from the same source GitHub, Linear into a single webhook configuration per platform, which gets deployed with islo webhook incoming create . After that, GitHub sends events to Islo, the trigger rules match against the payload, and the corresponding job starts automatically. The coordination between agents is event-driven: each agent’s verdict approve, request changes, pass, fail is a structured signal that the trigger rules can match on without parsing prose. The triggers are visible to humans too, so you can always see where a PR is in the loop by looking at its state on GitHub. The delegator: routing mentions to the right agent In practice, we saw that the loop rarely ends without humans. Follow-ups arrive from multiple surfaces: a teammate comments on the PR asking the agent to reconsider its approach, a Linear comment adds a new requirement, someone on Slack says the rate limiter is too aggressive. These are all about the same piece of work but they arrive on different platforms with no shared identifier linking them. The obvious first instinct is to solve this with code: parse the webhook, look up the PR number in a database, find the matching sandbox, forward the message. But the correlation problem is harder than it looks. A Linear comment might reference an issue that spawned work across two repos, each with its own implementer sandbox. A Slack message might mention a feature name that doesn’t map to any PR number at all. The mapping between “incoming event” and “right agent and session” requires judgment: reading the comment’s content, understanding which agent’s work it’s about, checking which sandboxes are still active. That’s a correlation problem you can’t solve with a lookup table. So the delegator is itself an agent - you can think about it like a smart router. It’s ephemeral 1 vCPU, 1 GB, torn down after completion , and it doesn’t implement, review, or verify anything. It reads the webhook payload, reasons about which worker sandbox and session the message belongs to, and hands it off. It lists active sandboxes, reads their recent agent session logs, and decides which sandbox and session the incoming message belongs to. Once it finds a match, it resumes that session with a short handoff prompt containing the new context and gets out of the way. This is the sandbox persistence I mentioned earlier in action. An implementer sandbox survives across the initial implementation, review feedback cycles, CI failure fixes handled by the babysit agent on the same PR branch , and ad hoc human comments. Each run resumes the same Claude session, so the agent has the full history of what it tried, what failed, and what feedback it received. The filesystem holds the codebase state, the disk holds the session history, and when the sandbox is paused between runs, all of it is preserved. The sandbox is the agent’s identity, capability, and memory in one object. The human stays in the loop The loop runs autonomously, but the human is never locked out. At any point you can comment on the PR, manually re-trigger review or verification, or SSH into the sandbox and take over the session directly. The delegator makes this easy on the GitHub side: an @islo-agent comment on any of our PRs routes straight into the right worker session with full context. The default is autonomous. A human adds a label to a Linear ticket and the system takes it from there: sandbox boots from a snapshot, agent reads the ticket, implements the change, opens a PR, review runs, CI failures get fixed by the babysit agent, verification boots the full stack and tests end-to-end, and the PR comes back with a structured verification report. The human’s role shifts from doing every step to reviewing evidence and merging when satisfied. The agents make mistakes regularly. The loop works because the mistakes get caught by other agents, the feedback routes back automatically, and the evidence at the end is real: HTTP responses, screenshots, database queries, command output. You’re looking at the proof, not trusting the agent’s word for it. What we learned building this Your agent deserves its own computer. The reviewer is a 4-vCPU VM with every repo on disk. The verifier is an 8-vCPU VM with the full stack running. The implementer is a long-lived workspace that persists across feedback cycles. Each agent gets a machine shaped exactly to its task, and each machine is dedicated to exactly one agent. When we tried sharing environments or giving agents access to a shared server, everything broke in subtle ways: state leaked between runs, one agent’s packages conflicted with another’s, session files got overwritten. The one-to-one mapping is a correctness requirement. Verification is the hardest thing to automate, and the most valuable. Everyone wants to start with implementation because that’s the visible bottleneck. But without automated review and verification, you’re just creating more work for the humans who now have to evaluate agent-generated code with less context than they’d have had if they’d written it themselves. Start with the safety net, then let the agents swing. Observability is how you know what to fix. Without visibility into what agents are actually doing, you’re guessing at why a review missed something or why a verification burned through its token budget. We audit out agent sessions regularly: which tools it called, how many tokens it used, where it got stuck, how long each step took. That data is what drives prompt tuning, environment decisions, and cost control. Integration work is important. The agent code, the prompts, the harness, all of that was the smaller part of the effort. OAuth flows, webhook wiring, credential injection through the gateway, trigger rule assembly, session routing in the delegator, cross-platform event correlation, the iteration guards, making launch-platform reliable enough that the verify agent doesn’t spend too much debugging the boot sequence: that’s where the days went. If you’re planning to build something like this, budget your time for plumbing. Related Reading islo-agents https://github.com/islo-labs/islo-agents - The open-source repo with the prompts, job definitions, and trigger rules described in this post Stripe Minions https://stripe.dev/blog/minions-stripes-one-shot-end-to-end-coding-agents - Stripe’s journey into building a software factory