Running an agent fleet

A single developer running the coding-eval startup Stet.sh reports building a 1M+ line-of-code product alone by operating an "agent fleet" of parallel coding agents, a setup the developer says is achievable on a $200/month lab subscription rather than the 30x-payroll token spend seen elsewhere. The post argues fleets succeed only if an engineer minimizes "interaction slippage" — the coordination and polling overhead between agents — and maintains long-horizon alignment so agents pursue KPIs without Goodharting. It cites Spawn.co's 10 people spending $3.1M/month on tokens and Coinbase CEO Brian Armstrong's lead-agent-per-focus-area structure as examples, and distinguishes single-player fleets from multiplayer, code-defined "software factories.

November 12, 20269 min read Running an agent fleet Coding agents make individual engineers faster at closing tickets, but they quickly become limited by attention bandwidth. Engineers who break past 5-10 parallel chats have moved up a layer of abstraction to one of two things: an agent fleet or a software factory. Think of the difference between the two like an IDE vs GitHub. A fleet is single player, experimental, with high coupling to one engineer’s mental model and context, to automate an org chart that hill climbs open ended goals. A factory is multiplayer, decomposable, code-defined infrastructure that automates the software assembly line owned by a team. Running a fleet lets one engineer build and steer a massive product, provided you master two things: minimizing interaction slippage the coordination and polling overhead between agents in the fleet and maintaining long horizon alignment — letting the agents cook on KPIs without Goodharting, all while knowing when to step in and steer the fleet. In this post, I’ll go into depth on how I build and run my agent fleet. What am I doing with all of these tokens? To answer the inevitable question of what I’m doing with all of these tokens, and why I need a fleet at all: I’m building Stet.sh, a product that automates coding agent eval creation so operators can find the cost/quality Pareto curve on their repo. This is a greenfield-ish project 1M++ LOC with me as the only developer. 1M LOC is NOT a flex — left to their own devices, agents write a bunch of bloated, additive, grotesque code, and never clean up after the mess they made. Examples are building parallel pipelines, tons of opt in flags, and wrapping layers, all while a simple refactor would have been cleaner. It’s easy to dismiss this as a 1 engineer, greenfield startup, whose founder is likely suffering from AI psychosis. Despite the differences to enterprise development which I am also intimately familiar with , I believe the core concepts here should transfer. Agent fleets vs software factories Both fleets and factories are layers of abstraction on top of a single coding agent, and both can hill climb metrics, but they operate pretty differently and serve different purposes: Examples in the wild: - Spawn.co: 10 people spending 30x payroll $3.1M/mo on tokens; permanent named agents owning engine, ops, finance, and deploys source https://x.com/jsnnsa/status/2115623382721126826 . - Brian Armstrong Coinbase CEO : Lead agent per focus area + Chief of Staff feeding a single prioritized local review queue source https://x.com/brian armstrong/status/2119112609223827547 . - Math & Science Swarms: OpenAI Navier-Stokes, OpenAI HuggingFace, Anthropic Fermat’s Last Theorem source https://claude.dev/blog/safely-scaling-multi-agent-teams-with-fable-5-2/ . You don’t need to be a crypto billionaire or a research lab to run a fleet though — it’s achievable with a $200/mo lab subscription. However, these categories are not rigid. For example, an individual might start off with a fleet, stabilize the repeatable parts into shared infrastructure factory , and continue using more flexible agents for the fuzzy bits. These two are continually feeding each other. Why factories work for teams: - Decomposable & modular: Swap different agents/models in and out of each stage Zach Lloyd https://x.com/zachlloydtweets/status/2118725953966715217 . - Stage optimizable: Hill climb specific stages like ticket quality, QA catch rate, or “regret-free merge rate” Eno Reyes https://x.com/EnoReyes/status/2118767454721851680 . - Multiplayer floor raising: Standardizes how an entire org ships; often best started as a specific bug factory , perf factory , or security factory Claire Vo https://x.com/clairevo/status/2119099298575192119 . Components of my fleet Instead of managing a team of coding agents, you are now the VP of a firm of coding agents. - Chief of Staff / Orchestrator: Persistent agent that conveys my intent and direction into fleet outcomes. Almost all of my messages new work, corrections, concerns, status come to this agent. - Team Leads: These are persistent, thread-level agents that own high level, end-to-end outcomes. I give TLs clear, defined ownership, and authority to execute on that vision, clearing blockers themselves instead of playing hot potato with other TLs. - Leaves: These are ephemeral, bounded subagents that TLs use to parallelize execution. I do role based routing for leaf agents, based off of provider quota and the cost/quality determined by Stet. As of writing this November 2026 , Opus 5.5 is the only model I trust to run the fleet’s management layer Chief of Staff and Team Leads . Using GPT-6.1 Sol and GPT-6 Astra resulted in decoherence and misalignment after a few days. A Codex agent is also extremely hard to understand, talking in cryptic messages that only it can decipher. Use Opus for the brains, and Sol/Luna for the hands Leaves . For context on scale: in a typical 7-day window, this fleet spawns 1,800+ leaf threads, with each Team Lead waking ~45 times a day. Interaction slippage Every inter-agent message costs tokens using a turn , causing you to pay interaction slippage . For example, every TL messaging Chief of Staff about every status update causes the Chief to use turns ack’ing information it didn’t need. The fix is to minimize slippage while increasing communication efficiency. One example is replacing peer to peer DMs with append only broadcast channels, banning blocker hot potato TLs fix blockers on their own , and replacing 15 minute polling timers with background waiters or heartbeats using harness automations. - Moving from peer-to-peer DMs to append only broadcast channels fleet.md , decisions.md cut inter-agent DMs from 65/day down to 28, and banning status updates cut channel noise in half previously ~50% of our 300-400 daily channel lines were pure status chatter . - Multi-hop handoffs watcher - launcher - Chief of Staff relay - TL were adding 30-50 minutes of dead latency per failure 52 minutes via relay vs 4 minutes when the owning TL drove the rerun directly . - 24% of our entire Claude token spend was coming from bloated owner turns past their 20th tool call. Adding a turn budget hook that nudges TLs to wrap up or delegate at tool call 20 cut completion turn calls from 5.6 to 3.2 and helped drop Claude token burn by 44%. Efficient agent communication is still an area of active exploration, and I expect the tooling here to get better over time. Long horizon alignment A central tension of the fleet is autonomy vs alignment : you don’t have the attention bandwidth to be in the loop for every decision, so you align the fleet at a high level, and let it work towards your goals on its own. But, left alone for too long, agents drift and get misaligned, Goodhart their metrics, or get stuck in local maxima. Given enough time, agents tend to subtly start optimizing for proxies / facades instead of what you actually care about. I was happy here for a few months, until context started to rot. Over a three week stretch, AGENTS.md bloated by 166% from 1,000 to 2,661 words and our fleet process docs grew 60%. Because new decisions were being appended to a log instead of replacing old rules, the Grading TL drifted in five different places over four days. We split fleet state into two primitives: standing direction DIRECTION.md is rewritten in place under a hard 1,200 word cap. Every new decision rewrites a specific line and explicitly names what old rule or code path it retires. Meanwhile, history and cross owner channels decisions.md , fleet.md are append only logs , where owners post a single stamped line only when a fact changes another owner’s next action. Hill climbing Hill climbing is perhaps the most powerful technique available for running a fleet. And, as much as I dislike the industry’s fetishization of KPIs — you gotta have some KPIs. Figure out what metrics are vital to your success. Give each TL a metric to optimize, a fast way to verify/replay, and let them cook. You’ll be surprised at what they come up with. The main failure mode here is letting the agents Goodhart optimize against the metric without achieving the underlying goal . If you give the agent a single metric and a time budget, it will Goodhart it, hardcoding metrics, cheating, and ultimately pulling the wool over your eyes as to what’s really going on. Even then, getting the reward function right took painful iteration: Early on, I gave the fleet a hill-climb metric for “failure classes found” during repo onboarding. Within 24 hours, the fleet spun up 120 runs and dutifully logged 47 new failure classes, while producing 0 working tasks. When I switched the metric to “failure classes closed,” the agents produced 47 “closes” by splitting 14 underlying issues into sub-labels, and marked 92 classes as “eliminated” when 26 of them didn’t even have a fix commit. The fix was tightening the metric to count only failure classes closed by a stamped fix SHA that does not re-emit for 24 hours across at least two repos . This can be prevented by: - Using holdouts — give targets a holdout metric that the agent structurally can’t optimize against. - Ban narrow heuristics - don’t use a metric like of tests. As an example: optimizing repo onboarding on tRPC while replaying every fix against hono as a shadow holdout any tRPC lift that regresses hono is rejected as overfit , and forbidding repo or test-runner specific heuristics in our code. Over five days, the share of new held-out repos whose environment setup passed on the first attempt went from 25% 1 of 4 to 78% 18 of 23 . As Matt Huang puts it https://x.com/matthuang/status/2120209025099043286?a=FayPfwT47Kb5KsBbITA&s=20 : The art of swarm riding is different from software engineering and more akin to machine learning, where practitioners grapple with black box systems: defining reward functions and evals, transmuting non-verifiable problems into verifiable problems, running experiments, etc. The art of hill climbing is all about defining reward functions and optimizing against them. Parachuting Breaking Out of Local Maxima Even if they’re not expressly cheating, agents can get misaligned and stuck in local maxima, eking out 1-2% gains on a strained architecture, or piling up parallel code paths. As the operator, your responsibility is to detect when that’s happening, and parachute in, find a new, more ambitious / effective direction, and align the agent in this new direction. How I know when to parachute in aka The Tells : 1. Repeated patching and additive bloat or whack-a-mole : Seeing TLs land patch after patch for individual errors on a specific codepath indicates that there’s an opportunity for fixing a root cause architectural issue. The agent is stuck fixing symptoms instead of changing the abstraction. 2. Long iteration cycles. Seeing TLs wait hours for feedback on their patch because the test- build- replay loop is slow. They will dutifully wait and accept the pain unless you step in. Short feedback loops are crucial to a successful fleet, so this is a big drag. In one 7 day window, the fleet added +75,981 net lines of non test code and 13 separate opt-in flags. Each TL was landing narrow symptom patches and parallel codepaths to avoid breaking peers. That triggered a parachute drop: freezing new mining and spinning up a consolidation sprint to delete the 13 flags, and collapse parallel paths into one state machine. Recipe for Success The role of the engineer has shifted from making the product itself to making the fleet that makes the product . You become a director, systems architect, and curator rather than a doer. Putting all this together — how do you run a fleet? Here’s a few steps to get started: 1. Use a harness that allows for thread coordination at minimum. 2. Create a Chief of Staff thread on Opus 5.5. 3. Give it this article. 4. Ask it to brainstorm Team Lead ownership areas and hillclimb metrics for those TLs. 5. Have the Chief of Staff create TLs and start working using Opus for TLs, and whatever you want for research / implementation leaves . 6. Parachute in, using your check ins to watch for whack-a-mole patches and slow iteration loops. FAQ What's the difference between an agent fleet and a software factory? A fleet is single player and experimental: an org chart of persistent agents coupled to one engineer's context, hill climbing open ended goals. A software factory is multiplayer, code defined infrastructure that runs the SDLC as an assembly line owned by a team. Should I start with an agent fleet or a software factory? Start with a fleet. Stabilize the repeatable parts into shared infrastructure a factory , and keep using flexible agents for the fuzzy work. The two keep feeding each other. How do I stop agents from Goodharting their hill-climb metrics? Give each Team Lead a holdout metric it structurally can't optimize against, and ban narrow heuristics. Replaying every tRPC onboarding fix against hono as a shadow holdout took first-attempt environment setup on new held-out repos from 25% 1 of 4 to 78% 18 of 23 in five days.