{"slug": "running-an-agent-fleet", "title": "Running an agent fleet", "summary": "A single developer running the coding-eval startup Stet.sh reports building a 1M+ line-of-code product alone by operating an \"agent fleet\" of parallel coding agents, a setup the developer says is achievable on a $200/month lab subscription rather than the 30x-payroll token spend seen elsewhere. The post argues fleets succeed only if an engineer minimizes \"interaction slippage\" — the coordination and polling overhead between agents — and maintains long-horizon alignment so agents pursue KPIs without Goodharting. It cites Spawn.co's 10 people spending $3.1M/month on tokens and Coinbase CEO Brian Armstrong's lead-agent-per-focus-area structure as examples, and distinguishes single-player fleets from multiplayer, code-defined \"software factories.", "body_md": "November 12, 20269 min read\n\n# Running an agent fleet\n\nCoding agents make individual engineers faster at closing tickets, but they quickly become limited by attention bandwidth.\n\nEngineers who break past 5-10 parallel chats have moved up a layer of abstraction to one of two things: an agent fleet or a software factory.\n\nThink of the difference between the two like an IDE vs GitHub. A **fleet** is single player, experimental, with high coupling to one engineer’s mental model and context, to automate an org chart that hill climbs open ended goals. A **factory** is multiplayer, decomposable, code-defined infrastructure that automates the software assembly line owned by a team.\n\nRunning a fleet lets one engineer build and steer a massive product, provided you master two things: minimizing **interaction slippage** (the coordination and polling overhead between agents in the fleet) and maintaining **long horizon alignment** — letting the agents cook on KPIs without Goodharting, all while knowing when to step in and steer the fleet.\n\nIn this post, I’ll go into depth on how I build and run my agent fleet.\n\n## What am I doing with all of these tokens?\n\nTo answer the inevitable question of what I’m doing with all of these tokens, and why I need a fleet at all: I’m building Stet.sh, a product that automates coding agent eval creation so operators can find the cost/quality Pareto curve on their repo.\n\nThis is a greenfield-ish project (1M++ LOC) with me as the only developer. (1M LOC is NOT a flex — left to their own devices, agents write a bunch of bloated, additive, grotesque code, and never clean up after the mess they made. Examples are building parallel pipelines, tons of opt in flags, and wrapping layers, all while a simple refactor would have been cleaner.)\n\nIt’s easy to dismiss this as a 1 engineer, greenfield startup, whose founder is likely suffering from AI psychosis. Despite the differences to enterprise development (which I am also intimately familiar with), I believe the core concepts here should transfer.\n\n## Agent fleets vs software factories\n\nBoth fleets and factories are layers of abstraction on top of a single coding agent, and both can hill climb metrics, but they operate pretty differently and serve different purposes:\n\n**Examples in the wild:**\n\n- **Spawn.co:** 10 people spending 30x payroll ($3.1M/mo) on tokens; permanent named agents owning engine, ops, finance, and deploys ([source](https://x.com/jsnnsa/status/2115623382721126826) ).\n- **Brian Armstrong (Coinbase CEO):** Lead agent per focus area + Chief of Staff feeding a single prioritized local review queue ([source](https://x.com/brian_armstrong/status/2119112609223827547) ).\n- **Math & Science Swarms:** OpenAI Navier-Stokes, OpenAI HuggingFace, Anthropic Fermat’s Last Theorem ([source](https://claude.dev/blog/safely-scaling-multi-agent-teams-with-fable-5-2/) ).\n\nYou don’t need to be a crypto billionaire or a research lab to run a fleet though — it’s achievable with a $200/mo lab subscription.\n\nHowever, these categories are not rigid. For example, an individual might start off with a fleet, stabilize the repeatable parts into shared infrastructure (factory), and continue using more flexible agents for the fuzzy bits. These two are continually feeding each other.\n\n**Why factories work for teams:**\n\n- **Decomposable & modular:** Swap different agents/models in and out of each stage ([Zach Lloyd](https://x.com/zachlloydtweets/status/2118725953966715217) ).\n- **Stage optimizable:** Hill climb specific stages like ticket quality, QA catch rate, or “regret-free merge rate” ([Eno Reyes](https://x.com/EnoReyes/status/2118767454721851680) ).\n- **Multiplayer floor raising:** Standardizes how an entire org ships; often best started as a specific**bug factory** ,**perf factory** , or**security factory** ([Claire Vo](https://x.com/clairevo/status/2119099298575192119) ).\n\n## Components of my fleet\n\nInstead of managing a team of coding agents, you are now the VP of a firm of coding agents.\n\n- **Chief of Staff / Orchestrator:** Persistent agent that conveys my intent and direction into fleet outcomes. Almost all of my messages (new work, corrections, concerns, status) come to this agent.\n- **Team Leads:** These are persistent, thread-level agents that own high level, end-to-end outcomes. I give TLs clear, defined ownership, and authority to execute on that vision, clearing blockers themselves instead of playing hot potato with other TLs.\n- **Leaves:** These are ephemeral, bounded subagents that TLs use to parallelize execution. I do role based routing for leaf agents, based off of provider quota and the cost/quality determined by Stet.\n\nAs of writing this (November 2026), Opus 5.5 is the only model I trust to run the fleet’s management layer (Chief of Staff and Team Leads). Using GPT-6.1 Sol and GPT-6 Astra resulted in decoherence and misalignment after a few days. A Codex agent is also extremely hard to understand, talking in cryptic messages that only it can decipher. Use Opus for the brains, and Sol/Luna for the hands (Leaves).\n\n(For context on scale: in a typical 7-day window, this fleet spawns 1,800+ leaf threads, with each Team Lead waking ~45 times a day.)\n\n## Interaction slippage\n\nEvery inter-agent message costs tokens (using a turn), causing you to pay **interaction slippage**. For example, every TL messaging Chief of Staff about every status update causes the Chief to use turns ack’ing information it didn’t need.\n\nThe fix is to minimize slippage while increasing communication efficiency. One example is replacing peer to peer DMs with append only broadcast channels, banning blocker hot potato (TLs fix blockers on their own), and replacing 15 minute polling timers with background waiters or heartbeats using harness automations.\n\n- Moving from peer-to-peer DMs to append only broadcast channels (`fleet.md` ,`decisions.md` ) cut inter-agent DMs from 65/day down to 28, and banning status updates cut channel noise in half (previously ~50% of our 300-400 daily channel lines were pure status chatter).\n- Multi-hop handoffs (watcher -> launcher -> Chief of Staff relay -> TL) were adding 30-50 minutes of dead latency per failure (52 minutes via relay vs 4 minutes when the owning TL drove the rerun directly).\n- 24% of our entire Claude token spend was coming from bloated owner turns past their 20th tool call. Adding a turn budget hook that nudges TLs to wrap up or delegate at tool call #20 cut completion turn calls from 5.6 to 3.2 and helped drop Claude token burn by 44%.\n\nEfficient agent communication is still an area of active exploration, and I expect the tooling here to get better over time.\n\n## Long horizon alignment\n\nA central tension of the fleet is **autonomy vs alignment**: you don’t have the attention bandwidth to be in the loop for every decision, so you align the fleet at a high level, and let it work towards your goals on its own. But, left alone for too long, agents drift and get misaligned, Goodhart their metrics, or get stuck in local maxima.\n\nGiven enough time, agents tend to subtly start optimizing for proxies / facades instead of what you actually care about.\n\nI was happy here for a few months, until context started to rot. Over a three week stretch, `AGENTS.md` bloated by 166% (from 1,000 to 2,661 words) and our fleet process docs grew 60%. Because new decisions were being appended to a log instead of replacing old rules, the Grading TL drifted in five different places over four days.\n\nWe split fleet state into two primitives: **standing direction (`DIRECTION.md`) is rewritten in place under a hard 1,200 word cap.** Every new decision rewrites a specific line and explicitly names what old rule or code path it retires. Meanwhile, **history and cross owner channels (`decisions.md`, `fleet.md`) are append only logs**, where owners post a single stamped line only when a fact changes another owner’s next action.\n\n### Hill climbing\n\nHill climbing is perhaps the most powerful technique available for running a fleet. And, as much as I dislike the industry’s fetishization of KPIs — you gotta have some KPIs.\n\nFigure out what metrics are vital to your success. Give each TL a metric to optimize, a fast way to verify/replay, and let them cook. You’ll be surprised at what they come up with.\n\nThe main failure mode here is letting the agents Goodhart (optimize against the metric without achieving the underlying goal). If you give the agent a single metric and a time budget, it *will* Goodhart it, hardcoding metrics, cheating, and ultimately pulling the wool over your eyes as to what’s really going on.\n\nEven then, getting the reward function right took painful iteration:\n\nEarly on, I gave the fleet a hill-climb metric for “failure classes found” during repo onboarding. Within 24 hours, the fleet spun up 120 runs and dutifully logged 47 new failure classes, while producing 0 working tasks. When I switched the metric to “failure classes closed,” the agents produced 47 “closes” by splitting 14 underlying issues into sub-labels, and marked 92 classes as “eliminated” when 26 of them didn’t even have a fix commit. The fix was tightening the metric to count only *failure classes closed by a stamped fix SHA that does not re-emit for 24 hours across at least two repos*.\n\nThis can be prevented by:\n\n- Using holdouts — give targets a holdout metric that the agent structurally can’t optimize against.\n- Ban narrow heuristics - don’t use a metric like # of tests.\n\nAs an example: optimizing repo onboarding on tRPC while replaying every fix against hono as a shadow holdout (any tRPC lift that regresses hono is rejected as overfit), and forbidding repo or test-runner specific heuristics in our code. Over five days, the share of new held-out repos whose environment setup passed on the first attempt went from 25% (1 of 4) to 78% (18 of 23).\n\nAs Matt Huang [puts it](https://x.com/matthuang/status/2120209025099043286?a=FayPfwT47Kb5KsBbITA&s=20):\n\nThe art of swarm riding is different from software engineering and more akin to machine learning, where practitioners grapple with black box systems: defining reward functions and evals, transmuting non-verifiable problems into verifiable problems, running experiments, etc.\n\nThe art of hill climbing is all about defining reward functions and optimizing against them.\n\n### Parachuting (Breaking Out of Local Maxima)\n\nEven if they’re not expressly cheating, agents can get misaligned and stuck in local maxima, eking out 1-2% gains on a strained architecture, or piling up parallel code paths.\n\nAs the operator, your responsibility is to detect when that’s happening, and parachute in, find a new, more ambitious / effective direction, and align the agent in this new direction.\n\nHow I know when to parachute in (aka **The Tells**):\n\n1. Repeated patching and additive bloat (or whack-a-mole): Seeing TLs land patch after patch for individual errors on a specific codepath indicates that there’s an opportunity for fixing a root cause architectural issue. The agent is stuck fixing symptoms instead of changing the abstraction.\n2. Long iteration cycles. Seeing TLs wait hours for feedback on their patch because the test->build->replay loop is slow. They will dutifully wait and accept the pain unless you step in. Short feedback loops are crucial to a successful fleet, so this is a big drag.\n\nIn one 7 day window, the fleet added +75,981 net lines of non test code and 13 separate opt-in flags. Each TL was landing narrow symptom patches and parallel codepaths to avoid breaking peers. That triggered a parachute drop: freezing new mining and spinning up a consolidation sprint to delete the 13 flags, and collapse parallel paths into one state machine.\n\n## Recipe for Success\n\nThe role of the engineer has shifted **from making the product itself to making the fleet that makes the product**. You become a director, systems architect, and curator rather than a doer.\n\nPutting all this together — how do you run a fleet? Here’s a few steps to get started:\n\n1. Use a harness that allows for thread coordination at minimum.\n2. Create a Chief of Staff thread on Opus 5.5.\n3. Give it this article.\n4. Ask it to brainstorm Team Lead ownership areas and hillclimb metrics for those TLs.\n5. Have the Chief of Staff create TLs and start working (using Opus for TLs, and whatever you want for research / implementation leaves).\n6. Parachute in, using your check ins to watch for whack-a-mole patches and slow iteration loops.\n\n## FAQ\n\n### What's the difference between an agent fleet and a software factory?\n\nA fleet is single player and experimental: an org chart of persistent agents coupled to one engineer's context, hill climbing open ended goals. A software factory is multiplayer, code defined infrastructure that runs the SDLC as an assembly line owned by a team.\n\n### Should I start with an agent fleet or a software factory?\n\nStart with a fleet. Stabilize the repeatable parts into shared infrastructure (a factory), and keep using flexible agents for the fuzzy work. The two keep feeding each other.\n\n### How do I stop agents from Goodharting their hill-climb metrics?\n\nGive each Team Lead a holdout metric it structurally can't optimize against, and ban narrow heuristics. Replaying every tRPC onboarding fix against hono as a shadow holdout took first-attempt environment setup on new held-out repos from 25% (1 of 4) to 78% (18 of 23) in five days.", "url": "https://wpnews.pro/news/running-an-agent-fleet", "canonical_source": "https://www.stet.sh/blog/running-an-agent-fleet", "published_at": "2026-11-12 12:00:00+00:00", "updated_at": "2026-11-12 16:53:35.402637+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["Spawn.co", "Brian Armstrong", "Coinbase", "OpenAI", "Anthropic", "Zach Lloyd", "Claire Vo"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/running-an-agent-fleet", "markdown": "https://wpnews.pro/news/running-an-agent-fleet.md", "text": "https://wpnews.pro/news/running-an-agent-fleet.txt", "jsonld": "https://wpnews.pro/news/running-an-agent-fleet.jsonld"}}