Finish your software factory: take a bad change back before anyone notices

A developer has released Shipping Floor, an open-source example repo that demonstrates an agentic software factory capable of automatically rolling back a bad change without human intervention. The project uses three AI agents that generate electronic music in the browser, with LaunchDarkly guarded rollouts and feature flags controlling agent behavior and validation limits at runtime. The walkthrough shows how a deliberately tightened gain gate causes one agent's output to be rejected while the others continue, illustrating bounded blast radius and automated reversal.

Most agentic factory stacks stop at the deploy. They generate, they validate, they ship, and then the instructions turn into a vague "monitor" step, which in practice means a person watching a dashboard. The capability that is actually missing is narrower than monitoring and harder to build: a change that moves the process gets taken back without a person. That is a throughput problem before it is a tooling problem. When a person wrote every change, review was the quality system, and a few changes a day could each be read before they shipped. A factory whose agents deploy continuously has no room for that, and the failures it produces are the kind review misses anyway. One of the two failures here passes every check the build side runs. The other one is the checks, all of them firing correctly while the pipeline quietly stops producing. So a factory without this is not slower than one with it. It is less certain. It ships improvements and degradations at the same rate, reports success both times, and leaves you one lever: a deploy-level rollback that takes back everything else along with the change you wanted gone. What you build here instead is a bounded blast radius while a change proves itself, a verdict that does not depend on who is paying attention, and a reversal that touches one change and leaves the rest of the band playing. The example repo builds an agentic band. Its loop generates, validates, and deploys about eleven times a minute. In this walkthrough, you start the music factory, listen to it, break it on purpose, and watch a guarded rollout https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts put it back. Three agents make up your virtual band: a drummer, a bassist, and a keys player. They generate electronic music in the browser while a conductor keeps tempo and key. Every few bars each agent asks an AI model for the next version of its part. A parser accepts or rejects the result. If the new pattern is well-formed, the next cycle plays it. If it is not, the last good pattern keeps looping. Shipping Floor https://github.com/launchdarkly-labs/shipping-floor contains no musical vocabulary of its own, so personas, gain limits, and groove recipes all come from LaunchDarkly prompt snippets https://launchdarkly.com/docs/home/agentcontrol/snippets at runtime. If LaunchDarkly is unreachable, the band holds and then falls silent. There is no local fallback for this demo. Two things about the band can change without a deploy, and each uses a different LaunchDarkly object. What an agent is , its model, its parameters, and a system message assembled from pinned prompt snippets, is a variation in an AgentControl config https://launchdarkly.com/docs/home/agentcontrol . The code around the agent, including the limit its validator enforces, sits behind a boolean flag. Either one changes what the running band does on its next generation, and neither needs a restart. The parser only asks whether a pattern is well-formed, which is a narrower question than whether the music got better. An agent asked to make its part more interesting can comply by turning itself up. Past the point where the speakers reproduce the mix cleanly the sound stops improving, and every check still passes and reports another good version. That is why a variation carries a gain ceiling, and why a metric https://launchdarkly.com/docs/home/metrics reads the peak gain of what actually shipped. This walkthrough ships that limit as a flag, strict-mix-gate , set tighter than the drummer can satisfy at all. Every pattern it generates is rejected, so hold-last-known-good replays the previous bar while the bassist and keys, whose limits tighten only slightly, keep publishing around it. One instrument is dead, the other two carry the mix, and nothing crashes. You will not hear it. The publish rate will. The correction is not to loosen the gate, since nothing yet tells you the gate is the problem. It is to count the attempt and the success separately, so a rejected candidate contributes a zero instead of contributing nothing, and then to guard the change with the metric that can see its failure shape. That is the whole argument in one line: gate each artifact on an invariant you can compute, measure the process for drift, and let a change that moves the process be taken back without a person. Those are three different clocks, and the last one is a release control rather than a gate, which is why it can catch a change that every individual gate approved. A loud regression and a silent one are not visible to the same instrument, so which metric watches is decided per change. A factory gates in three places, and they run on different clocks. Only the first is a runtime gate: | Layer | Clock | Decides | On failure | Here | |---|---|---|---|---| | Parser and maxGain | Every generation | Is this artifact allowed | Reject and retry | Already in the repo | | Hold-last-known-good | Every generation, on rejection | What plays instead | Replay the previous bar | Already in the repo | | Guarded rollout | One monitoring window | Is this change allowed to continue | Revert the change | You configure it in step 4 | Run this from a directory that is not already a Shipping Floor checkout. Cloning inside an existing clone nests a second copy and fails. Here is the clone: git clone https://github.com/launchdarkly-labs/shipping-floor.git cd shipping-floor npm install If you already have the repo, skip the clone and run npm install in that directory. Do not start the app yet. Until LaunchDarkly is serving prompts, there is nothing for it to play. The application already evaluates a boolean flag called strict-mix-gate and already emits the guardrail metrics. Your job is to create the LaunchDarkly resources the seed lists in a new project, not to edit application code, and not to write into an existing Shipping Floor project. Using the shared demo is not a new implementation. https://mcp.launchdarkly.com/mcp/launchdarkly under the MCP settings or use the claude mcp add configures Claude Code only. list-projects . This real API call is the authentication check. If it returns token expired even after an auth helper reported success, remove the MCP server, add it again, and finish the browser consent flow once more. Here is the equivalent command for Claude Code: claude mcp add --transport http "launchdarkly" \ "https://mcp.launchdarkly.com/mcp/launchdarkly" Cursor may hold production MCP writes behind an approval card. If targeting, metrics, or the rollout appears to stop, approve the card and let the assistant retry. These writes stand up the project; they are not a customer rollout. Connect the application to the new project before bootstrap, so the assistant does not default to an existing shipping-floor project: cp .env.example .env Set these three values in .env : LAUNCHDARKLY PROJECT KEY=<new-project-key LAUNCHDARKLY SDK KEY=<server-side-sdk-key ANTHROPIC API KEY=<anthropic-api-key Print the resource specification: npm run seed Run /factory-bootstrap in your assistant the command file is .claude/commands/factory-bootstrap.md and give it the project key from .env . Before it pauses, the command: seed/ strict-mix-gate , turns it false request is not yet available for experiments Check the two project lines in the seed output. seed default project: shipping-floor names the checked-in example; this run targets: must show the new project key from .env . Do not create the context kinds first Add kind is restricted to admin users, and a new project does not grant that permission. The Contexts list https://launchdarkly.com/docs/home/flags/contexts-list shows instances; it does not create kinds. Let SDK evaluations create the kinds instead. When /factory-bootstrap pauses before metrics, complete these steps: npm run seed -- --verify . The evaluations create the musician , performance , listener , and peak-gain , ceiling-breach-rate , and publish-success-rate with randomizationUnits: "request" . The order matters. Creating a metric before request is available fails with Randomization unit "request" not found . Omitting randomizationUnits avoids that error but silently defaults the metric to user , and MCP has no update-metric tool to repair it. The context kind API https://launchdarkly.com/docs/api/contexts/put-context-kind can set the experiments checkbox, but the hosted MCP server has no context-kind tool. That is why this walkthrough uses one UI edit after the SDK creates the kinds. Run the verification command again after bootstrap creates the metrics: npm run seed -- --verify This evaluates every config through the SDK and checks that strict-mix-gate is serving false . The command has to reach stream.launchdarkly.com . A run without network access, or an offline run, fails on DNS. Do not start the application until this exits 0. Start the application: npm start Open http://localhost:3000 and press Play . Audio does not start until then. Three agents start trading patterns, each one regenerating every few bars. npm start runs a preflight check first and refuses to start if LaunchDarkly is not delivering usable configs. The heads-up display HUD has two kinds of activity on one screen: the model calls that keep the band playing, and the release work that changes how those calls are governed. Here is the running band after a few minutes of Play: An earlier capture of the Shipping Floor HUD while the band plays. Generate is 279 and Measure is 6; Classify, Flag, Release, and Clean up stay at 0. The current HUD, described below, leads with publish rate and the ceiling actually enforced. The HUD has two clocks on one screen. The production line and the counters are the per-generation factory: generate, gate, deploy, hold last good. The Control Tower's strict-mix-gate row and the LaunchDarkly rollout are the per-change factory: a flag arm, a metric verdict, a revert. Only the first clock is a gate. Read the current layout in three passes. Generate is the count of model calls since the server started. Retries count as additional calls, so this is not a song count or a count of completed bars. The dot turns green when a candidate ships and flashes an alarm color when one is rejected. The labels after the seam describe the release side of the factory; they are not stages that every candidate passes through. This demo updates two totals in the line: Generate counts model calls, and Measure counts ceiling breaches. Classify , Flag , Release , and Clean up remain at zero because the browser does not receive activity from those assistant-driven workflows. A published increment in the ledger is one musician shipping one bar, not one pass through that six-node line. Serving names the config, variation, model, evaluation contexts, and the strict-mix-gate arm this musician last ran under false , or true · strict . An empty variation key is the alarm: LaunchDarkly served the SDK default and is not driving this musician. Prompt composition names the three pinned snippet versions. The HUD never renders the assembled prompt text. Guardrails lead with two rates, in the same direction LaunchDarkly reads them: publish rate higher is better and breach rate lower is better . Under those, peak gain / enforced ceiling is the loudest .gain in the current accepted pattern against the ceiling the validator is actually using. When the gate is off, that is the variation's gain ceiling . When it is on, it is strict gain ceiling . Showing the nominal ceiling here used to make a legitimate rejection look like a HUD bug. Published · held is how many candidates shipped versus how many turns kept the last good bar. Breaches is how many candidates exceeded the enforced ceiling since this process started. The footer is band-wide and leads with publish rate , then generated, shipped, rejected, and held. The tower is one musician; this line is all three. A live capture can be one call ahead of its results while a request is in flight. Served split summarizes the AgentControl variations behind recent published patterns. It is observed traffic, not configured rollout progress, and it does not show the strict-mix-gate flag arms. Let the band run until the healthy pattern is familiar. Generated and shipped keep climbing, the code in the rack changes, peak gain stays below the enforced ceiling, and held moves rarely. In step 4, requests assigned to the stricter gate fail more often: publish rate falls, held rises, and the drummer repeats its last good bar more often. The music may hide an occasional repeat; the counters do not. The flag is already in the code and already in your project. You are going to roll true out against false . When true is served, the validator uses strict gain ceiling 0.60 on the drummer instead of gain ceiling 1.15 . Any drum candidate above 0.60 now fails. When every retry for a regeneration fails, hold-last-good takes over and the drummer repeats the previous bar. Nothing throws. Start the guarded rollout https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts first, then generate traffic. A burn-in that finishes before the rollout is live does not feed it. If you are starting the rollout through MCP, start-guarded-rollout requires regressionThreshold in its schema, and the API rejects a real threshold. Send regressionThreshold: 0 . From the strict-mix-gate flag's Targeting tab, on the rule already serving false : | Setting | Value | Why | |---|---|---| | Randomization unit | request | One coin flip per generation, so the rollout is audible within a single session | | Guardrail metric | publish-success-rate | The failure you are watching for is output that stops arriving | | On regression | Roll back automatically | No human in the loop | | First stage | 5% | A real failure reaches little traffic before it reverts | Then start the load generator: npm run burn-in -- --state section=lift,energy=high,isBoundary=false --generations 500 Keep the browser tab open. Requests assigned true use the tighter limit. The drummer's strict ceiling is 0.6 against a kick range its own prompt puts at 1.0 to 1.15, so it cannot comply and stops publishing entirely; the bassist and keys tighten only slightly and keep regenerating. Nothing fails loudly. The drummer's held count climbs, the band-level publish rate falls partway because two thirds of the work still lands, and if you had walked in partway through you might not hear that anything was wrong. The failure is not silent in LaunchDarkly. The Targeting tab shows the rollout status, and the Monitoring tab next to it has a tile for publish-success-rate . When LaunchDarkly ends the rollout, the banner reads Default rule rolled back automatically after detecting a regression for Publish success rate . The guarded rollout summary after automatic rollback. Publish success rate dropped 33.5 pp. The default rule is serving false again. Read this panel as a comparison between two policies, not as a second copy of the HUD. The true arm received 23 unique request contexts while the rollout was at 10% traffic. LaunchDarkly estimated that 52% of requests in that arm published a pattern, compared with 86% for the original false arm. Because publish success is a higher-is-better metric, the −33.5 pp difference is a regression: the stricter gate is stopping new work from reaching the player. Rolled back automatically, now serving false is the action LaunchDarkly took in response. The HUD and the rollout tile answer related questions at different levels. The HUD's publish rate asks, “Of this musician's generations since the process started, how often did a pattern ship?” The rollout metric asks, “For the request contexts assigned to each arm, how often did a generation publish at all?” That is why the tile can compare true with false even though the HUD shows one process-lifetime rate. Timing matters here. The first 500-generation run finished before the rollout was live, so its 100% publish rate and zero breaches never contributed to the comparison. The second run happened under the active rollout and exposed the gate: | Variation | Published | Ceiling breaches | Held | |---|---|---|---| | breakbeat-chopping drummer | 93.4% | 22 | 11 | | rolling-synth-bass bassist | 100% | 11 | Not reported | | stab-chords keys | 98.8% | 8 | Not reported | The drummer is the musician the tight ceiling hits. Held 11 is hold-last-good: those bars never reached the speakers, which is what dropped publish-success-rate . Without anyone touching the flag, the drummer starts varying again. The default rule is serving false . Open change history https://launchdarkly.com/docs/home/releases/change-history for the timestamps and the actor: 21:18:01 Scarlett Attensil Starting guarded release: false 5% and true 5% 21:19:08 via API member: null Guarded release advanced to the next stage: false 10% and true 10% 21:19:58 via API member: null Reverted the guarded release on the default rule and is now serving false member: null is the line worth keeping. No person advanced or reverted that rollout. In the run this tutorial draws on, it started at 5%, advanced to 10%, and reverted at 2:19 PM, 117 seconds after it began. Detection under sequential testing varies by run, so do not plan around a duration. You can stop here. What follows is what to copy if you are adapting your own pipeline. The three layers from the opening table, in the detail you need to rebuild them. The design gets clearer once you stop calling all three of them gates. Per artifact, inline. The parser and maxGain in src/validate.js judge one pattern before it reaches the speakers. This is the only true runtime gate. It runs on every generation, it decides in microseconds, and it publishes or rejects. It can enforce any invariant you can compute from the artifact, and nothing beyond that. A gate that has to call a model to reach its verdict is not a gate, it is another generator. Per artifact, on rejection. Hold-last-known-good decides what plays when the gate says no. This is the layer that turns a hard failure into a quiet one, so it carries its own counters: attempts, holds, and the age of what you are still serving. Per change, over a window. The guarded rollout is not a runtime gate. It never inspects a pattern and it cannot stop one from playing. It decides how much traffic a change is allowed while a metric gathers enough evidence to judge it, then keeps the change or takes it back. Its clock is the monitoring window, not the generation. Conflating the third with the first two is the common mistake. A gate answers whether this artifact is allowed. A guarded rollout answers whether this change is allowed to keep going. No amount of tightening the first produces the second, which is why a factory needs both. Three rules follow. Gate on invariants, measure for drift. A per-artifact verdict cannot express a distribution, and a distribution cannot stop one bad artifact. The gain ceiling is the invariant; peak-gain and publish-success-rate are the drift. Count every fallback. A hold that increments no counter is a silent failure by construction. Absence has to be recordable, which is why publish-success-rate fires on the publish rather than on the attempt: a generation that never publishes contributes a zero instead of contributing nothing. Put every gate behind a flag. strict-mix-gate is a flag, not a constant, so a limit that turns out to be wrong is a targeting change rather than a release. A gate nobody can loosen in seconds is a gate somebody eventually disables for good. Three implementation notes carry the rest. The prompt is pinned snippets. A variation's system message is three prompt snippets https://launchdarkly.com/docs/home/agentcontrol/snippets by key and version. The limits snippet sits above the vocabulary snippet so the model reads the invariant first. Open src/config/launchdarkly.js to find the resolve and the length check that catches a snippet reference that rendered to nothing. The unit is one generation. The application builds a multi-context https://launchdarkly.com/docs/home/flags/multi-contexts per generation. The request kind is the randomization unit. A single static context gives a rollout a sample size of one forever, and LaunchDarkly reverts it for failing the minimum-context requirement. A flag and a config variation are the same shape at this layer, one evaluation per generation, which is why the same rollout mechanism reverses either one. The metric comes from the artifact. src/validate.js reads peak gain from the pattern string. It does not call a model or inspect audio, which is why the headless load generator and the live application agree. publish-success-rate fires when a regeneration publishes a pattern; a held result emits no success event, so the rate falls when the gate blocks new work. Swap the artifact and the rule, and the same shape sits in front of generated SQL, generated JSON, or generated code. A moderation classifier whose auto-approve threshold rises is the same failure: nothing errors, every item routes to human review, and only the publish rate shows it. A finished run side has five requirements. This walkthrough proved two of them and started a third. The remaining two are named so you know what you did not do. | Requirement | In this walkthrough | |---|---| | Know what shipped | Proved. The identity of the failure is the strict-mix-gate flag serving true , not a deploy job. The Control Tower names the config, variation, pinned snippets, model, gate arm, and enforced ceiling on every bar, including rejections. | | Keep a record that outlasts the change | Started. Change history https://launchdarkly.com/docs/home/releases/change-history records the rollout and the revert, including that no person triggered it. Long-term export is out of scope. | | Decide exposure by policy, not by a person | Skipped. You filled in the rollout dialog yourself. A release policy https://launchdarkly.com/docs/home/releases/release-policies is how that default attaches to every release later. | | Reverse one change, not the deploy | Proved. strict-mix-gate went back. The rest of the band kept playing. | | Clean up what the change left behind | Skipped. The flag and snippets stay. Read flag cleanup https://launchdarkly.com/docs/home/flags/manage/flag-cleanup-vega before you let agents create the next dozen flags. | This walkthrough only shows a change being taken back, because that is the part that leaves evidence. You do not hear the freeze. You see the metric tile fall and the default rule go back on its own. The rails do not know the difference. A candidate that stays inside its ceiling advances through the same stages, completes its last one, and becomes the new baseline. That symmetry is the point. A run side trusted only to block changes is a run side people route around.