{"slug": "finish-your-software-factory-take-a-bad-change-back-before-anyone-notices", "title": "Finish your software factory: take a bad change back before anyone notices", "summary": "A developer has released Shipping Floor, an open-source example repo that demonstrates an agentic software factory capable of automatically rolling back a bad change without human intervention. The project uses three AI agents that generate electronic music in the browser, with LaunchDarkly guarded rollouts and feature flags controlling agent behavior and validation limits at runtime. The walkthrough shows how a deliberately tightened gain gate causes one agent's output to be rejected while the others continue, illustrating bounded blast radius and automated reversal.", "body_md": "Most agentic factory stacks stop at the deploy. They generate, they validate, they ship, and then the instructions turn into a vague \"monitor\" step, which in practice means a person watching a dashboard. The capability that is actually missing is narrower than monitoring and harder to build: a change that moves the process gets taken back without a person.\n\nThat is a throughput problem before it is a tooling problem. When a person wrote every change, review was the quality system, and a few changes a day could each be read before they shipped. A factory whose agents deploy continuously has no room for that, and the failures it produces are the kind review misses anyway. One of the two failures here passes every check the build side runs. The other one *is* the checks, all of them firing correctly while the pipeline quietly stops producing.\n\nSo a factory without this is not slower than one with it. It is less certain. It ships improvements and degradations at the same rate, reports success both times, and leaves you one lever: a deploy-level rollback that takes back everything else along with the change you wanted gone. What you build here instead is a bounded blast radius while a change proves itself, a verdict that does not depend on who is paying attention, and a reversal that touches one change and leaves the rest of the band playing.\n\nThe example repo builds an agentic band. Its loop generates, validates, and deploys about eleven times a minute. In this walkthrough, you start the music factory, listen to it, break it on purpose, and watch a [guarded rollout](https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts) put it back.\n\nThree agents make up your virtual band: a drummer, a bassist, and a keys player. They generate electronic music in the browser while a conductor keeps tempo and key. Every few bars each agent asks an AI model for the next version of its part. A parser accepts or rejects the result. If the new pattern is well-formed, the next cycle plays it. If it is not, the last good pattern keeps looping.\n\n[Shipping Floor](https://github.com/launchdarkly-labs/shipping-floor) contains no musical vocabulary of its own, so personas, gain limits, and groove recipes all come from LaunchDarkly [prompt snippets](https://launchdarkly.com/docs/home/agentcontrol/snippets) at runtime. If LaunchDarkly is unreachable, the band holds and then falls silent. There is no local fallback for this demo.\n\nTwo things about the band can change without a deploy, and each uses a different LaunchDarkly object. What an agent *is*, its model, its parameters, and a system message assembled from pinned prompt snippets, is a variation in an [AgentControl config](https://launchdarkly.com/docs/home/agentcontrol). The code *around* the agent, including the limit its validator enforces, sits behind a boolean flag. Either one changes what the running band does on its next generation, and neither needs a restart.\n\nThe parser only asks whether a pattern is well-formed, which is a narrower question than whether the music got better. An agent asked to make its part more interesting can comply by turning itself up. Past the point where the speakers reproduce the mix cleanly the sound stops improving, and every check still passes and reports another good version. That is why a variation carries a gain ceiling, and why a [metric](https://launchdarkly.com/docs/home/metrics) reads the peak gain of what actually shipped.\n\nThis walkthrough ships that limit as a flag, `strict-mix-gate`, set tighter than the drummer can satisfy at all. Every pattern it generates is rejected, so hold-last-known-good replays the previous bar while the bassist and keys, whose limits tighten only slightly, keep publishing around it. One instrument is dead, the other two carry the mix, and nothing crashes. You will not hear it. The publish rate will.\n\nThe correction is not to loosen the gate, since nothing yet tells you the gate is the problem. It is to count the attempt and the success separately, so a rejected candidate contributes a zero instead of contributing nothing, and then to guard the change with the metric that can see its failure shape.\n\nThat is the whole argument in one line: gate each artifact on an invariant you can compute, measure the process for drift, and let a change that moves the process be taken back without a person. Those are three different clocks, and the last one is a release control rather than a gate, which is why it can catch a change that every individual gate approved. A loud regression and a silent one are not visible to the same instrument, so which metric watches is decided per change.\n\nA factory gates in three places, and they run on different clocks. Only the first is a runtime gate:\n\n| Layer | Clock | Decides | On failure | Here | \n|---|---|---|---|---|\n| Parser and `maxGain()` | Every generation | Is this artifact allowed | Reject and retry | Already in the repo | \n| Hold-last-known-good | Every generation, on rejection | What plays instead | Replay the previous bar | Already in the repo | \n| Guarded rollout | One monitoring window | Is this change allowed to continue | Revert the change | You configure it in step 4 | \n\nRun this from a directory that is not already a Shipping Floor checkout. Cloning inside an existing clone nests a second copy and fails.\n\nHere is the clone:\n\n```\ngit clone https://github.com/launchdarkly-labs/shipping-floor.git\ncd shipping-floor\nnpm install\n```\n\nIf you already have the repo, skip the clone and run `npm install` in that directory. Do not start the app yet. Until LaunchDarkly is serving prompts, there is nothing for it to play.\n\nThe application already evaluates a boolean flag called `strict-mix-gate` and already emits the guardrail metrics. Your job is to create the LaunchDarkly resources the seed lists in a **new** project, not to edit application code, and not to write into an existing Shipping Floor project. Using the shared demo is not a new implementation.\n\n`https://mcp.launchdarkly.com/mcp/launchdarkly` under the MCP settings (or use the `claude mcp add` configures Claude Code only.`list-projects`. This real API call is the authentication check. If it returns `token_expired` even after an auth helper reported success, remove the MCP server, add it again, and finish the browser consent flow once more.\nHere is the equivalent command for Claude Code:\n\n```\nclaude mcp add --transport http \"launchdarkly\" \\\n    \"https://mcp.launchdarkly.com/mcp/launchdarkly\"\n```\n\nCursor may hold production MCP writes behind an approval card. If targeting, metrics, or the rollout appears to stop, approve the card and let the assistant retry. These writes stand up the project; they are not a customer rollout.\n\nConnect the application to the new project before bootstrap, so the assistant does not default to an existing `shipping-floor` project:\n\n```\ncp .env.example .env\n```\n\nSet these three values in `.env`:\n\n```\nLAUNCHDARKLY_PROJECT_KEY=<new-project-key>\nLAUNCHDARKLY_SDK_KEY=<server-side-sdk-key>\nANTHROPIC_API_KEY=<anthropic-api-key>\n```\n\nPrint the resource specification:\n\n```\nnpm run seed\n```\n\nRun `/factory-bootstrap` in your assistant (the command file is `.claude/commands/factory-bootstrap.md`) and give it the project key from `.env`. Before it pauses, the command:\n\n`seed/`\n`strict-mix-gate`, turns it `false`\n`request` is not yet available for experiments\nCheck the two project lines in the seed output. `seed default project: shipping-floor` names the checked-in example; `this run targets:` must show the new project key from `.env`.\n\n**Do not create the context kinds first**\n\n**Add kind** is restricted to admin users, and a new project does not grant that permission. The [Contexts list](https://launchdarkly.com/docs/home/flags/contexts-list) shows instances; it does not create kinds. Let SDK evaluations create the kinds instead.\n\nWhen `/factory-bootstrap` pauses before metrics, complete these steps:\n\n`npm run seed -- --verify`. The evaluations create the `musician`, `performance`, `listener`, and `peak-gain`, `ceiling-breach-rate`, and `publish-success-rate` with `randomizationUnits: [\"request\"]`.\nThe order matters. Creating a metric before `request` is available fails with `Randomization unit \"request\" not found`. Omitting `randomizationUnits` avoids that error but silently defaults the metric to `user`, and MCP has no `update-metric` tool to repair it.\n\nThe [context kind API](https://launchdarkly.com/docs/api/contexts/put-context-kind) can set the experiments checkbox, but the hosted MCP server has no context-kind tool. That is why this walkthrough uses one UI edit after the SDK creates the kinds.\n\nRun the verification command again after bootstrap creates the metrics:\n\n```\nnpm run seed -- --verify\n```\n\nThis evaluates every config through the SDK and checks that `strict-mix-gate` is serving `false`. The command has to reach `stream.launchdarkly.com`. A run without network access, or an offline run, fails on DNS. Do not start the application until this exits 0.\n\nStart the application:\n\n```\nnpm start\n```\n\nOpen `http://localhost:3000` and press **Play**. Audio does not start until then. Three agents start trading patterns, each one regenerating every few bars. `npm start` runs a preflight check first and refuses to start if LaunchDarkly is not delivering usable configs.\n\nThe heads-up display (HUD) has two kinds of activity on one screen: the model calls that keep the band playing, and the release work that changes how those calls are governed. Here is the running band after a few minutes of Play:\n\n*An earlier capture of the Shipping Floor HUD while the band plays. Generate is 279 and Measure is 6; Classify, Flag, Release, and Clean up stay at 0. The current HUD, described below, leads with publish rate and the ceiling actually enforced.*\n\nThe HUD has two clocks on one screen. The production line and the counters are the per-generation factory: generate, gate, deploy, hold last good. The Control Tower's **strict-mix-gate** row and the LaunchDarkly rollout are the per-change factory: a flag arm, a metric verdict, a revert. Only the first clock is a gate. Read the current layout in three passes.\n\n**Generate** is the count of model calls since the server started. Retries count as additional calls, so this is not a song count or a count of completed bars. The dot turns green when a candidate ships and flashes an alarm color when one is rejected.\n\nThe labels after the seam describe the release side of the factory; they are not stages that every candidate passes through. This demo updates two totals in the line: **Generate** counts model calls, and **Measure** counts ceiling breaches. **Classify**, **Flag**, **Release**, and **Clean up** remain at zero because the browser does not receive activity from those assistant-driven workflows. A **published** increment in the ledger is one musician shipping one bar, not one pass through that six-node line.\n\n**Serving** names the config, variation, model, evaluation contexts, and the `strict-mix-gate` arm this musician last ran under (`false`, or `true · strict`). An empty variation key is the alarm: LaunchDarkly served the SDK default and is not driving this musician.\n\n**Prompt composition** names the three pinned snippet versions. The HUD never renders the assembled prompt text.\n\n**Guardrails** lead with two rates, in the same direction LaunchDarkly reads them: **publish rate** (higher is better) and **breach rate** (lower is better). Under those, **peak gain / enforced ceiling** is the loudest `.gain()` in the current accepted pattern against the ceiling the validator is actually using. When the gate is off, that is the variation's `gain_ceiling`. When it is on, it is `strict_gain_ceiling`. Showing the nominal ceiling here used to make a legitimate rejection look like a HUD bug. **Published · held** is how many candidates shipped versus how many turns kept the last good bar. **Breaches** is how many candidates exceeded the enforced ceiling since this process started.\n\nThe footer is band-wide and leads with **publish rate**, then generated, shipped, rejected, and held. The tower is one musician; this line is all three. A live capture can be one call ahead of its results while a request is in flight.\n\n**Served split** summarizes the AgentControl variations behind recent published patterns. It is observed traffic, not configured rollout progress, and it does not show the `strict-mix-gate` flag arms.\n\nLet the band run until the healthy pattern is familiar. Generated and shipped keep climbing, the code in the rack changes, peak gain stays below the **enforced** ceiling, and held moves rarely. In step 4, requests assigned to the stricter gate fail more often: publish rate falls, held rises, and the drummer repeats its last good bar more often. The music may hide an occasional repeat; the counters do not.\n\nThe flag is already in the code and already in your project. You are going to roll `true` out against `false`. When `true` is served, the validator uses `strict_gain_ceiling` (0.60 on the drummer) instead of `gain_ceiling` (1.15). Any drum candidate above 0.60 now fails. When every retry for a regeneration fails, hold-last-good takes over and the drummer repeats the previous bar. Nothing throws.\n\nStart the [guarded rollout](https://launchdarkly.com/docs/home/releases/creating-guarded-rollouts) first, then generate traffic. A burn-in that finishes before the rollout is live does not feed it. If you are starting the rollout through MCP, `start-guarded-rollout` requires `regressionThreshold` in its schema, and the API rejects a real threshold. Send `regressionThreshold: 0`.\n\nFrom the `strict-mix-gate` flag's **Targeting** tab, on the rule already serving `false`:\n\n| Setting | Value | Why | \n|---|---|---|\n| Randomization unit | `request` | One coin flip per generation, so the rollout is audible within a single session | \n| Guardrail metric | `publish-success-rate` | The failure you are watching for is output that stops arriving | \n| On regression | Roll back automatically | No human in the loop | \n| First stage | 5% | A real failure reaches little traffic before it reverts | \n\nThen start the load generator:\n\n```\nnpm run burn-in -- --state section=lift,energy=high,isBoundary=false --generations 500\n```\n\nKeep the browser tab open. Requests assigned `true` use the tighter limit. The drummer's strict ceiling is 0.6 against a kick range its own prompt puts at 1.0 to 1.15, so it cannot comply and stops publishing entirely; the bassist and keys tighten only slightly and keep regenerating. Nothing fails loudly. The drummer's held count climbs, the band-level publish rate falls partway because two thirds of the work still lands, and if you had walked in partway through you might not hear that anything was wrong.\n\nThe failure is not silent in LaunchDarkly. The **Targeting** tab shows the rollout status, and the **Monitoring** tab next to it has a tile for `publish-success-rate`. When LaunchDarkly ends the rollout, the banner reads **Default rule rolled back automatically after detecting a regression for Publish success rate**.\n\n*The guarded rollout summary after automatic rollback. Publish success rate dropped 33.5 pp. The default rule is serving false again.*\n\nRead this panel as a comparison between two policies, not as a second copy of the HUD. The `true` arm received 23 unique `request` contexts while the rollout was at 10% traffic. LaunchDarkly estimated that 52% of requests in that arm published a pattern, compared with 86% for the original `false` arm. Because publish success is a higher-is-better metric, the −33.5 pp difference is a regression: the stricter gate is stopping new work from reaching the player. **Rolled back automatically, now serving false** is the action LaunchDarkly took in response.\n\nThe HUD and the rollout tile answer related questions at different levels. The HUD's publish rate asks, “Of this musician's generations since the process started, how often did a pattern ship?” The rollout metric asks, “For the request contexts assigned to each arm, how often did a generation publish at all?” That is why the tile can compare `true` with `false` even though the HUD shows one process-lifetime rate.\n\nTiming matters here. The first 500-generation run finished before the rollout was live, so its 100% publish rate and zero breaches never contributed to the comparison. The second run happened under the active rollout and exposed the gate:\n\n| Variation | Published | Ceiling breaches | Held | \n|---|---|---|---|\n| `breakbeat-chopping` (drummer) | 93.4% | 22 | 11 | \n| `rolling-synth-bass` (bassist) | 100% | 11 | Not reported | \n| `stab-chords` (keys) | 98.8% | 8 | Not reported | \n\nThe drummer is the musician the tight ceiling hits. Held 11 is hold-last-good: those bars never reached the speakers, which is what dropped `publish-success-rate`.\n\nWithout anyone touching the flag, the drummer starts varying again. The default rule is serving `false`. Open [change history](https://launchdarkly.com/docs/home/releases/change-history) for the timestamps and the actor:\n\n```\n21:18:01  Scarlett Attensil\n          Starting guarded release: false (5%) and true (5%)\n\n21:19:08  (via API)   member: null\n          Guarded release advanced to the next stage: false (10%) and true (10%)\n\n21:19:58  (via API)   member: null\n          Reverted the guarded release on the default rule\n          and is now serving `false`\n```\n\n`member: null` is the line worth keeping. No person advanced or reverted that rollout. In the run this tutorial draws on, it started at 5%, advanced to 10%, and reverted at 2:19 PM, 117 seconds after it began. Detection under sequential testing varies by run, so do not plan around a duration.\n\nYou can stop here. What follows is what to copy if you are adapting your own pipeline.\n\nThe three layers from the opening table, in the detail you need to rebuild them. The design gets clearer once you stop calling all three of them gates.\n\n**Per artifact, inline.** The parser and `maxGain()` in `src/validate.js` judge one pattern before it reaches the speakers. This is the only true runtime gate. It runs on every generation, it decides in microseconds, and it publishes or rejects. It can enforce any invariant you can compute from the artifact, and nothing beyond that. A gate that has to call a model to reach its verdict is not a gate, it is another generator.\n\n**Per artifact, on rejection.** Hold-last-known-good decides what plays when the gate says no. This is the layer that turns a hard failure into a quiet one, so it carries its own counters: attempts, holds, and the age of what you are still serving.\n\n**Per change, over a window.** The guarded rollout is not a runtime gate. It never inspects a pattern and it cannot stop one from playing. It decides how much traffic a *change* is allowed while a metric gathers enough evidence to judge it, then keeps the change or takes it back. Its clock is the monitoring window, not the generation.\n\nConflating the third with the first two is the common mistake. A gate answers whether this artifact is allowed. A guarded rollout answers whether this change is allowed to keep going. No amount of tightening the first produces the second, which is why a factory needs both. Three rules follow.\n\n**Gate on invariants, measure for drift.** A per-artifact verdict cannot express a distribution, and a distribution cannot stop one bad artifact. The gain ceiling is the invariant; `peak-gain` and `publish-success-rate` are the drift.\n\n**Count every fallback.** A hold that increments no counter is a silent failure by construction. Absence has to be recordable, which is why `publish-success-rate` fires on the publish rather than on the attempt: a generation that never publishes contributes a zero instead of contributing nothing.\n\n**Put every gate behind a flag.** `strict-mix-gate` is a flag, not a constant, so a limit that turns out to be wrong is a targeting change rather than a release. A gate nobody can loosen in seconds is a gate somebody eventually disables for good.\n\nThree implementation notes carry the rest.\n\n**The prompt is pinned snippets.** A variation's system message is three [prompt snippets](https://launchdarkly.com/docs/home/agentcontrol/snippets) by key and version. The limits snippet sits above the vocabulary snippet so the model reads the invariant first. Open `src/config/launchdarkly.js` to find the resolve and the length check that catches a snippet reference that rendered to nothing.\n\n**The unit is one generation.** The application builds a [multi-context](https://launchdarkly.com/docs/home/flags/multi-contexts) per generation. The `request` kind is the randomization unit. A single static context gives a rollout a sample size of one forever, and LaunchDarkly reverts it for failing the minimum-context requirement. A flag and a config variation are the same shape at this layer, one evaluation per generation, which is why the same rollout mechanism reverses either one.\n\n**The metric comes from the artifact.** `src/validate.js` reads peak gain from the pattern string. It does not call a model or inspect audio, which is why the headless load generator and the live application agree. `publish-success-rate` fires when a regeneration publishes a pattern; a held result emits no success event, so the rate falls when the gate blocks new work. Swap the artifact and the rule, and the same shape sits in front of generated SQL, generated JSON, or generated code. A moderation classifier whose auto-approve threshold rises is the same failure: nothing errors, every item routes to human review, and only the publish rate shows it.\n\nA finished run side has five requirements. This walkthrough proved two of them and started a third. The remaining two are named so you know what you did not do.\n\n| Requirement | In this walkthrough | \n|---|---|\n| Know what shipped | Proved. The identity of the failure is the `strict-mix-gate` flag serving`true` , not a deploy job. The Control Tower names the config, variation, pinned snippets, model, gate arm, and enforced ceiling on every bar, including rejections. | \n| Keep a record that outlasts the change | Started. [Change history](https://launchdarkly.com/docs/home/releases/change-history) records the rollout and the revert, including that no person triggered it. Long-term export is out of scope. | \n| Decide exposure by policy, not by a person | Skipped. You filled in the rollout dialog yourself. A [release policy](https://launchdarkly.com/docs/home/releases/release-policies) is how that default attaches to every release later. | \n| Reverse one change, not the deploy | Proved. `strict-mix-gate` went back. The rest of the band kept playing. | \n| Clean up what the change left behind | Skipped. The flag and snippets stay. Read [flag cleanup](https://launchdarkly.com/docs/home/flags/manage/flag-cleanup-vega) before you let agents create the next dozen flags. | \n\nThis walkthrough only shows a change being taken back, because that is the part that leaves evidence. You do not hear the freeze. You see the metric tile fall and the default rule go back on its own. The rails do not know the difference. A candidate that stays inside its ceiling advances through the same stages, completes its last one, and becomes the new baseline. That symmetry is the point. A run side trusted only to block changes is a run side people route around.", "url": "https://wpnews.pro/news/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices", "canonical_source": "https://dev.to/launchdarkly/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices-8dn", "published_at": "2026-09-16 16:48:26+00:00", "updated_at": "2026-09-16 17:13:19.514423+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "generative-ai", "mlops"], "entities": ["LaunchDarkly", "Shipping Floor", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices", "markdown": "https://wpnews.pro/news/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices.md", "text": "https://wpnews.pro/news/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices.txt", "jsonld": "https://wpnews.pro/news/finish-your-software-factory-take-a-bad-change-back-before-anyone-notices.jsonld"}}