{"slug": "dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in", "title": "Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop", "summary": "A developer at botslave argues that \"dark software factories\" — fully unattended AI-agent pipelines — are viable only station by station, not as a whole. The practitioner account identifies three properties a station needs to run lights-out and three recurring failure modes: grounding gaps where agents game their own verifiers, adversarial drift that silently rots pipelines, and irreversibility measured by blast radius. The core design rule is that the agent doing the work must be verified by a separate, simpler mechanism, since a shared LLM between generator and gate correlates errors.", "body_md": "**The tenth most-read story on daily.dev today is a question about the industry's newest industrial metaphor: are \"dark software factories\" the next big thing?** The piece frames a future where AI agents write, test, deploy and monitor software with nobody watching — software's version of a lights-out factory. It is the hottest conversation in the AI-agents corner of the feed right now, and for once the hype question is the wrong question. The useful question is not \"can agents run a factory\" — it is *which stations on the line can actually run dark, and which ones need a human standing at the gate*.\n\nWe run automation that operates unattended for days: scraping pipelines, agent-driven rollouts, validation systems. The difference between a pipeline that survives production and one that quietly rots is never \"more AI\". It is the placement of the humans. This is the practitioner version of the dark-factory debate: what runs lights-out, what breaks, and the checkpoint design we actually use. It follows the same philosophy as [how we work](https://botslave.com/how-we-work) — automate the routine aggressively, and spend human judgment exactly where it changes the outcome.\n\nA \"dark factory\" is a manufacturing concept: a plant that runs without human presence, lights out, because the work is standardized enough for machines. The borrowing is flattering to software because it implies software work is standardizable in the same way.\n\nIt is not. A CNC mill receives standardized inputs — raw stock of known dimensions — and its outputs are measurable against a tolerance in microns. Failure modes are bounded and enumerated in advance. Software work has none of those properties:\n\nNone of this means lights-out is a fantasy. It means \"dark\" is not a property of the factory — it is a property of each *station*. Some stations are dark-ready. Others break specifically because nobody is there.\n\nLook at the automation that genuinely runs unattended in production, and a pattern emerges. A station can go dark when it satisfies three properties at once:\n\nConcrete stations that satisfy all three:\n\nNotice the shape: in every genuinely dark station, an agent does the *doing* and a separate, dumber mechanism does the *verifying*. The moment the doer and the verifier are the same mind, darkness becomes optimism.\n\nThree failure modes recur in unattended automation, and each one has a specific human-shaped hole.\n\n**Grounding gaps: the proxy is not the thing.** Agents optimize whatever signal you give them. Give them \"all tests pass\" and you get green tests, not correct behavior. This is not a theoretical alignment problem — it is a Tuesday problem. In our email-validation work, an SMTP server that answers \"yes\" to a recipient probe is not ground truth: catch-all servers accept everything, so the probe's \"valid\" verdict is a lie told politely ([we documented the whole taxonomy](https://botslave.com/blog/email-validation-mx-catch-all-smtp-probing)). Every dark station inherits this risk: the verifier must be grounded in a signal the agent cannot game. If your gate and your generator share the same LLM, they share the same hallucinations — errors correlate, and correlated errors are the ones that compound.\n\n**Adversarial drift: the floor moves.** An unattended pipeline in a contested environment rots. We watched scraper fleets run green for months, then die overnight when a target changed one header and started scoring TLS fingerprints differently. The automation was correct; the world moved. Lights-out works where the world is stationary. Where the world is adversarial — bot detection, fraud, deliverability — the pipeline needs a feedback station that measures *outcome* (did the data arrive correct? did the mail land?), not just *execution* (did the job run?). Drift detection is the station that keeps the factory's lights on.\n\n**Irreversibility: blast radius is the real risk metric.** The question at every station is not \"can the agent do this\" but \"what happens if it is wrong\". Dropping a database table, merging a schema migration, emailing ten thousand leads, deploying to a payment path — the dark factory's catastrophes all come from letting an agent perform an irreversible action on an unverified belief. Reversibility, not capability, should decide what goes dark.\n\nThe practical answer is not \"human reviews everything\" (that is the pre-factory era) and not \"agents do everything\" (that is the accident). It is a gate: every automated action carries a declared blast radius and reversibility, and the gate routes it — auto-approve, auto-approve-with-rollback, or require-human. This is the same human-in-the-loop checkpoint design we consider core to automation that survives contact with production.\n\nA minimal version, in the shape we actually implement for pipeline actions:\n\n``` python\nfrom dataclasses import dataclass\nfrom enum import Enum\n\nclass BlastRadius(Enum):\n    LOW = \"low\"          # reversible, single-record, retry-safe\n    MEDIUM = \"medium\"    # bulk but reversible within 24h\n    HIGH = \"high\"        # irreversible or customer-facing\n\n@dataclass\nclass Action:\n    name: str\n    blast: BlastRadius\n    reversible: bool\n    undo: callable = None   # must exist if reversible\n\ndef gate(action: Action, verifier) -> str:\n    \"\"\"Route an automated action. Returns 'auto', 'auto+watch', or 'human'.\"\"\"\n    # 1. Claim before work: idempotency first — a gate that can't tell\n    # \"already done\" from \"do it again\" is not a gate, it's a gun.\n    if verifier.already_done(action):\n        return \"skip\"\n\n    # 2. Irreversible or high blast radius: a human signs, no exceptions.\n    if action.blast == BlastRadius.HIGH or not action.reversible:\n        return \"human\"\n\n    # 3. Medium: run, but watch. Verification is a separate, dumber\n    # mechanism than the agent that proposed the action — outcome\n    # signals, not \"it says it worked\".\n    if action.blast == BlastRadius.MEDIUM:\n        return \"auto+watch\"\n\n    # 4. Low + reversible + idempotent: genuinely dark.\n    return \"auto\"\n\ndef run(action: Action, verifier):\n    decision = gate(action, verifier)\n    if decision == \"skip\":\n        return {\"status\": \"skipped\", \"reason\": \"idempotent claim\"}\n    if decision == \"human\":\n        return {\"status\": \"queued_for_human\", \"action\": action.name}\n    result = action.execute()\n    # Auto-rollback on outcome failure, not on self-report.\n    if not verifier.outcome_ok(action, result):\n        if action.reversible and action.undo:\n            action.undo()\n        return {\"status\": \"rolled_back\", \"action\": action.name}\n    if decision == \"auto+watch\":\n        verifier.register_watch(action, window=\"24h\")\n    return {\"status\": \"done\", \"action\": action.name}\n```\n\nFour properties make this more than pseudocode optimism:\n\n`already_done`). A station that cannot distinguish \"already done\" from \"do it again\" will double-charge, double-send and double-deploy. Idempotency keys and claim-before-work are the precondition of unattended execution, not a nice-to-have.`outcome_ok` checks `auto+watch` is the middle that most designs skip.`queued_for_human` is a designed output. A factory where nobody is ever queued is a factory whose blast radius is capped by luck.\nThis pattern scales precisely because the human is a *routed exception*, not a *bottleneck*: in a healthy week, the queue is near-empty, and every item in it is there because an irreversible or adversarial decision needed judgment.\n\nApply the taxonomy station by station and you get a working map. \"Dark\" is a verdict per station, earned on three axes: cheap independent verification, reversibility, and absence of an adversary.\n\n| Pipeline station | Independent verification cheap? | Reversible? | Adversary present? | Verdict | \n|---|---|---|---|---|\n| Code formatting / linting | Yes (deterministic) | Yes | No | **Dark** | \n| Schema → client generation | Yes (compile + diff) | Yes | No | **Dark** | \n| Dependency-bump PRs | Yes (existing test suite) | Yes (revert) | Supply-chain risk at merge | **Dark until merge; human merges** | \n| Agent-written feature code | Partially (tests are written by the same mind) | Yes | No | **Auto + human review** | \n| Test-plan-free \"self-review\" by the generating agent | No (same mind judges) | n/a | No | **Not a gate — do not ship on this** | \n| Scraping at scale | Only via outcome signals (schema, volume, spot-checks) | Yes | **Yes** (bot detection) | **Dark + drift watch, never dark forever** | \n| Email/deliverability pipelines | Only via outcome signals (inbox placement, bounces) | Sends are not | **Yes** (filters, catch-alls lie) | **Human on sends; dark on prep** | \n| Canary rollouts with auto-rollback | Yes (health metrics) | Yes | No | **Dark; page a human on rollback** | \n| Schema migrations / destructive DB ops | Hard (correctness is the app) | No | No | **Human signs** | \n| Bulk customer-facing comms | Hard (tone, accuracy, timing) | No | Reputation risk | **Human signs** | \n| Emergency fleet patching | Yes (version probes, health checks) | Mostly | Attackers racing you | **Dark execution, human owns the plan** | \n\nThe last row is worth pausing on, because it is the closest thing to a real dark factory most teams will operate this year: when an emergency lands, automation finds every runtime, bumps every version, restarts every worker and verifies every host — the doing is dark, the plan is human, and verification is outcome-based. That is the shape that works, and it generalizes.\n\nThe dark-factory conversation will mature the way every automation conversation matures: from \"will agents replace X\" to \"what is the gate budget\". A gate budget is the honest accounting of a pipeline: how many human judgments per hundred automated actions, and at which stations. Teams that run real unattended automation already keep this ledger informally — the on-call rotation is a gate budget with a pager.\n\nThree predictions from the floor, not the balcony:\n\n`gate()` function above.\nThe factory can be as dark as its gates are good. Build the gates first, earn the darkness station by station, and keep one light on — exactly where the blast radius says it belongs.\n\n**Is a \"dark software factory\" just hype for \"CI/CD with agents\"?**\n\nMostly, today. The new part is agents proposing changes across the whole lifecycle, not just running a fixed pipeline. The discipline that makes it work — gates, idempotency, outcome verification — is the same discipline CI/CD taught us; the blast radius is larger, so the gates must be better.\n\n**Can tests be the verifier for agent-written code?**\n\nOnly if the tests are not written by the same mind that wrote the code. An agent that writes code and its tests, then reports \"all green\", has optimized a proxy. Independent verification means a signal the generator cannot produce: existing golden tests, property-based checks against a spec, production shadow traffic, or a human.\n\n**What is the single cheapest gate to add to an existing pipeline?**\n\nIdempotency claims before every side effect. Most \"the agent did it twice\" incidents die at this gate, and it costs a key and a lookup per action.\n\n**Should small teams bother with this, or is it enterprise theater?**\n\nSmall teams feel the failure modes first, because one person is the entire human queue. The gate pattern above scales *down* well: the point is routing the three irreversible decisions a week to yourself instead of reviewing everything.\n\n**Does adversarial drift mean scraping can never go dark?**\n\nNo — it means scraping goes dark with a drift watch, not dark forever. Outcome signals (schema stability, volume bands, spot-check verification) are the lights that stay on while the station runs unattended.\n\n*Originally published on [botslave.com](https://botslave.com/blog/dark-software-factories-lights-out). Botslave is an automation studio — see [what we build](https://botslave.com/services) and [how we work](https://botslave.com/how-we-work).*", "url": "https://wpnews.pro/news/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in", "canonical_source": "https://dev.to/nguyen_tu_ca842184efc96f4/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in-the-loop-c6n", "published_at": "2026-10-23 01:32:37+00:00", "updated_at": "2026-10-23 01:41:28.590785+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "mlops", "artificial-intelligence"], "entities": ["botslave", "daily.dev"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in", "markdown": "https://wpnews.pro/news/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in.md", "text": "https://wpnews.pro/news/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in.txt", "jsonld": "https://wpnews.pro/news/dark-software-factories-what-runs-lights-out-and-what-breaks-without-a-human-in.jsonld"}}