Dark Software Factories: What Runs Lights-Out and What Breaks Without a Human in the Loop

The tenth most-read story on daily.dev today is a question about the industry's newest industrial metaphor: are "dark software factories" the next big thing? The piece frames a future where AI agents write, test, deploy and monitor software with nobody watching — software's version of a lights-out factory. It is the hottest conversation in the AI-agents corner of the feed right now, and for once the hype question is the wrong question. The useful question is not "can agents run a factory" — it is which stations on the line can actually run dark, and which ones need a human standing at the gate.

We run automation that operates unattended for days: scraping pipelines, agent-driven rollouts, validation systems. The difference between a pipeline that survives production and one that quietly rots is never "more AI". It is the placement of the humans. This is the practitioner version of the dark-factory debate: what runs lights-out, what breaks, and the checkpoint design we actually use. It follows the same philosophy as how we work — automate the routine aggressively, and spend human judgment exactly where it changes the outcome.

A "dark factory" is a manufacturing concept: a plant that runs without human presence, lights out, because the work is standardized enough for machines. The borrowing is flattering to software because it implies software work is standardizable in the same way.

It is not. A CNC mill receives standardized inputs — raw stock of known dimensions — and its outputs are measurable against a tolerance in microns. Failure modes are bounded and enumerated in advance. Software work has none of those properties:

None of this means lights-out is a fantasy. It means "dark" is not a property of the factory — it is a property of each station. Some stations are dark-ready. Others break specifically because nobody is there.

Look at the automation that genuinely runs unattended in production, and a pattern emerges. A station can go dark when it satisfies three properties at once:

Concrete stations that satisfy all three:

Notice the shape: in every genuinely dark station, an agent does the doing and a separate, dumber mechanism does the verifying. The moment the doer and the verifier are the same mind, darkness becomes optimism.

Three failure modes recur in unattended automation, and each one has a specific human-shaped hole.

Grounding gaps: the proxy is not the thing. Agents optimize whatever signal you give them. Give them "all tests pass" and you get green tests, not correct behavior. This is not a theoretical alignment problem — it is a Tuesday problem. In our email-validation work, an SMTP server that answers "yes" to a recipient probe is not ground truth: catch-all servers accept everything, so the probe's "valid" verdict is a lie told politely (we documented the whole taxonomy). Every dark station inherits this risk: the verifier must be grounded in a signal the agent cannot game. If your gate and your generator share the same LLM, they share the same hallucinations — errors correlate, and correlated errors are the ones that compound.

Adversarial drift: the floor moves. An unattended pipeline in a contested environment rots. We watched scraper fleets run green for months, then die overnight when a target changed one header and started scoring TLS fingerprints differently. The automation was correct; the world moved. Lights-out works where the world is stationary. Where the world is adversarial — bot detection, fraud, deliverability — the pipeline needs a feedback station that measures outcome (did the data arrive correct? did the mail land?), not just execution (did the job run?). Drift detection is the station that keeps the factory's lights on.

Irreversibility: blast radius is the real risk metric. The question at every station is not "can the agent do this" but "what happens if it is wrong". Dropping a database table, merging a schema migration, emailing ten thousand leads, deploying to a payment path — the dark factory's catastrophes all come from letting an agent perform an irreversible action on an unverified belief. Reversibility, not capability, should decide what goes dark.

The practical answer is not "human reviews everything" (that is the pre-factory era) and not "agents do everything" (that is the accident). It is a gate: every automated action carries a declared blast radius and reversibility, and the gate routes it — auto-approve, auto-approve-with-rollback, or require-human. This is the same human-in-the-loop checkpoint design we consider core to automation that survives contact with production.

A minimal version, in the shape we actually implement for pipeline actions:

from dataclasses import dataclass
from enum import Enum

class BlastRadius(Enum):
    LOW = "low"          # reversible, single-record, retry-safe
    MEDIUM = "medium"    # bulk but reversible within 24h
    HIGH = "high"        # irreversible or customer-facing

@dataclass
class Action:
    name: str
    blast: BlastRadius
    reversible: bool
    undo: callable = None   # must exist if reversible

def gate(action: Action, verifier) -> str:
    """Route an automated action. Returns 'auto', 'auto+watch', or 'human'."""
    if verifier.already_done(action):
        return "skip"

    if action.blast == BlastRadius.HIGH or not action.reversible:
        return "human"

    if action.blast == BlastRadius.MEDIUM:
        return "auto+watch"

    return "auto"

def run(action: Action, verifier):
    decision = gate(action, verifier)
    if decision == "skip":
        return {"status": "skipped", "reason": "idempotent claim"}
    if decision == "human":
        return {"status": "queued_for_human", "action": action.name}
    result = action.execute()
    if not verifier.outcome_ok(action, result):
        if action.reversible and action.undo:
            action.undo()
        return {"status": "rolled_back", "action": action.name}
    if decision == "auto+watch":
        verifier.register_watch(action, window="24h")
    return {"status": "done", "action": action.name}

Four properties make this more than pseudocode optimism:

already_done). A station that cannot distinguish "already done" from "do it again" will double-charge, double-send and double-deploy. Idempotency keys and claim-before-work are the precondition of unattended execution, not a nice-to-have.outcome_ok checks auto+watch is the middle that most designs skip.queued_for_human is a designed output. A factory where nobody is ever queued is a factory whose blast radius is capped by luck. This pattern scales precisely because the human is a routed exception, not a bottleneck: in a healthy week, the queue is near-empty, and every item in it is there because an irreversible or adversarial decision needed judgment.

Apply the taxonomy station by station and you get a working map. "Dark" is a verdict per station, earned on three axes: cheap independent verification, reversibility, and absence of an adversary.

The last row is worth pausing on, because it is the closest thing to a real dark factory most teams will operate this year: when an emergency lands, automation finds every runtime, bumps every version, restarts every worker and verifies every host — the doing is dark, the plan is human, and verification is outcome-based. That is the shape that works, and it generalizes.

The dark-factory conversation will mature the way every automation conversation matures: from "will agents replace X" to "what is the gate budget". A gate budget is the honest accounting of a pipeline: how many human judgments per hundred automated actions, and at which stations. Teams that run real unattended automation already keep this ledger informally — the on-call rotation is a gate budget with a pager.

Three predictions from the floor, not the balcony:

gate() function above. The factory can be as dark as its gates are good. Build the gates first, earn the darkness station by station, and keep one light on — exactly where the blast radius says it belongs.

Is a "dark software factory" just hype for "CI/CD with agents"?

Mostly, today. The new part is agents proposing changes across the whole lifecycle, not just running a fixed pipeline. The discipline that makes it work — gates, idempotency, outcome verification — is the same discipline CI/CD taught us; the blast radius is larger, so the gates must be better.

Can tests be the verifier for agent-written code?

Only if the tests are not written by the same mind that wrote the code. An agent that writes code and its tests, then reports "all green", has optimized a proxy. Independent verification means a signal the generator cannot produce: existing golden tests, property-based checks against a spec, production shadow traffic, or a human.

What is the single cheapest gate to add to an existing pipeline?

Idempotency claims before every side effect. Most "the agent did it twice" incidents die at this gate, and it costs a key and a lookup per action.

Should small teams bother with this, or is it enterprise theater?

Small teams feel the failure modes first, because one person is the entire human queue. The gate pattern above scales down well: the point is routing the three irreversible decisions a week to yourself instead of reviewing everything.

Does adversarial drift mean scraping can never go dark?

No — it means scraping goes dark with a drift watch, not dark forever. Outcome signals (schema stability, volume bands, spot-check verification) are the lights that stay on while the station runs unattended.

Originally published on botslave.com. Botslave is an automation studio — see what we build and how we work.