The dark-factory dream hit a wall #
Original ambition for the AI Software Factory was a dark-factory dream: feed it an issue or PRD, and it would autonomously plan, code, verify, and even merge changes directly to production. This hands-off ideal promised high-volume pull request (PR) generation with minimal human intervention, effectively creating a self-piloting development team.
That dream hit a wall. Creator Cole Medin initially built a "Dark Factory" with no human in the loop, aiming for an idea-to-shipped-code pipeline without even a merge gate. While suitable for proofs-of-concept, this fully autonomous approach proved unreliable for production-grade software.
Medin’s practical lesson was that autonomous execution excels at much of the middle ground of development, like generating boilerplate, implementing features, and running verification. However, critical decisions regarding product intent, architectural choices, and breaking schema changes demand human judgment.
The system evolved into a "Software Factory" with carefully placed human-in-the-loop points. A Mission Control Dashboard now surfaces ambiguous decisions and high-level architectural queries, allowing human developers to make high-leverage choices rather than sifting through code. This shift balances AI efficiency with essential human oversight, optimizing throughput while maintaining quality and strategic alignment.
Mission control makes the machine legible #
A central dashboard serves as mission control for the Software Factory, transforming an opaque "dark factory" into a legible, manageable system. This control plane offers a visual map of factory workflows and individual agent skills, tracking live runs, key decisions awaiting human attention, and system health signals. Version history provides an audit trail of factory evolution.
Run-level details are crucial for diagnosis and improvement. Medin reports that throughout hundreds of recent “laps” (factory runs), 21 failures stemmed from workflow issues, while others pointed to rate-limit problems. This granularity helps operators identify systemic weaknesses and refine the factory’s autonomous loops.
Centralizing this information within a dedicated dashboard surpasses the limitations of tracking everything in GitHub issues and PR comments. Operators gain a holistic understanding of the system’s shape, enabling them to drive improvements without losing the high-level view. This approach allows for the minimal, high-leverage human decisions necessary to guide and evolve the factory.
Keep humans at the judgment points #
The factory’s ambition isn't unconstrained autonomy, but rather high-leverage intervention. Instead of silently settling assumptions or requiring constant human supervision, the system escalates specific, critical questions to an owner-decision queue. This ensures that ambiguities aren't buried in a run, and humans retain control over decisions the system shouldn't make unilaterally.
This approach contrasts sharply with the early "dark factory" dream—a fully lights-out operation from PRD to merged code. While pure generative AI can produce impressive proof-of-concept code, reliable production requires more than just generation. It demands robust verification, oversight, and a clear mechanism to address unclear assumptions.
The goal is a minimal set of decisions for the human that yields maximum control. The factory is designed to bubble up architectural fork points, schema changes, or other high-impact queries. This preserves agent autonomy for routine tasks while empowering developers to guide the system at crucial junctures, ensuring reliable shipping. For further details on the system's inner workings, see GitHub - coleam00/ai-software-factory.
This human-in-the-loop design, managed through the Mission Control dashboard, reflects a pivot from unmonitored autonomy to a structured system that acknowledges the complexity of real-world software development. It's about orchestrating intelligence, not eliminating human judgment.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
Build a feedback loop, not a black box #
Archon, an open-source multi-agent orchestration framework, forms the backbone of the factory, coordinating reusable workflows for planning, implementation, and validation. Each run's detailed logs and execution traces are stored, allowing for thorough inspection and debugging when issues arise.
Mission Control dashboard transforms these artifacts into actionable intelligence. Scorecards track performance metrics, test scenarios validate outputs, and workflow diffs highlight changes between factory versions. A robust rollback capability allows developers to revert to previous configurations, creating a controlled environment for experimentation.
This iterative approach fosters a continuous improvement cycle. Teams inspect outcomes, adjust the factory’s parameters, and compare version performance, transforming the system from a black box into an observable, tunable instrument. Scaling AI coding is fundamentally an operations problem, not just a model problem.
Success hinges on observable workflows, explicit checkpoints, and controlled experiments. To explore further, consult the open-source factory repository and Port’s detailed explainer.
Frequently Asked Questions #
What is an AI software factory?
It is a system that uses orchestrated AI workflows to turn issues or PRDs into planned, implemented, and verified software changes.
Why did Cole Medin move away from a dark factory?
Fully hands-off automation proved insufficient for dependable product work, so he added human checkpoints for decisions that need judgment.
What does the mission-control dashboard track?
It shows workflows, active runs, escalated decisions, factory health, test results, and changes between versions.
What role does Archon play in the factory?
Archon runs the reusable workflows behind planning, implementation, and validation, while storing run details and logs for observability.