A practical guide to planning, ticket-splitting, and the build-validate-review loop behind a reliable AI software factory for coding agents.
What is an AI software factory? #
An AI software factory is a structured system for running coding agents through planning, ticket creation, and implementation with humans still checking the output at key points. It differs from a fully autonomous “dark factory,” where you hand an agent a single high-level document and expect it to build, validate, and ship an entire application without stopping. The factory model keeps the core loop of building and validating tickets automated, but it keeps humans in charge of planning scope and reviewing results.
TL;DR #
-
Fully autonomous “dark factories” that take a one-line product brief and ship to production without human checkpoints tend to break down because agents can’t reliably infer human intent at scale.
-
The real bottleneck in agentic coding isn’t model capability, it’s communicating intent : even very strong LLMs can’t fully guess what a human actually wants from a feature or codebase.
-
A working AI software factory keeps the same build/validate/review loop as a dark factory but narrows the scope to one release or epic at a time, instead of an entire application.
-
Planning stays outside the automated loop on purpose. Humans write the PRD or architecture doc, and that document becomes the input the factory splits into tickets.
-
Holdout scenarios , tests the implementing agent never sees, help prevent agents from overfitting their code to known test cases instead of building the feature properly.
-
Splitting an epic into roughly 10 to 20 tickets with clear in-scope and out-of-scope notes, plus dependency mapping, sets up the factory’s automated loop to run in the right order.
-
Code maintainability is a known failure mode for highly autonomous agent systems, which is one of the main reasons human review stays built into the process rather than being automated away.
-
✕a coding agent
-
✕no-code
-
✕vibe coding
-
✕a faster Cursor
The one that tells the coding agents what to build.
Why doesn’t a fully autonomous “dark factory” work? #
The dark factory represents the most hands-off version of agentic software development. You give the system a high-level document, an epic or a release description, and it is supposed to figure out how to split the work, build it, validate it, and ship straight to production with no human steering the direction along the way.
The appeal is obvious. It mirrors the idea of self-driving software development: you specify the destination, not the route. This framing comes from the “five levels of autonomy” concept popularized by Dan Shapiro, where level five removes the steering wheel entirely.
The problem shows up well before model capability becomes the limiting factor. As language models get better at writing code, they don’t get proportionally better at reading human intent. People rarely describe exactly what they want on the first try, and they almost never fully specify how they want a codebase maintained over time. A dark factory has to fill in those gaps itself, and filling gaps means guessing. Guessing at scale, across dozens of tickets, compounds into software that technically works but drifts from what was actually wanted.
There’s a second, more mechanical failure mode: agents tend to let codebases sprawl. Without a human periodically checking structure, conventions, and technical debt, a fully autonomous loop can produce a pile of code that passes its own tests but becomes harder to maintain over time. This maintainability problem has been raised publicly as one of the core reasons software factories built entirely without human checkpoints tend to fail in practice.
How does an AI software factory differ from a dark factory? #
The key difference is scope, not capability. A dark factory tries to take an entire application from a single broad brief to production. A software factory takes one release or one epic at a time, and a human defines that scope before any automation starts.
The core loop inside the factory, where an individual ticket goes in and a pull request comes out, looks similar in both models. What changes is everything that happens before and after that loop:
- Before: a human writes the planning document (PRD, architecture doc, whatever the team already uses) and defines what’s in scope and out of scope for this release.
- During: the factory splits that document into tickets, maps dependencies between them, and runs agents through the build/validate/review loop for each one.
- After: humans review the output rather than letting it ship untouched.
This keeps most of the day-to-day execution automated while keeping humans in the loop at the two points that matter most: deciding what to build, and checking what got built.
How do you plan and scope a release for the factory? #
#
Plans first. Then code.
Remy writes the spec, manages the build, and ships the app.
Planning is deliberately left outside the automated system. The idea is that most teams and individual builders already have a planning process they trust, whether that’s a PRD, an architecture document, or some other format, and the factory shouldn’t try to replace it. Instead, whatever planning document you already produce becomes the input to the factory.
Two practical scoping rules make this work:
For greenfield projects, the first release the factory targets should be a proof of concept, not the full application. For brownfield projects (building on an existing codebase), the release should be scoped as a small epic rather than a sprawling set of features. A reasonable target is an epic that splits into somewhere between 10 and 20 tickets. That’s a wide range on purpose: ticket size varies a lot depending on the work, and the right number depends on how your team or project naturally breaks down work. Over time, running the factory repeatedly builds a sense of what “small enough” actually means for your context.
The planning document should also explicitly state what is out of scope, not just what’s being built. That negative space matters just as much as the feature list, because it keeps the factory from expanding its own interpretation of the work as it churns through tickets.
What are holdout scenarios and why do they matter? #
A holdout scenario is a test or validation check that gets run at the end of implementation, but that the implementing agent is never shown while it’s writing the code. The agent doing the building has no visibility into what it will ultimately be tested against.
This matters because coding agents tend to optimize for whatever signal they can see. If an agent knows the exact test scenarios ahead of time, it has an incentive to write code that satisfies those specific tests rather than genuinely solving the underlying problem. That’s a form of overfitting, and it shows up as code that passes validation but doesn’t actually do what the feature was supposed to do in less obvious cases.
Keeping holdout scenarios hidden from the builder, and only using them at the review and validation stage, forces the agent to implement the actual intent of the ticket rather than gaming a known checklist. This is one of the more transferable lessons from the software factory model: even outside a full factory setup, withholding test cases from the agent that writes the implementation is a simple way to reduce sycophantic, test-chasing behavior.
How does ticket splitting and dependency mapping work? #
Once the planning document exists, the factory’s job is to turn it into a set of individual tickets that can each move through the build/validate/review loop independently. This isn’t just a matter of chopping a document into chunks. It involves figuring out:
- Which tickets can be worked on in parallel because they don’t depend on each other.
- Which tickets have dependencies and need to be completed in a specific order.
- How to label or tag tickets (for example, with GitHub labels) so the automated loop knows the correct sequence to work through them.
Getting this right matters because the core loop treats each ticket as a self-contained unit: an issue goes in, a pull request comes out. If dependencies aren’t mapped correctly, the factory can end up building or testing things in an order that doesn’t make sense, which undermines the reliability the whole system is trying to achieve.
Is this approach worth adopting if you’re not building a full factory? #
Yes, even a partial version of this structure has practical value. You don’t need a fully automated pipeline to benefit from separating planning from execution, scoping work into small epics instead of massive briefs, defining explicit out-of-scope boundaries, or hiding test scenarios from the agent doing the implementation. Each of these is a standalone improvement to an existing agentic coding workflow, independent of whether you build the full build/validate/review loop around it.
The broader argument is that this kind of structure, scoped releases, planned tickets, a repeatable build/validate/review loop, and human checkpoints at the planning and review stages, is where agentic engineering workflows are generally headed as harnesses and models improve. More autonomy gets added to the execution loop over time, but the planning and review responsibilities stay with humans because intent and codebase judgment aren’t things current models reliably replace.
Frequently Asked Questions #
What is the difference between a dark factory and an AI software factory?
A dark factory takes a single high-level brief and tries to build, validate, and ship an entire application with no human steering after that point. An AI software factory uses the same kind of automated build/validate/review loop but scopes the work to one release or epic at a time, with a human defining the plan upfront and reviewing the output.
Why do fully autonomous coding pipelines tend to fail?
The main issue is that language models can’t reliably infer what a human actually wants, even as their raw coding ability improves. A second issue is code maintainability: agents working without human checkpoints tend to let codebases sprawl in ways that are hard to clean up later.
What is a holdout scenario in agentic coding?
It’s a test or validation check used to verify a ticket’s implementation that the agent writing the code never sees. Hiding it prevents the agent from writing code that’s overfit to pass known tests rather than actually solving the problem.
How many tickets should an epic be split into for a factory?
A rough guideline is 10 to 20 tickets per epic, though ticket size varies considerably and the right number depends on how a given team or project naturally scopes work.
Does using a software factory mean giving up control over planning?
No. Planning is intentionally kept outside the automated loop. Humans still write the PRD or architecture document that defines scope, and that document is what the factory uses to generate and sequence tickets.