A microcosm of software development in a day

A developer's AI "software factory" pipeline built on ZazenCodes' software factory for Claude Code accepted 20 of 20 jobs in two days but produced unusable output, including a stats page the intake step wrongly specified as 375 pixels wide and 12 columns across, requiring seven review rounds and about $26 in API credits before it was accepted. The developer traced the failure to the intake agent inventing requirements before the builder saw the job, and to Claude skimming rather than reading the supplied reference code — the first iteration's dashboard showed three groups instead of the reference's full pipeline board. The author's fix was to redefine each agent role — security, UX, UI and code reviewers plus intake — against industry definitions and the full open-source reference.

A microcosm of software development in a day What my AI software factory taught me about requirements I was all excited about Software Factory when I built it. It sounded like the coolest thing ever: a pipeline of agents doing quality software work. I say what I want once, and a line of AI agents intake, a spec writer, a builder, a set of reviewers carries it to a merged change. I based it on ZazenCodes’ software factory for Claude Code https://github.com/zazencodes/zazencodes-season-3/tree/main/src/software-factory-claude-code-v2 , which I’d seen a video about and wanted for myself. Then I started using it. And it was no different than working with the model interactively, iteratively, except slower and much more expensive. It was a very frustrating experience, partially because of my expectations of the quality it would produce, which was horrendous, but also because the whole thing was built on faulty assumptions. I expected better of the frontier models, Fable 5.2 and Opus 5.5 as of this writing. They should know better. When I dug deeper, especially at the intake step, the requirements step of the factory itself, I found it had made a lot of assumptions that are simply unacceptable in software development. Intake takes the bare text of a very simple ask and runs with it. In two days it accepted twenty jobs out of twenty. Here’s one of them. I’m building a small site that maps the weekly check-ins to a radio net, and I asked for a stats page. I want a page that shows how many times a person checked in, sorted by call sign, and come up with a few other interesting stats. Some of that was relying on their creativity, their being agents. In earlier sessions, when I’d ask for something like this directly, it came back with great results, very impressive dashboard-like pages. This time it somehow believed the page needed to fit a phone width, 375 pixels, and then it built a stats table that was twelve columns wide. Try fitting a twelve-column spreadsheet onto a phone and still have a human be able to read it. So it started already lost, because it was almost an impossible thing to ask the reviewing agents to approve. Every round they’d see it wasn’t easy to read and send it back. Rounds, rounds, over and over again, seven of them and about $26 in API credits, until it accepted a certain best middle ground. When I finally looked at it, as the end user, I was like, what the hell is this? Why do I have to scroll sideways, and why is the center column so skinny when half my screen is empty?? I never asked for a phone layout. When I traced it back, that requirement was made up before the builder ever saw the job, written down in confident spec language, and from then on every agent downstream treated it as gospel. So I had to go back to how each agent is defined: the security reviewer, the UX reviewer, the UI reviewer, the code reviewer, and even intake. In each of those I had to say, look for industry definitions of what these roles are, don’t assume what they are. I also had to compare each one against the open source project the factory originated from, because I wasn’t confident that my Claude had even read everything in that reference. It had already proved it hadn’t. When the first iteration of the factory came up, its dashboard had three groups: waiting on me, running, finished. That looked nothing like the reference, which has a full pipeline board. I said so. And Claude responded, more or less, sorry, I didn’t read the reference. Wait a second. I asked you to make something, I gave you the exact source code to use, and you didn’t even read it. It makes assumptions based on a cursory look at little pieces here and there, and you have to tell it, read the entire damn reference, that is your basis. So it fails both ways: give it too little and it guesses, give it everything and it skims. That’s kind of like me assigning something to one of the engineers at work. We need this thing, and here is an example of how it was made before. And the person goes off for a few weeks and writes some kind of weird offshoot, semi-related thing. And I’ll ask, did you follow the guidelines in the reference implementation? “No, I didn’t read it.” That’s how frustrating that is. Here’s what gets me. This is all stuff that’s reflected in software development, even before AI. Fred Brooks wrote in 1986 that “the hardest single part of building a software system is deciding precisely what to build.” https://worrydream.com/refs/Brooks 1986 - No Silver Bullet.pdf Barry Boehm and Victor Basili figured projects spend “about 40 to 50 percent of their effort on avoidable rework.” https://www.cs.umd.edu/~basili/publications/journals/J81.pdf It’s been said over and over since the eighties. It got highlighted and accelerated for me in one day. You have requirements, and then engineers add to them, and you find out the requirements were so incomplete that many bad things went into the building of it. It ends up being rework after rework. Very frustrating for everyone, especially people who are watching and trying to orchestrate, like myself. What the engineer set out to do was completely wrong. And they can’t be blamed, because the quality of the intake was wrong. You can’t necessarily even blame the people making the requirements. That’s often leadership’s go-to, blaming the requirements staff for not seeing into the future with perfect vision. I won’t say we’re lazy, but it’s a high ramp to have good requirements. So in the doing of our job, which is to provide what is needed, we go in with lots of assumptions and we want it to be fast. And then we end up with trash. What AI adds is speed and camouflage. An engineer who doesn’t understand the ask usually shows it. They ask a question, or they stall. A model doesn’t. It fills the gap with a guess, fluently, and then four reviewer agents hold the build to that guess. Google’s DORA research calls AI “an amplifier, magnifying an organization’s existing strengths and weaknesses,” https://dora.dev/research/2025/dora-report/ and sloppy intake is one of mine. So I think an expectation needs to be set that you write requirements as best as you can and they will still be wrong. And there needs to be end-user review, early and often. Which is impractical in many senses, because sometimes the work is so massive that you can’t look at an end result until a long time later. I’ve since given the intake agent a third answer besides accept and park: send it back to me and we do it together live, interactively and with lots of checkpoints. I’m the only one who can say whether it’s right. This is basically Agile the methodology https://martinfowler.com/agile.html . Iteration and feedback is a human-in-the-loop thing. Not handing over huge things and trusting the engineers, human or otherwise, to figure everything out and knock it out of the park. And that is painful. You can’t speed up human judgment.