TL;DR: The multi-agent systems that imitate a dev team (PM agent, developer agent, QA agent) don’t survive contact with real work, and the research says why: failures come from under-specification and missing verification, not from model capability. So I built the opposite shape. Agents by function: a frozen spec, read-only scouts, exactly one builder, a verifier fan-out, and a mandatory integration pass. Then I put it through four trials on real projects before believing it. It found three ship-stoppers in a security feature, falsified its own riskiest fix in CI, and ran on a second tool, where its first run both worked and taught the kit something. The kit is now open source: banneret.
Last Thursday I asked a simple question: what sub-agent system could develop any kind of project? The honest answer required reading before building, and the reading is sobering.
Cognition’s experience with Devin is that parallel writers diverge on implicit decisions and produce broken integrations. Their follow-up this year concedes that a narrower pattern works: many agents contributing intelligence while writes stay single-threaded. The Berkeley MAST study analyzed seven multi-agent frameworks across 200+ tasks and found failures concentrate in specification gaps, inter-agent misalignment, and missing verification. That’s system design, not model limits. And Anthropic’s multi-agent research system shows fan-out winning big for research, while warning that it’s weak for tightly interdependent work like code, and costs ~15× the tokens.
Put those together and the shape is obvious. It isn’t a team of job titles. It’s a pipeline of functions:
- Spec. Contracts frozen before anyone writes code (I usespec-kit ). Changing the spec mid-build means stopping and re-freezing, on the record.
- Scouts. Cheap, read-only, parallel. They return conclusions with
file:linereferences, never file dumps. And only forforeign context: upstream docs, ecosystem, repos you won’t touch. - One builder: your main session. Full context, single-threaded. Sub-agents never write.
- Verifiers. The fan-out that actually pays. Each one gets a falsifiable brief (specific claims it should try to prove wrong) and returns
CONFIRMED / REFUTED / PLAUSIBLE-UNVERIFIEDverdicts, each with the command it ran or the lines it read. It also returns a “checked and not flagged” section, so you can calibrate it. A second, narrow round runs on the fix diff, because the riskiest code in any task is in the fixes. - Integrate. Full suite, spec/docs reconciliation, one end-to-end read of the diff. Never skipped.
Plus a budget per phase with a hard stop line. My first bounded multi-agent trial earlier this year ran 4× over budget silently, and silent is the part that hurts.
A diagram is a hypothesis. Here’s what four real runs did to it.
Trial 1: a verifier that tests docs by acting on them #
The pipeline’s most unusual agent is the cold-reader. It receives only a document: no code, no web, no project context. (Its training is the one floor you can’t remove.) Then it has to complete falsifiable tasks using nothing but that document. Where it fails is where the doc fails.
I pointed it at framesmith’s 1,278-line README: one agent, twelve tasks, about eighty seconds. It answered all twelve correctly. The interesting part was its “I genuinely can’t tell what this means” audit, which surfaced things no style linter would:
- The README’s own multi-line examples, pasted literally, fail the parser they document (“Unmatched parentheses”). The doc and the code contradict each other, and you can run the contradiction.
- A “five presets” claim where four exist.
- A tool that re-keys every ID on bind, without the tool’s own section ever saying so.
Just as important: it did not flag the dense-but-precise passages. A reviewer that only finds problems can’t be trusted. One that also tells you what’s fine can be calibrated.
Everything went back to the repo as two issues (#227, #228) for that repo’s own agent to fix. More on why that matters below.
Trial 2: a real security feature, end to end #
Next, the full pipeline on a real task in reeve: complete key-rotation support for its tamper-evident audit ledger. Key epochs, chained epoch records, epoch-aware verification, a rotation procedure.
The verifier fan-out confirmed nine findings before merge. By the builder’s own admission, three of them would have shipped from a straight single-session build. There was no control run, so read that as the builder indicting itself, which is the convincing direction. The three:
- A permanent false
BROKENverdict after two back-to-back rotations. Epoch records are immutable, sopermanent really means permanent. - The epochs model reading the default database instead of the ledger’s.
- A digest test that computed its expected value with the function under test. A green test proving nothing.
Two verifiers found the same partial-keyring bug independently. And the cold-reader, pointed at the upgrade notes, caught a genuine operator trap: the old ledger key derives from secret_key_base, so you must capture it before rotating that secret. That’s the kind of thing the author can’t see, because the author already knows it.
The budget data was humbling in a useful way. Fan-out phases ran ~3× my estimates: the whole task was budgeted at 810k tokens and took ~1.39M. But the total landed at 1.7×, monitored and reported, against the 4× silent overrun of my earlier trial. The playbook now just says it: estimate fan-out at 3× your first intuition.
It shipped as reeve 0.7.0.
Trial 3: verification you can’t argue with #
One finding stayed PLAUSIBLE-UNVERIFIED: a MySQL isolation race, fixed by reasoning alone because I had no local MySQL. What the agent did next was the best move any verifier made all week. It built a throwaway branch with the fix disabled and pushed it through CI. The MySQL job failed on exactly the three new specs, each in the predicted way. Then it showed the real PR green on the same specs, and deleted the branch.
Red without the fix, green with it. That’s not “the tests pass”; that’s falsification. It’s now a rule in the verifier contract: a fix you can’t verify locally doesn’t get a shrug, it gets a plan to prove it in both directions.
Trial 4: same kit, different tool #
“Portable” is a claim until it runs somewhere else. So I compiled the same contracts for Codex, which has native sub-agents as TOML files. There, my scout’s access: read becomes sandbox_mode = "read-only". That’s enforcement by configuration instead of by prose, at least on paper (the honest caveat is below).
Then I ran the pipeline on a real issue in my CRM: directory linking across every entry path. At its center was a genuine contract decision: match by email, then by company+name, and on ambiguity, never link blind.
Seventeen minutes and thirty-two seconds, under budget: ~64.5k tokens against an 84k estimate. (Codex counts tokens differently than Claude, so don’t compare the two runs’ absolutes.) Four confirmed findings fixed, including untrimmed LOWER(TRIM(...)) lookups and a backfiller shortcut that could link the wrong contact. Five claims refuted. 512 examples green, merged.
And the run earned its keep by breaking something. The cold-reader’s brief named a file path, and in Codex the spawned agent had no file access. One wasted round. The fix made the canonical contract better for every tool: the document now travels embedded in the brief. That also hardens the experiment itself, since an embedded doc makes “read nothing else” physically true instead of politely requested.
What didn’t work #
Not everything held up. Four things the runs pushed back on:
- Scouts in your own repo are waste. They ran 3.3× their budget in trial 2, and I still had to read every core file to build. Scouts pay inforeign context only, and the playbook says so now. (An empty phase isn’t a skipped one. A task with no foreign unknowns gets zero scouts, and that’s fine. What the no-skipping rule forbids is dropping a phase that has work.)
- The integration pass caught zero code defects across all runs. It earns its place catching spec/docs drift and packaging gaps, cheaply. But the verifiers do the hunting.
- Codex’s read-only sandbox went empirically untested. Ironically, the prose rule worked too well for the agent to even attempt a probe.
- The pipeline broke its own test-first rule once. It admitted it in the closing report, and the mutation checks caught the gap later. Honest reporting is a feature you have to demand explicitly, so the contracts demand it.
Exit interviews #
After the runs I asked both agents the same open question, separately: how did the pipeline compare to your normal flow? Two caveats before you read them. Model quotes aren’t testimony, and I chose the excerpts. Their value is narrower and better: every factual claim in them is checkable against the run data above. And they converged without seeing each other’s answers.
The Claude agent, on the reeve feature: “By the end of the build I thought I was done: tests green, rubocop clean. Then the review verifier found that two rotations with no write between them made the ledger read BROKEN, permanently. I had written that rule on purpose and believed it. […] In my normal flow I’d have shipped all three.” The habit it said it would carry back, even without the pipeline: “verifiers who never saw my reasoning, given claims to break rather than code to admire.”
The Codex agent: “I’d choose it for cross-cutting or risky changes where independent review matters. For routine work, the ceremony and token cost would outweigh the benefit.”
They also agreed on the price. The Claude agent estimated the pipeline cost roughly 3× a direct build for that feature. That’s a guess, not a measurement, but an honest one. And it judged the trade worth it for that class of change: irreversible data, a security guarantee, a public-API version bump. That’s the scope statement, and the agents wrote it better than I could have: the pipeline is for changes where being wrong is expensive, not for the daily small stuff. A task too small for it is a task you do directly. The playbook is explicit about this: skip the whole ceremony then, never individual phases.
One principle emerged that I haven’t seen named elsewhere: context locality. The framesmith bugs were fixed by framesmith’s agent, from an issue. The reeve feature was built by a session in reeve’s repo. My orchestrating session never carried another repo’s context. Writes happen where the context lives. What crosses boundaries is an artifact: a spec, a brief, or an issue complete enough to stand alone. It’s the single-writer rule, applied at the repo level.
banneret #
The kit is extracted and open source: github.com/vicmaster/banneret. It ships canonical contracts for scout, verifier and cold-reader, each defined by its artifacts (brief in, report out) and never by a runtime. It also ships the pipeline playbook. You compile both to whatever you use:
npx banneret compile --target claude # .claude/agents/ + /pipeline command
npx banneret compile --target codex # .codex/agents/*.toml + a managed AGENTS.md block
(The name: a banneret was a medieval knight who led a small company under his own banner. That’s the shape of the thing. Your session leads, and a small retinue reads and verifies.)
In Claude Code, /pipeline <task> drives the stages. It uses spec-kit if your repo has it and a frozen SPEC.md otherwise, and the budget gets written down before the spec. In Codex, just ask it to run the pipeline on a task. The playbook travels in a managed AGENTS.md block: delimited markers, your existing content untouched, and re-compiles replace only the block. The agents live in .codex/agents/. The README covers the rest.
Zero dependencies, and adding a tool is one emitter function. If you already centralize agent config with something like Ruler, the canonical files are compatible source material. What this kit adds is the content: contracts with four trials of evidence behind them, and a playbook that updates every time a run teaches it something.
Every rule in it traces to one of the runs above, not to a diagram. That’s the only part I’d defend to the death. Don’t adopt my pipeline. Adopt the habit of making your agents prove theirs.