Three months ago bayze-mcp was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships five multi-model collaboration patterns — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included.
| Pattern | Mechanism | Measured cost | Use when |
|---|---|---|---|
| Cross-vendor review | A drafts → B structured-review gate → C revises | ~$0.001/run | Default quality tier |
| Parallel + judge | N models answer → judge picks by hard constraints | $0.0013/run | Tasks with length/format caps |
| MoA aggregation | N models answer → synthesizer merges the best of each | $0.006/run | Open-ended generation |
| Revision loop | Review bounces → revise → re-review → escalate to a stronger model when exhausted | $0.002/escalation | Unstable first drafts |
| Debate | Two debaters open independently → K rebuttal rounds → arbiter verdict | $0.02–0.04/run | Analysis / decisions / risk assessment |
The single most useful finding: pattern-task fit beats model quality. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both.
Five samples across six rounds:
No model's self-count is trustworthy. So bayze treats length like memory safety: the engine counts, deterministically — every output passes max_chars / max_words checks, and the model's opinion is never consulted.
We stress-tested strategy: debate with a real question: "Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?"
Rounds 1–2, two invisible bugs. The test agent complained "my constraints are being ignored." They were — but not by the models: the context field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, maxOutputTokens: 2000 allowed ~3000 characters of output while a max_chars: 2000 gate rejected at 2000 — the engine was permitting output it was guaranteed to reject.
Round 3, the debate actually worked. Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run.
Rounds 4–5, engineering answers to "vibes" problems. Models treated "opening / rebuttal / verdict" discipline as their assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by self-calibration: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen fabricated "we audited this team's real invoices (redacted)" plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict.
Round 6, the last design flaw. Both models ended up on the same side ("worth migrating"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional stances: [pro, con], with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.
Condensed from two debates plus verdicts:
Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using "we can leave" as negotiating leverage.
Caveat that matters more than the conclusion: every dollar figure both models produced was fabricated (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible.
npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml
docs/adr/0006-multi-ai-collaboration.md
MIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.