We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed

The bayze-mcp project expanded its multi-model orchestration tool from a single draft-review-revise waterfall into five collaboration patterns, including a debate mode where two models argue over K rebuttal rounds and a third arbitrates. Stress-testing debate across six rounds surfaced seven bugs, among them a silently dropped context field, a token cap that permitted output the character gate would reject, and a fabricated-invoice claim by qwen, which the team addressed with self-calibrating chars/token measurement and an arbiter rule discounting unverifiable quantitative claims. The team reports that pattern-task fit matters more than model quality, with MoA aggregation overshooting hard length caps by 59% and 74% while a judge-based pattern respected them.

Three months ago bayze-mcp https://github.com/caijianxiong2007-prog/bayze-mcp was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships five multi-model collaboration patterns — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included. | Pattern | Mechanism | Measured cost | Use when | |---|---|---|---| | Cross-vendor review | A drafts → B structured-review gate → C revises | ~$0.001/run | Default quality tier | | Parallel + judge | N models answer → judge picks by hard constraints | $0.0013/run | Tasks with length/format caps | | MoA aggregation | N models answer → synthesizer merges the best of each | $0.006/run | Open-ended generation | | Revision loop | Review bounces → revise → re-review → escalate to a stronger model when exhausted | $0.002/escalation | Unstable first drafts | | Debate | Two debaters open independently → K rebuttal rounds → arbiter verdict | $0.02–0.04/run | Analysis / decisions / risk assessment | The single most useful finding: pattern-task fit beats model quality . With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps , while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both. Five samples across six rounds: No model's self-count is trustworthy. So bayze treats length like memory safety: the engine counts, deterministically — every output passes max chars / max words checks, and the model's opinion is never consulted. We stress-tested strategy: debate with a real question: "Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?" Rounds 1–2, two invisible bugs. The test agent complained "my constraints are being ignored." They were — but not by the models: the context field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, maxOutputTokens: 2000 allowed ~3000 characters of output while a max chars: 2000 gate rejected at 2000 — the engine was permitting output it was guaranteed to reject . Round 3, the debate actually worked. Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run. Rounds 4–5, engineering answers to "vibes" problems. Models treated "opening / rebuttal / verdict" discipline as their assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by self-calibration : measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen fabricated "we audited this team's real invoices redacted " plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict. Round 6, the last design flaw. Both models ended up on the same side "worth migrating" . Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional stances: pro, con , with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it. Condensed from two debates plus verdicts: Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first — then using "we can leave" as negotiating leverage. Caveat that matters more than the conclusion: every dollar figure both models produced was fabricated the arbiter flagged and excluded them . What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible. npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml docs/adr/0006-multi-ai-collaboration.md MIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.