{"slug": "we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs", "title": "We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed", "summary": "The bayze-mcp project expanded its multi-model orchestration tool from a single draft-review-revise waterfall into five collaboration patterns, including a debate mode where two models argue over K rebuttal rounds and a third arbitrates. Stress-testing debate across six rounds surfaced seven bugs, among them a silently dropped context field, a token cap that permitted output the character gate would reject, and a fabricated-invoice claim by qwen, which the team addressed with self-calibrating chars/token measurement and an arbiter rule discounting unverifiable quantitative claims. The team reports that pattern-task fit matters more than model quality, with MoA aggregation overshooting hard length caps by 59% and 74% while a judge-based pattern respected them.", "body_md": "Three months ago [bayze-mcp](https://github.com/caijianxiong2007-prog/bayze-mcp) was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships **five multi-model collaboration patterns** — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included.\n\n| Pattern | Mechanism | Measured cost | Use when | \n|---|---|---|---|\n| Cross-vendor review | A drafts → B structured-review gate → C revises | ~$0.001/run | Default quality tier | \n| Parallel + judge | N models answer → judge picks by hard constraints | $0.0013/run | Tasks with length/format caps | \n| MoA aggregation | N models answer → synthesizer merges the best of each | $0.006/run | Open-ended generation | \n| Revision loop | Review bounces → revise → re-review → escalate to a stronger model when exhausted | $0.002/escalation | Unstable first drafts | \n| Debate | Two debaters open independently → K rebuttal rounds → arbiter verdict | $0.02–0.04/run | Analysis / decisions / risk assessment | \n\nThe single most useful finding: **pattern-task fit beats model quality**. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you \"the better of two drafts\"; only synthesis takes the best of both.\n\nFive samples across six rounds:\n\nNo model's self-count is trustworthy. So bayze treats length like memory safety: **the engine counts, deterministically** — every output passes `max_chars` / `max_words` checks, and the model's opinion is never consulted.\n\nWe stress-tested `strategy: debate` with a real question: *\"Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?\"*\n\n**Rounds 1–2, two invisible bugs.** The test agent complained \"my constraints are being ignored.\" They were — but not by the models: the `context` field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, `maxOutputTokens: 2000` allowed ~3000 characters of output while a `max_chars: 2000` gate rejected at 2000 — **the engine was permitting output it was guaranteed to reject**.\n\n**Round 3, the debate actually worked.** Rebuttals quoted each other point by point. One side caught the other promising \"RPO < 5s\" in its opening while admitting \"primary failover takes 3.2 hours\" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run.\n\n**Rounds 4–5, engineering answers to \"vibes\" problems.** Models treated \"opening / rebuttal / verdict\" discipline as *their* assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by **self-calibration**: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen **fabricated \"we audited this team's real invoices (redacted)\"** plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict.\n\n**Round 6, the last design flaw.** Both models ended up on the same side (\"worth migrating\"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional `stances: [pro, con]`, with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.\n\nCondensed from two debates plus verdicts:\n\nDon't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using \"we can leave\" as negotiating leverage.\n\nCaveat that matters more than the conclusion: **every dollar figure both models produced was fabricated** (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: *model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible.*\n\n```\nnpx -y bayze-mcp@0.14.0 mcp --config your-config.yaml\n```\n\n`docs/adr/0006-multi-ai-collaboration.md`\nMIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.", "url": "https://wpnews.pro/news/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs", "canonical_source": "https://dev.to/caijianxiong2007prog/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs-fixed-4c5", "published_at": "2026-11-06 13:43:41+00:00", "updated_at": "2026-11-06 13:54:02.326723+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "ai-safety", "developer-tools"], "entities": ["bayze-mcp", "qwen", "Claude", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs", "markdown": "https://wpnews.pro/news/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs.md", "text": "https://wpnews.pro/news/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs.txt", "jsonld": "https://wpnews.pro/news/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs.jsonld"}}