# We Made Two AIs Debate and Hired a Third as the Arbiter — 6 Test Rounds, 7 Bugs Fixed

> Source: <https://dev.to/caijianxiong2007prog/we-made-two-ais-debate-and-hired-a-third-as-the-arbiter-6-test-rounds-7-bugs-fixed-4c5>
> Published: 2026-11-06 13:43:41+00:00

Three months ago [bayze-mcp](https://github.com/caijianxiong2007-prog/bayze-mcp) was a single waterfall: model A drafts, model B reviews, model C revises. Today it ships **five multi-model collaboration patterns** — and the most interesting one, debate, only works because we broke it six times with real runs and fixed what broke. This post is the full debugging log, costs included.

| Pattern | Mechanism | Measured cost | Use when | 
|---|---|---|---|
| Cross-vendor review | A drafts → B structured-review gate → C revises | ~$0.001/run | Default quality tier | 
| Parallel + judge | N models answer → judge picks by hard constraints | $0.0013/run | Tasks with length/format caps | 
| MoA aggregation | N models answer → synthesizer merges the best of each | $0.006/run | Open-ended generation | 
| Revision loop | Review bounces → revise → re-review → escalate to a stronger model when exhausted | $0.002/escalation | Unstable first drafts | 
| Debate | Two debaters open independently → K rebuttal rounds → arbiter verdict | $0.02–0.04/run | Analysis / decisions / risk assessment | 

The single most useful finding: **pattern-task fit beats model quality**. With a hard length cap, MoA aggregation overshot by +59% and +74% in two runs (rewrite steps systematically lengthen text — same family as the 1.66× inflation we measured in revision steps), while a judge picking between candidates naturally respects the cap because it just picks the shorter one. For open-ended writing the judge can only hand you "the better of two drafts"; only synthesis takes the best of both.

Five samples across six rounds:

No model's self-count is trustworthy. So bayze treats length like memory safety: **the engine counts, deterministically** — every output passes `max_chars` / `max_words` checks, and the model's opinion is never consulted.

We stress-tested `strategy: debate` with a real question: *"Should an 8-person startup migrate off a cloud vendor's full stack within 12 months?"*

**Rounds 1–2, two invisible bugs.** The test agent complained "my constraints are being ignored." They were — but not by the models: the `context` field was accepted by the MCP layer and silently dropped by the orchestrator. Separately, `maxOutputTokens: 2000` allowed ~3000 characters of output while a `max_chars: 2000` gate rejected at 2000 — **the engine was permitting output it was guaranteed to reject**.

**Round 3, the debate actually worked.** Rebuttals quoted each other point by point. One side caught the other promising "RPO < 5s" in its opening while admitting "primary failover takes 3.2 hours" in the same document. Cross-model adversarial review genuinely surfaces blind spots a single model has. Then the arbiter wrote 786 characters against a 600-character gate and the entire verdict was dropped — that one arbiter call cost $0.021, 90% of the run.

**Rounds 4–5, engineering answers to "vibes" problems.** Models treated "opening / rebuttal / verdict" discipline as *their* assignment and performed the whole debate solo — fixed with a system prompt scoping each side to its own position. A token cap derived at 1.4 chars/token got bypassed at 2.43 chars/token on mixed Chinese-English Claude output — fixed by **self-calibration**: measure the actual chars/token ratio on the first call, use it to clamp a compress-retry. And qwen **fabricated "we audited this team's real invoices (redacted)"** plus fake benchmarks and Git-history claims — a workflow has no access to any of that. Fixed with an arbiter rule: unverifiable quantitative claims are discounted, suspected fabrications are called out in the verdict.

**Round 6, the last design flaw.** Both models ended up on the same side ("worth migrating"). Explicitly assigning sides did nothing, because both steps received the same input — there was no injection point for stances. Emergent disagreement works on open analysis questions; on yes/no questions both models anchor to the same conclusion. Fixed with optional `stances: [pro, con]`, with one rule baked into the prompt: argue your assigned side as hard as you honestly can, but never fabricate evidence for it.

Condensed from two debates plus verdicts:

Don't do a 12-month full migration. An 8-person team needs 0.5–1 FTE of SRE capacity to self-host; the savings don't cover half a person. AI services are the last thing to move. The correct use of 12 months is making exit cheap (containers, S3/Postgres-compatible interfaces, moving traffic-heavy spend first) — then using "we can leave" as negotiating leverage.

Caveat that matters more than the conclusion: **every dollar figure both models produced was fabricated** (the arbiter flagged and excluded them). What survives is the decision framework and the risk checklist. That's the meta-lesson of all six rounds: *model quantification is not directly trustworthy — cross-model adversarial review plus an arbiter discount just makes the untrustworthiness visible.*

```
npx -y bayze-mcp@0.14.0 mcp --config your-config.yaml
```

`docs/adr/0006-multi-ai-collaboration.md`
MIT, local-first, keys never leave your environment, 193 automated tests. If you're orchestrating multiple models, I'd genuinely like to hear what yours fabricates — we keep a list.