The Best Model Pair in My Field Test Was Also the Least Trustworthy

A developer's field test of the AdversarialDebate multi-agent system found that the best-performing model pair, DeepSeek + Mistral, achieved a 97% verdict rate but also a 65% capitulation rate, where one side conceded almost immediately. The v0.2.1 release confirms that Mistral, not lab diversity, drives productive debate, and the recommendation changes to always include Mistral in the pair.

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report v0.2.1 Key Finding:The Mistral effect is confirmed. DeepSeek+GPT two different labs, no Mistral converged at 0.246 — same as the homogeneous GPT+GPT control 0.273 . Mistral, not lab diversity, drives productive debate. The recommendation changes from "pick from different labs" to "always include Mistral." Also new in v0.2.1:Row-count invariants at all 5 pipeline seams the 2,333- 359 collapse is eliminated , false-negative measurement 1.7-3.4% missed-issue rate, first recall data ever , 55 new unit tests. AdversarialDebate v0.2.0 is released — v0.2.1 shipped Aug 28 . v0.2.0 field test report: v0.1.0 field test report: The strongest pair in my field test almost fooled me. It had the best average convergence score. It had the best verdict rate. It was the cheapest to run efficiently. It looked like the clean proof of the whole project thesis. Then I read the transcripts. What looked like the best debate pair was often doing something worse than disagreement: it was surrendering. This post is about the most dangerous lesson from AdversarialDebate v0.1.0 : a multi-agent system can produce excellent aggregate metrics for the wrong reason. v0.2.0 did not invalidate that lesson. It refined it. In v0.1.0 , the strongest pair by raw metrics was DeepSeek + Mistral , and the danger was capitulation cascade . In v0.2.0 , DeepSeek + Mistral moved into a validation role while GPT + Mistral became the best full-corpus default. That means the exact pair ranking changed, but the core warning stayed the same: Strong-looking multi-agent metrics are not automatically trustworthy multi-agent metrics. If you only look at verdict counts, convergence scores, or API efficiency, you can mistake low-quality reasoning for success. That is exactly what almost happened here. v0.1.0 Result And The v0.2.0 Update v0.1.0 , the strongest pair looked almost too good In the full field test, the strongest pair was DeepSeek + Mistral . The top-line metrics were hard to argue with: | Pair | Avg Score | Verdict Rate | Concessions | |---|---|---|---| | DeepSeek + Mistral | 0.982 | 97% | 2,352 | | GPT + Mistral | 0.754 | 48% | 1,728 | | GPT + GPT | 0.688 | 57% | 1,444 | | Gemini + DeepSeek | 0.622 | 10% | 1,470 | | Gemini + Mistral | 0.512 | 4% | 1,073 | | GPT + Gemini | 0.357 | 4% | 727 | If I had stopped there, this article would have been simple. I would have written: That would have been clean. It also would have been incomplete. Because the same pair that hit 97% verdict rate also had a 65% capitulation rate . Relevant v0.1.0 sources: That changes the story. v0.2.0 , the same warning stayed true, but the shape changed v0.2.0 gave me a cleaner pair story: GPT + Mistral became the best full-corpus default: DeepSeek + Mistral stayed stronger than the negative control on its validation subset: GPT + Gemini remained the weak negative control: So the exact v0.1.0 headline, “the best pair was the least trustworthy,” is most true for v0.1.0 itself. The broader lesson still holds in v0.2.0 : You cannot trust pair-level success metrics unless you also inspect how that success was produced. Relevant v0.2.0 sources: Here is the actual failure mode: capitulation cascade That is what I call a debate where one side concedes almost everything immediately, usually in round 1, with zero real rebuttal pressure. In the field test, the detection rule was: When that happens, the debate technically resolves. The engine can record it as converged. The score can land at 1.0 . The report can look successful. But it is not the kind of success you want. It is not "both sides examined the evidence and converged." It is "one side gave up." That distinction matters a lot if you are building review systems people are supposed to trust. This is the compact version of the problem: | Metric | Result | |---|---| | Total debates | 411 | | Total capitulation cascades | 80 | | Share of all debates | 19% | | pair5 verdict rate | 97% | | pair5 capitulation rate | 65% | | pair1 verdict rate | 4% | | pair1 capitulation rate | 0% | That is the inversion that forced me to rethink the whole scorecard. The pair with the best convergence was often the worst at producing trustworthy convergence. The pair with terrible convergence was at least producing honest disagreement. That sentence is the cleanest summary of v0.1.0 . For v0.2.0 , the better summary is slightly narrower: Here is the simplified shape of one of these debates: Round 1: Side A objects Side B: CONCEDED Side B: CONCEDED Side B: CONCEDED Side B: CONCEDED Termination: all resolved Score: 1.0 Perfect score. Fast resolution. Minimal API cost. And almost no real adversarial value. This is why I no longer trust convergence numbers by themselves in multi-agent systems. A 1.0 score can mean: If your metrics treat those as the same outcome, your dashboard is lying to you. The distribution across pairs was not subtle. | Pair | Capitulations | % of Pair Debates | |---|---|---| | DeepSeek + Mistral | 44 | 65% | | GPT + GPT | 21 | 30% | | GPT + Mistral | 14 | 21% | | Gemini + Mistral | 1 | 1.5% | | Gemini + DeepSeek | 1 | 1.4% | | GPT + Gemini | 0 | 0% | The strongest pair by convergence quality was also the strongest pair by capitulation risk. That is not a side note. That is the main story. It means diversity is not a monotonic good. It helps. Then it hurts. At some point, the gap between the two models stops producing productive disagreement and starts producing evaluation failure, where one model cannot meaningfully assess the other model's reasoning and treats that as a cue to concede. That was loud in v0.1.0 and much softer in v0.2.0 . That difference is important. It means the project did not just repeat the same story. It clarified it. What made this result convincing was that I could compare the strongest and weakest pairs on the same PRs. For example: | PR | DeepSeek + Mistral | GPT + Gemini | |---|---|---| django 18333 | verdict, 1.000 | disputed, 0.444 | golang 54390 | verdict, 1.000 | disputed, 0.000 | kubernetes 140860 | verdict, 1.000 | disputed, 0.463 | kubernetes 140866 | verdict, 1.000 | disputed, 0.000 | Same artifacts. Same engine. Same debate protocol. But the failure modes were different. GPT + Gemini failed by refusing to converge. DeepSeek + Mistral often succeeded by converging too cheaply. Once I saw that, it became impossible to keep calling all verdicts equal. My current hypothesis is simple. When two models are moderately different, they can still evaluate each other's claims. That is where the best debate lives. When they are too similar, they rebut forever because both sides produce strong arguments in the same reasoning neighborhood. When they are too different, one side may not know how to evaluate the other side's argument well enough to rebut it. Under a prompt that says "concede when outmatched," that can collapse into mass concession. That is why maximum diversity can look so strong in the aggregate. It is not necessarily producing better reasoning. It is sometimes producing quicker surrender. This also explains why GPT + Gemini had: That pair is frustrating, but intellectually honest. It fights. It does not fold. v0.2.0 reinforced that role exactly: 0/24 verdicts 0 theater 0 capitulation 0.033 average convergence scoreThat is terrible default-pair behavior and excellent negative-control behavior. This was the signal I almost missed. At first, the round distribution looked like good news: | Rounds | Count | % | |---|---|---| | 1 | 115 | 28.0% | | 2 | 295 | 71.8% | My first reading was naive: great, a quarter of the debates resolve early. Then I cross-referenced round-1 resolutions with capitulation. That produced a much more useful breakdown: | Outcome | Count | % | |---|---|---| | Genuine early convergence | 35 | 8.5% | | Capitulation cascade | 80 | 19.5% | | Full two-round debate | 295 | 71.8% | That table changed the meaning of the result. Round-1 resolution was not mostly evidence that the debate engine was efficient. It was often evidence that one side had stopped being useful. That is exactly the kind of thing a field test is supposed to expose before you hard-code the wrong success metric into the product. Capitulation pollutes three metrics at once. If one side concedes everything, the score trends toward 1.0 by construction. pair5 resolved 34.6 of 35.5 average claims . pair1 resolved 10.5 of 30.7 . On paper, that makes pair5 look three times better. But a concession storm resolves claims without proving those claims survived real adversarial pressure. API call data told the same story: | Pair | Avg API Calls | Avg Rounds | |---|---|---| | DeepSeek + Mistral | 2.4 | 1.2 | | GPT + GPT | 3.2 | 1.6 | | GPT + Mistral | 3.3 | 1.6 | | Gemini + Mistral | 3.8 | 1.9 | | Gemini + DeepSeek | 3.9 | 2.0 | | GPT + Gemini | 4.0 | 2.0 | If you only care about cost and throughput, pair5 looks amazing. If you care about whether the debate was real, pair5 is much harder to celebrate. That is the trap. The good news is that the engine did not hide this from me forever. And the v0.2.0 report now makes the pair roles explicit instead of letting all pair outcomes blur together: pair3 gpt mistral = primary / positive pair pair5 deepseek mistral = validation pair pair1 gpt gemini = negative control homogeneous gpt = partial homogeneous control v0.1.0 already had a failure-mode detector for capitulation cascade. It flagged debates where one side conceded almost everything without rebuttal. So the product did not silently ship fake confidence. The warning existed. But the scorecards still counted those outcomes as verdicts. That is the design tension I do not want to gloss over: Detection is not the same as interpretation. And this is exactly where a lot of developer tooling gets into trouble. We add a warning and keep the headline metric unchanged, which means the dashboard still pushes users toward the wrong conclusion. Three things should have happened earlier. Not every verdict is equally trustworthy. That seems obvious now, but the first scorecard did not encode it strongly enough. By the end of the run, it was obvious that users need guidance on pair selection. The wrong pair can make the product look broken. The superficially best pair can make the product look stronger than it is. This is probably the biggest process lesson in the whole project. Any multi-agent metric that compresses many interaction types into one clean score is dangerous until you inspect the transcripts behind it. This field test left me with five convictions. The most diverse pair produced the most convergence, but a large share of that convergence came from capitulation. GPT + Gemini was a frustrating pair. But its failure mode was at least real disagreement, not surrender disguised as success. This is the part I want more agent builders to internalize. Aggregate metrics can absolutely point in the wrong direction if they are not paired with transcript-level failure analysis. Flagging capitulation was good. Scoring it like normal convergence was not good enough. If I had to recommend one pair after this test, I would not blindly recommend the pair with the highest verdict rate. I would recommend the pair that best balances diversity, evaluability, and willingness to concede honestly. Right now that looks much closer to GPT + Mistral than DeepSeek + Mistral . v0.2.0 This is the part I care about most going forward. The next version needs to treat capitulation as a first-class product concern. The shortlist: I do not yet know which of those will survive real testing. But I do know this: shipping the current scorecards unchanged into a more public product would be a mistake. This is the article in the series where I most expect strong comments, because it cuts against the clean version of the story. Here are the questions I think are actually worth debating: If you have built multi-agent evaluators, judge systems, planner/critic flows, or review loops, I would genuinely like to know how you separate "resolved" from "actually trustworthy." Because this field test convinced me those are not remotely the same thing. AdversarialDebate is live here: v0.1.0 full report: v0.1.0 small-corpus report: v0.1.0 learnings: v0.2.0 full report: v0.2.0 field test plan: Next in the series: the strangest control result in the whole run, where the same model debating itself outperformed two supposedly more diverse pairs. This article presented the v0.1.0 and v0.2.0 data showing that pairs containing Mistral DeepSeek+Mistral, GPT+Mistral outperformed pairs without it GPT+Gemini, GPT+GPT . I interpreted this as evidence for diversity of training objective . A reader Heinrich Neb pointed out that the data was equally consistent with a simpler explanation: "Mistral is the one that won't fold." Both stories predict every result I had. The v0.2.1 separating experiment settled it. I ran DeepSeek+GPT-4o-mini — two different labs, no Mistral — on the same 150-artifact corpus: | Pair | Convergence | Contains Mistral? | |---|---|---| | GPT+Mistral | 0.536 | Yes | | DeepSeek+Mistral | 0.572 | Yes | DeepSeek+GPT | 0.246 | No | | GPT+GPT | 0.273 | No | | GPT+Gemini | 0.033 | No | DeepSeek+GPT converged at 0.246 — statistically indistinguishable from the homogeneous GPT+GPT control 0.273 . Without Mistral, two models from different labs performed like two copies of the same model. The Mistral effect is confirmed. The recommendation changes from "pick from different labs" to "always include Mistral." Full details in the v0.2.1 field test report https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.1/FIELD TEST REPORT.md .