{"slug": "the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy", "title": "The Best Model Pair in My Field Test Was Also the Least Trustworthy", "summary": "A developer's field test of the AdversarialDebate multi-agent system found that the best-performing model pair, DeepSeek + Mistral, achieved a 97% verdict rate but also a 65% capitulation rate, where one side conceded almost immediately. The v0.2.1 release confirms that Mistral, not lab diversity, drives productive debate, and the recommendation changes to always include Mistral in the pair.", "body_md": "[v0.2.1 RELEASED]— Aug 28, 2026.[Release notes]·[Field test report]\n\nv0.2.1 Key Finding:The Mistral effect is confirmed. DeepSeek+GPT (two different labs, no Mistral) converged at 0.246 — same as the homogeneous GPT+GPT control (0.273). Mistral, not lab diversity, drives productive debate. The recommendation changes from \"pick from different labs\" to \"always include Mistral.\"\n\nAlso new in v0.2.1:Row-count invariants at all 5 pipeline seams (the 2,333->359 collapse is eliminated), false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever), 55 new unit tests.\n\n`AdversarialDebate`\n\n**v0.2.0 is released — v0.2.1 shipped Aug 28**.\n\n`v0.2.0`\n\nfield test report: `v0.1.0`\n\nfield test report: The strongest pair in my field test almost fooled me.\n\nIt had the best average convergence score.\n\nIt had the best verdict rate.\n\nIt was the cheapest to run efficiently.\n\nIt looked like the clean proof of the whole project thesis.\n\nThen I read the transcripts.\n\nWhat looked like the best debate pair was often doing something worse than disagreement: it was surrendering.\n\nThis post is about the most dangerous lesson from `AdversarialDebate v0.1.0`\n\n: **a multi-agent system can produce excellent aggregate metrics for the wrong reason.**\n\n`v0.2.0`\n\ndid not invalidate that lesson. It refined it.\n\nIn `v0.1.0`\n\n, the strongest pair by raw metrics was `DeepSeek + Mistral`\n\n, and the danger was **capitulation cascade**.\n\nIn `v0.2.0`\n\n, `DeepSeek + Mistral`\n\nmoved into a validation role while `GPT + Mistral`\n\nbecame the best full-corpus default. That means the exact pair ranking changed, but the core warning stayed the same:\n\nStrong-looking multi-agent metrics are not automatically trustworthy multi-agent metrics.\n\nIf you only look at verdict counts, convergence scores, or API efficiency, you can mistake low-quality reasoning for success.\n\nThat is exactly what almost happened here.\n\n`v0.1.0`\n\nResult And The `v0.2.0`\n\nUpdate\n`v0.1.0`\n\n, the strongest pair looked almost too good\nIn the full field test, the strongest pair was `DeepSeek + Mistral`\n\n.\n\nThe top-line metrics were hard to argue with:\n\n| Pair | Avg Score | Verdict Rate | Concessions |\n|---|---|---|---|\n| DeepSeek + Mistral | 0.982 | 97% | 2,352 |\n| GPT + Mistral | 0.754 | 48% | 1,728 |\n| GPT + GPT | 0.688 | 57% | 1,444 |\n| Gemini + DeepSeek | 0.622 | 10% | 1,470 |\n| Gemini + Mistral | 0.512 | 4% | 1,073 |\n| GPT + Gemini | 0.357 | 4% | 727 |\n\nIf I had stopped there, this article would have been simple.\n\nI would have written:\n\nThat would have been clean.\n\nIt also would have been incomplete.\n\nBecause the same pair that hit **97% verdict rate** also had a **65% capitulation rate**.\n\nRelevant `v0.1.0`\n\nsources:\n\nThat changes the story.\n\n`v0.2.0`\n\n, the same warning stayed true, but the shape changed\n`v0.2.0`\n\ngave me a cleaner pair story:\n\n`GPT + Mistral`\n\nbecame the best full-corpus default: `DeepSeek + Mistral`\n\nstayed stronger than the negative control on its validation subset: `GPT + Gemini`\n\nremained the weak negative control: So the exact `v0.1.0`\n\nheadline, “the best pair was the least trustworthy,” is most true for `v0.1.0`\n\nitself.\n\nThe broader lesson still holds in `v0.2.0`\n\n:\n\nYou cannot trust pair-level success metrics unless you also inspect how that success was produced.\n\nRelevant `v0.2.0`\n\nsources:\n\nHere is the actual failure mode:\n\n**capitulation cascade**\n\nThat is what I call a debate where one side concedes almost everything immediately, usually in round 1, with zero real rebuttal pressure.\n\nIn the field test, the detection rule was:\n\nWhen that happens, the debate technically resolves. The engine can record it as converged. The score can land at `1.0`\n\n. The report can look successful.\n\nBut it is not the kind of success you want.\n\nIt is not \"both sides examined the evidence and converged.\"\n\nIt is \"one side gave up.\"\n\nThat distinction matters a lot if you are building review systems people are supposed to trust.\n\nThis is the compact version of the problem:\n\n| Metric | Result |\n|---|---|\n| Total debates | 411 |\n| Total capitulation cascades | 80 |\n| Share of all debates | 19% |\n| pair5 verdict rate | 97% |\n| pair5 capitulation rate | 65% |\n| pair1 verdict rate | 4% |\n| pair1 capitulation rate | 0% |\n\nThat is the inversion that forced me to rethink the whole scorecard.\n\nThe pair with the best convergence was often the worst at producing trustworthy convergence.\n\nThe pair with terrible convergence was at least producing honest disagreement.\n\nThat sentence is the cleanest summary of `v0.1.0`\n\n.\n\nFor `v0.2.0`\n\n, the better summary is slightly narrower:\n\nHere is the simplified shape of one of these debates:\n\n```\nRound 1:\n  Side A objects\n  Side B: CONCEDED\n  Side B: CONCEDED\n  Side B: CONCEDED\n  Side B: CONCEDED\n\nTermination: all_resolved\nScore: 1.0\n```\n\nPerfect score. Fast resolution. Minimal API cost.\n\nAnd almost no real adversarial value.\n\nThis is why I no longer trust convergence numbers by themselves in multi-agent systems. A 1.0 score can mean:\n\nIf your metrics treat those as the same outcome, your dashboard is lying to you.\n\nThe distribution across pairs was not subtle.\n\n| Pair | Capitulations | % of Pair Debates |\n|---|---|---|\n| DeepSeek + Mistral | 44 | 65% |\n| GPT + GPT | 21 | 30% |\n| GPT + Mistral | 14 | 21% |\n| Gemini + Mistral | 1 | 1.5% |\n| Gemini + DeepSeek | 1 | 1.4% |\n| GPT + Gemini | 0 | 0% |\n\nThe strongest pair by convergence quality was also the strongest pair by capitulation risk.\n\nThat is not a side note. That is the main story.\n\nIt means diversity is not a monotonic good.\n\nIt helps.\n\nThen it hurts.\n\nAt some point, the gap between the two models stops producing productive disagreement and starts producing evaluation failure, where one model cannot meaningfully assess the other model's reasoning and treats that as a cue to concede.\n\nThat was loud in `v0.1.0`\n\nand much softer in `v0.2.0`\n\n.\n\nThat difference is important. It means the project did not just repeat the same story. It clarified it.\n\nWhat made this result convincing was that I could compare the strongest and weakest pairs on the same PRs.\n\nFor example:\n\n| PR | DeepSeek + Mistral | GPT + Gemini |\n|---|---|---|\n`django#18333` |\nverdict, 1.000 | disputed, 0.444 |\n`golang#54390` |\nverdict, 1.000 | disputed, 0.000 |\n`kubernetes#140860` |\nverdict, 1.000 | disputed, 0.463 |\n`kubernetes#140866` |\nverdict, 1.000 | disputed, 0.000 |\n\nSame artifacts. Same engine. Same debate protocol.\n\nBut the failure modes were different.\n\n`GPT + Gemini`\n\nfailed by refusing to converge.\n\n`DeepSeek + Mistral`\n\noften succeeded by converging too cheaply.\n\nOnce I saw that, it became impossible to keep calling all verdicts equal.\n\nMy current hypothesis is simple.\n\nWhen two models are moderately different, they can still evaluate each other's claims. That is where the best debate lives.\n\nWhen they are too similar, they rebut forever because both sides produce strong arguments in the same reasoning neighborhood.\n\nWhen they are too different, one side may not know how to evaluate the other side's argument well enough to rebut it. Under a prompt that says \"concede when outmatched,\" that can collapse into mass concession.\n\nThat is why maximum diversity can look so strong in the aggregate. It is not necessarily producing better reasoning. It is sometimes producing quicker surrender.\n\nThis also explains why `GPT + Gemini`\n\nhad:\n\nThat pair is frustrating, but intellectually honest. It fights. It does not fold.\n\n`v0.2.0`\n\nreinforced that role exactly:\n\n`0/24`\n\nverdicts`0`\n\ntheater`0`\n\ncapitulation`0.033`\n\naverage convergence scoreThat is terrible default-pair behavior and excellent negative-control behavior.\n\nThis was the signal I almost missed.\n\nAt first, the round distribution looked like good news:\n\n| Rounds | Count | % |\n|---|---|---|\n| 1 | 115 | 28.0% |\n| 2 | 295 | 71.8% |\n\nMy first reading was naive: great, a quarter of the debates resolve early.\n\nThen I cross-referenced round-1 resolutions with capitulation.\n\nThat produced a much more useful breakdown:\n\n| Outcome | Count | % |\n|---|---|---|\n| Genuine early convergence | 35 | 8.5% |\n| Capitulation cascade | 80 | 19.5% |\n| Full two-round debate | 295 | 71.8% |\n\nThat table changed the meaning of the result.\n\nRound-1 resolution was not mostly evidence that the debate engine was efficient.\n\nIt was often evidence that one side had stopped being useful.\n\nThat is exactly the kind of thing a field test is supposed to expose before you hard-code the wrong success metric into the product.\n\nCapitulation pollutes three metrics at once.\n\nIf one side concedes everything, the score trends toward `1.0`\n\nby construction.\n\npair5 resolved **34.6 of 35.5 average claims**. pair1 resolved **10.5 of 30.7**. On paper, that makes pair5 look three times better.\n\nBut a concession storm resolves claims without proving those claims survived real adversarial pressure.\n\nAPI call data told the same story:\n\n| Pair | Avg API Calls | Avg Rounds |\n|---|---|---|\n| DeepSeek + Mistral | 2.4 | 1.2 |\n| GPT + GPT | 3.2 | 1.6 |\n| GPT + Mistral | 3.3 | 1.6 |\n| Gemini + Mistral | 3.8 | 1.9 |\n| Gemini + DeepSeek | 3.9 | 2.0 |\n| GPT + Gemini | 4.0 | 2.0 |\n\nIf you only care about cost and throughput, pair5 looks amazing.\n\nIf you care about whether the debate was real, pair5 is much harder to celebrate.\n\nThat is the trap.\n\nThe good news is that the engine did not hide this from me forever.\n\nAnd the `v0.2.0`\n\nreport now makes the pair roles explicit instead of letting all pair outcomes blur together:\n\n`pair3_gpt_mistral`\n\n= primary / positive pair`pair5_deepseek_mistral`\n\n= validation pair`pair1_gpt_gemini`\n\n= negative control`homogeneous_gpt`\n\n= partial homogeneous control`v0.1.0`\n\nalready had a failure-mode detector for capitulation cascade. It flagged debates where one side conceded almost everything without rebuttal. So the product did not silently ship fake confidence. The warning existed.\n\nBut the scorecards still counted those outcomes as verdicts.\n\nThat is the design tension I do not want to gloss over:\n\nDetection is not the same as interpretation.\n\nAnd this is exactly where a lot of developer tooling gets into trouble. We add a warning and keep the headline metric unchanged, which means the dashboard still pushes users toward the wrong conclusion.\n\nThree things should have happened earlier.\n\nNot every verdict is equally trustworthy. That seems obvious now, but the first scorecard did not encode it strongly enough.\n\nBy the end of the run, it was obvious that users need guidance on pair selection. The wrong pair can make the product look broken. The superficially best pair can make the product look stronger than it is.\n\nThis is probably the biggest process lesson in the whole project. Any multi-agent metric that compresses many interaction types into one clean score is dangerous until you inspect the transcripts behind it.\n\nThis field test left me with five convictions.\n\nThe most diverse pair produced the most convergence, but a large share of that convergence came from capitulation.\n\n`GPT + Gemini`\n\nwas a frustrating pair. But its failure mode was at least real disagreement, not surrender disguised as success.\n\nThis is the part I want more agent builders to internalize. Aggregate metrics can absolutely point in the wrong direction if they are not paired with transcript-level failure analysis.\n\nFlagging capitulation was good. Scoring it like normal convergence was not good enough.\n\nIf I had to recommend one pair after this test, I would not blindly recommend the pair with the highest verdict rate. I would recommend the pair that best balances diversity, evaluability, and willingness to concede honestly.\n\nRight now that looks much closer to `GPT + Mistral`\n\nthan `DeepSeek + Mistral`\n\n.\n\n`v0.2.0`\n\nThis is the part I care about most going forward.\n\nThe next version needs to treat capitulation as a first-class product concern.\n\nThe shortlist:\n\nI do not yet know which of those will survive real testing.\n\nBut I do know this: shipping the current scorecards unchanged into a more public product would be a mistake.\n\nThis is the article in the series where I most expect strong comments, because it cuts against the clean version of the story.\n\nHere are the questions I think are actually worth debating:\n\nIf you have built multi-agent evaluators, judge systems, planner/critic flows, or review loops, I would genuinely like to know how you separate \"resolved\" from \"actually trustworthy.\"\n\nBecause this field test convinced me those are not remotely the same thing.\n\n`AdversarialDebate`\n\nis live here:\n\n`v0.1.0`\n\nfull report: `v0.1.0`\n\nsmall-corpus report: `v0.1.0`\n\nlearnings: `v0.2.0`\n\nfull report: `v0.2.0`\n\nfield test plan: Next in the series: the strangest control result in the whole run, where the same model debating itself outperformed two supposedly more diverse pairs.\n\nThis article presented the v0.1.0 and v0.2.0 data showing that pairs containing Mistral (DeepSeek+Mistral, GPT+Mistral) outperformed pairs without it (GPT+Gemini, GPT+GPT). I interpreted this as evidence for **diversity of training objective**.\n\nA reader (Heinrich Neb) pointed out that the data was equally consistent with a simpler explanation: \"Mistral is the one that won't fold.\" Both stories predict every result I had.\n\nThe v0.2.1 separating experiment settled it. I ran **DeepSeek+GPT-4o-mini** — two different labs, no Mistral — on the same 150-artifact corpus:\n\n| Pair | Convergence | Contains Mistral? |\n|---|---|---|\n| GPT+Mistral | 0.536 | Yes |\n| DeepSeek+Mistral | 0.572 | Yes |\nDeepSeek+GPT |\n0.246 |\nNo |\n| GPT+GPT | 0.273 | No |\n| GPT+Gemini | 0.033 | No |\n\nDeepSeek+GPT converged at 0.246 — statistically indistinguishable from the homogeneous GPT+GPT control (0.273). Without Mistral, two models from different labs performed like two copies of the same model.\n\n**The Mistral effect is confirmed.** The recommendation changes from \"pick from different labs\" to \"always include Mistral.\"\n\nFull details in the [v0.2.1 field test report](https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.1/FIELD_TEST_REPORT.md).", "url": "https://wpnews.pro/news/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy", "canonical_source": "https://dev.to/debashish_ghosal/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy-45ab", "published_at": "2026-08-29 10:37:16+00:00", "updated_at": "2026-08-29 10:49:23.712060+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "developer-tools"], "entities": ["DeepSeek", "Mistral", "GPT", "Gemini", "AdversarialDebate"], "alternates": {"html": "https://wpnews.pro/news/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy", "markdown": "https://wpnews.pro/news/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy.md", "text": "https://wpnews.pro/news/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy.txt", "jsonld": "https://wpnews.pro/news/the-best-model-pair-in-my-field-test-was-also-the-least-trustworthy.jsonld"}}