The Same Model Debating Itself Was More Self-Critical Than Two Different Models

A developer's field test of an adversarial debate system found that a homogeneous pair of the same model outperformed heterogeneous pairs, with GPT+GPT scoring 0.688 versus 0.357 for GPT+Gemini and 0.512 for Gemini+Mistral. The results challenge the assumption that diversity improves multi-agent debate, suggesting weak diversity can be worse than none.

v0.2.1 RELEASED — Aug 28, 2026. Release notes · Field test report v0.2.1 Key Finding:DeepSeek+GPT 0.246 convergence, no Mistral performed the same as GPT+GPT 0.273, homogeneous control . The distinction is not "diversity vs homogeneity" — it isMistral vs no-Mistral. The v0.2.1 separating experiment reframes this article's thesis. AdversarialDebate v0.2.0 is released — v0.2.1 shipped Aug 28 . v0.2.0 field test report: v0.1.0 field test report: I included a homogeneous pair as a control because I expected it to fail. If the whole thesis behind AdversarialDebate is that diversity helps two models pressure each other's reasoning, then the same model debating itself should be the weak baseline. Same training distribution. Same safety tuning. Same blind spots. Same defaults. That was the theory. The data said something much stranger. In the full field test, GPT + GPT outperformed both GPT + Gemini and Gemini + Mistral . Not by a rounding error. By enough that I had to stop and rethink what "diversity" actually means in a multi-agent system. This post is about the most counterintuitive result in the whole project: weak diversity can be worse than no diversity at all. v0.2.0 did not fully resolve this question because the homogeneous control stayed a partial sample, but it did reinforce the broader lesson: GPT + Gemini is still bad GPT + Mistral is the best full-corpus defaultHere is the shortest version of the result: | Pair | Avg Score | Verdict Rate | Avg Concessions per Debate | |---|---|---|---| | GPT + GPT | 0.688 | 57% | 20.9 | | GPT + Gemini | 0.357 | 4% | 10.5 | | Gemini + Mistral | 0.512 | 4% | 15.6 | The same model did not just hold up as a control. It behaved like a better debate partner than two supposedly more diverse pairings. Relevant v0.1.0 sources: Relevant v0.2.0 sources: Here is the comparison that forced the issue: | Pair | Diversity | Avg Score | Verdict Rate | Concessions | Capitulation | |---|---|---|---|---|---| | GPT + GPT | Same model | 0.688 | 57% | 1,444 | 21 | | GPT + Gemini | US + US | 0.357 | 4% | 727 | 0 | | Gemini + Mistral | US + EU | 0.512 | 4% | 1,073 | 1 | The homogeneous pair, the one I expected to be the most boring and least useful, beat two heterogeneous pairs. That result is still the clean v0.1.0 story. In v0.2.0 , I only ran a partial homogeneous control sample : So v0.2.0 does not replace the original article claim. It leaves it as a strong v0.1.0 result with a smaller supporting signal in the next release. That is not supposed to happen if your mental model is just "more diversity = better debate." What the field test actually suggests is more uncomfortable: That is not a clean ladder. It is a spectrum with a bad middle and a dangerous far end. The control was there to test whether the whole architecture was doing something real. If two copies of the same model reviewed the same PR and then debated, I expected one of two outcomes: If the system had leaked context between the two passes, the homogeneous pair would have exposed it immediately. If both sides quietly saw the same reasoning too early, the outputs would collapse into fake consensus. So GPT + GPT was useful for two reasons: I did not expect it to become one of the most interesting pairings in the entire run. The strongest evidence that the homogeneous pair was doing something real is that it did not converge all the time. From the field test: If the same model were simply echoing itself, that should not happen. If the isolation were fake, that should not happen. If the homogeneous control were trivial, that should not happen. Instead, the same model in two isolated contexts produced enough variance to create real disagreement, and enough recognizability to create meaningful concession. That is a very different dynamic from what I saw in the weakest heterogeneous pair. And the v0.2.0 negative-control rerun reinforced the weak-pair side of that comparison: GPT + Gemini negative control: GPT + GPT partial control: That is a smaller sample than I would like, but it still points in the same direction: weak diversity can be worse than no diversity. And it matters for product design. If your evaluation only compares different models to each other, you can miss the fact that one model in two isolated contexts may actually produce cleaner self-correction than two loosely similar models forced into a debate loop. Now look at GPT + Gemini : This pair almost always used the full round budget and still rarely converged. That does not mean the pair was useless. In many cases the claims still aligned with the real issue. But as a debate pairing, it was much less productive. This was one of the most important differences in the entire field test: GPT + GPT produced disagreement plus self-correction. GPT + Gemini produced disagreement plus stubbornness. That is a materially different product behavior. I cannot prove this yet, but I think this is what is happening. When GPT-4o-mini debates itself, both sides speak in reasoning patterns the other side can fully evaluate. They share not just training distribution, but the shape of their errors. That means one side can say, in effect: "I know exactly how you got here, and I can see where that line of reasoning fails." That is different from heterogeneous debate. With a different model, the other side may disagree without ever fully recognizing the internal move that created the original error. So the same-model pairing can do something surprising: it can become more self-critical because it is arguing in a language it understands perfectly. The best analogy I have is editing your own draft the next day. You are the same person. You still have the same habits. But the separation in context is enough to let you spot flaws that were invisible in the first pass. That is what the isolation architecture is buying here: two rooms for the same brain. The GPT + Gemini result forced me to stop thinking about diversity as a binary property. They are different models. They are not the same provider. They are not identical systems. And yet the debates were much less productive than GPT + GPT . My current read is that weak diversity creates the worst of both worlds: That creates a nasty equilibrium. Neither side yields. Neither side collapses. Neither side fully persuades the other. You get evidence-backed rebuttal without convergence. That can still be useful for a human reader, but it is a terrible default if your product promise is "structured debate that improves resolution quality." That is why I no longer think of model pairing as a model-selection problem alone. It is an interaction-design problem. That conclusion survived v0.2.0 cleanly. The pair roles in the current release are now explicit: pair3 gpt mistral = primary / positive pair pair5 deepseek mistral = validation pair pair1 gpt gemini = negative control homogeneous gpt = partial homogeneous controlThat makes this result easier to place in the broader system story. The clean thesis would have been: more diversity, more useful debate. The actual spectrum coming out of the field test looks closer to this: | Diversity level | Example | Avg Score | Dominant behavior | |---|---|---|---| | None | GPT + GPT | 0.688 | Self-criticism | | Weak | GPT + Gemini | 0.357 | Stubborn disagreement | | Moderate | GPT + Mistral | 0.754 | Productive debate | | Strong | DeepSeek + Mistral | 0.982 | Capitulation risk | That spectrum explains a lot of the confusion in multi-agent product design. If you test only one weak heterogeneous pair, you may conclude the system is not very good. If you test only the strongest pair, you may conclude the system is amazing while missing that some of the "success" is surrender. If you do not include a homogeneous control at all, you miss the fact that the middle of the spectrum may be worse than one side of it. That is why this result matters beyond this one project. It changes how I would design multi-agent evaluations in general. The lesson is not "same model good, different model bad." The lesson is that pair behavior has to be measured, not assumed from provider diversity. This result was not just interesting from a pairing perspective. It also strengthened my confidence in the system design. If the isolation architecture were leaking, the same-model pair would have been the first place to see it. Instead, the opposite happened. The pair produced enough genuine disagreement that 43% of the debates stayed disputed . That means the two independent passes were really independent. The same model, given the same artifact, still produced meaningfully different reviews across contexts often enough to create real debate. That is a big deal. It means the delayed revelation rule did more than prevent contamination. It preserved natural model variance as a source of useful pressure. That is one of the few places where the architecture itself made me more optimistic than the initial product thesis did. I do not want to oversell the homogeneous pair. It had 21 capitulation cascades , which is about 30% of the pair's debates. So this was not some magical perfect control. The same-model pairing still showed the same failure mode as the strongest diverse pair, just at a lower rate. And I did not run the flakiness sweep on the homogeneous pair specifically. That matters. One possible explanation for the result is not self-criticism but variance noise. If GPT + GPT flips more often than the stronger cross-model pairs, the 57% verdict rate may be less meaningful than it looks. I do not think that is the whole story, but I cannot rule it out yet. This is one of the places where v0.1.0 gave me a valuable result and an unfinished question at the same time. This article is partly about homogeneous control. It is also partly about Gemini. Across the field test, Gemini looked like the most stubborn model in the pool: | Model | Avg concessions per debate | |---|---| | GPT-4o-mini | 19.4 | | Mistral Small 3.2 | 25.3 | | DeepSeek-V3 | 27.9 | | Gemini 2.5 Flash | 15.9 | That does not mean Gemini was worse at identifying issues. It means Gemini was less willing to yield in debate. That distinction matters a lot. The practical implication from this run is simple: do not pair Gemini with a model that is close enough to fight forever but not different enough to break the stalemate. That pairing pattern gave me some of the least productive debates in the whole experiment. Three things stand out here. I added the homogeneous pair as a sanity check. In hindsight, it was one of the most important experimental conditions in the entire test. It deserved more explicit planning, more reporting emphasis, and its own dedicated stability check. I have strong evidence that the engine is stable overall. I do not yet have strong evidence that the homogeneous self-criticism result is stable specifically. That should have been part of the first release. US vs US was a useful shorthand while testing. It is not a complete explanation. OpenAI and Google are not the same model family, and I do not want to pretend geography alone explains pair behavior. The interaction could be about alignment style, response incentives, or something else entirely. This is what I take from the result. If the isolation architecture is real, the same model can still produce useful disagreement and meaningful self-correction. Too similar to unlock real blind-spot discovery. Too different to easily self-correct. Just enough overlap to argue forever. It is a behavioral systems question. You are not choosing the best model. You are choosing the best interaction. Without the homogeneous pair, this whole result disappears. I would have told a much cleaner and much less truthful story. Right now the safest practical recommendation looks like GPT + Mistral , not GPT + Gemini and not automatically DeepSeek + Mistral . Different enough to find blind spots. Similar enough to evaluate each other. Not so different that the weaker model just folds. That is a much more useful product insight than "use the pair with the highest score." This result changes the roadmap more than I expected. The next version needs: The long-term goal is not just to know whether a pair converges. It is to know whether a pair debates well. Those are not the same problem. This result is weird enough that I want people to push on it. If you have seen similar behavior in judge systems, critic loops, or verifier chains, I want to hear about it. Because this is exactly the kind of result that makes a product more useful and a blog post less clean. AdversarialDebate is live here: v0.1.0 full report: v0.1.0 small-corpus report: v0.1.0 learnings: v0.2.0 full report: v0.2.0 field test plan: Next in the series: the field test design mistake that turned out to be one of the best decisions in the whole release, adding a fourth model mid-run and changing what the data could say. This article argued that "weak diversity can be worse than no diversity at all" — GPT+GPT 0.688 outperformed GPT+Gemini 0.357 . The v0.2.1 separating experiment deepens that finding. v0.2.1 added DeepSeek+GPT — two models from different labs, different training regimes, different regions. If diversity of training objective drives debate quality, this pair should outperform the homogeneous control. It did not. | Pair | Convergence | Mistral? | v0.2.0 Baseline | |---|---|---|---| | GPT+Mistral | 0.536 | Yes | 0.536 | | DeepSeek+Mistral | 0.572 | Yes | 0.572 | DeepSeek+GPT | 0.246 | No | New in v0.2.1 | | GPT+GPT | 0.273 | No | 0.273 partial | | GPT+Gemini | 0.033 | No | 0.033 | DeepSeek+GPT 0.246 performed at the same level as GPT+GPT 0.273 . Two models from different labs converged like two copies of the same model. This reframes the article's thesis. The distinction is not "diversity vs homogeneity." It is Mistral vs no-Mistral. The GPT+GPT control looked surprisingly strong because the non-Mistral pairs were all weak — not because homogeneity is inherently good. The v0.2.1 data confirms: without Mistral, lab diversity alone does not produce productive debate. The v0.2.1 release also added: Full details in the v0.2.1 field test report https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.1/FIELD TEST REPORT.md . Two v0.2.2 findings deepen this article's thesis: Noise-floor baseline. The homogeneous gpt pair n=7 has a 95% CI of 0.129–0.432 — its true convergence score could be anywhere from "worse than GPT+Gemini" to "competitive with Mistral pairs." The GPT+GPT comparison in this article should have carried a confidence interval, not just a point estimate. Shared RLHF priors. The Mistral effect is confirmed, but the causal mechanism has two competing explanations. The positive story Mistral-specific training is the one this article tells. The negative story — non-Mistral models share RLHF priors and rubber-stamp each other — is now documented as an equally consistent alternative. The next experiment should test RLHF-distance directly. Full details in the v0.2.2 field test report https://github.com/deghosal-2026/adversarial-debate/blob/main/docs/field-test/v0.2.2/FIELD TEST REPORT.md .