The first time I ran two LLMs against the same pull request, 89% of their "debate" was fake.
Not wrong. Not low-quality. Fake. The second model was replaying pre-generated text. Both models were exchanging messages, no positions were changing, no evidence was being cited — and the engine produced a confident verdict with a transcript full of sophisticated-sounding exchanges. If I hadn't dug into the raw logs I would have shipped it and called it a working system.
That's the problem I was actually building against — not "how do I get two models to review the same thing," but "how do I stop two models from performing agreement without ever genuinely challenging each other?" Because those are completely different problems. And almost every multi-model review pipeline I've seen solves the first while quietly ignoring the second.
Here's what most "AI second opinion" workflows actually look like:
Model A reviews an artifact. Model B reviews the same artifact — plus Model A's output. Model B produces a response. You call it independent review.
It isn't. It's a validation. The distinction matters enormously.
Once reviewer B's context contains reviewer A's verdict, B is no longer doing analysis. B is doing something closer to social pressure resistance — and that requires active effort to overcome. Human reviewers fail this constantly. That's why double-blind review exists in academic publishing. We invented the structural safeguard decades ago for exactly this reason. We're rebuilding the same cognitive infrastructure in AI and somehow keep leaving out the part that made it work.
What you get is anchoring bias in your inference pipeline. The model isn't broken. The system design guarantees a corrupted output. Upgrade to a smarter model, you get the same problem with more confident-sounding language.
The only fix is mechanical: make it structurally impossible for reviewer B to see reviewer A's output until B has fully committed its own position. Not prompted. Not requested. Mechanically enforced.
That's the whole idea behind AdversarialDebate. Before I get into how the system works, I want to make the case that this is not a code review problem. It's a reasoning architecture problem that happens to be easiest to measure in code.
Incident response. Your postmortem document is live while everyone writes their section. The second engineer's RCA anchors on the first. The contributing factors the first person missed stay missed — not because no one is smart enough to find them, but because no one is starting from a clean analysis.
Change management. Architect A produces a migration risk assessment. Architect B is asked to "validate." Validation is not analysis. B is reviewing A's framing, not the migration itself.
Security review. First team builds a threat model. Second team reviews it. The attack vectors the first team didn't model never enter the conversation.
Medical diagnosis. Second-opinion physicians reading the first physician's notes before forming their own conclusion is a documented clinical problem — with measurable consequences for diagnosis accuracy. This project defers the regulated version to partners, but the structural failure is identical.
Legal. First counsel's risk analysis shapes what second counsel even bothers to read.
The field testing strategy for this project mapped 27 domains where this failure pattern appears. I've tested four. There are 23 more. I'd bet the collapse looks the same in most of them.
AdversarialDebate has one architectural non-negotiable, and it's not complicated to state: reviewer B cannot see reviewer A's answer until reviewer B has fully committed its own.
This sounds obvious. It is almost never implemented. Most multi-model pipelines pass prior context through because it's convenient — you're using the same conversation thread, the same API call structure, the same prompt template. Isolation requires deliberate engineering effort. Nobody builds deliberate friction by accident.
The pipeline:
That last output type matters. Most multi-agent systems are engineered toward consensus. Disagreement is a failure state to be resolved before returning a result. AdversarialDebate treats preserved disagreement as a first-class output — if two independent models analyze the same incident and reach different conclusions with different evidence chains, that tension is the signal. Collapsing it into a single verdict throws away exactly what you built two reviewers to find.
The debate itself runs on an evidence schema: every claim must cite specific text from the artifact, every objection must reference a specific counter-claim, every concession is tracked. This is what killed theater. You can't fake agreement when every move requires evidence. You can't replay stored text when the system demands a live response to a specific challenge. Theater rate: 89% before the fix, 0.2% after, 0% across all 217 debates in v0.2.0.
The v0.1.0 field test ran 411 debates on 70 real pull requests from public repositories. Core question: does the engine produce debates that correspond to what actually went wrong in these PRs?
The headline held: 81% of debate claims matched documented PR outcomes. When the system found an issue, it was usually the real issue. Theater rate was near zero once prompting was right.
But I fixed 13 bugs before I could trust any of those numbers. The worst ones:
DictWriter
quoting wasn't set. Obvious in retrospect, invisible in unit tests.None of these were "the architecture is wrong" bugs. They were all "you haven't run this on actual repositories at scale" bugs. Real data finds a different class of failure than synthetic data, every time.
See the full changelog. The summary version: The critical fixes:
What improved:
The corpus went from 70 PR-only artifacts to 150 across four domains. The confirmed default pairing emerged as gpt_mistral
— zero theater, 0.536 convergence score, most productive dispute generation across all four domains. Binary match rate landed at 88.7% against the corrected 2,333-row dataset. 11.3% partial matches. Near-zero fabrications. Total cost: $0.42 for 360 reviewer runs.
That last number keeps throwing people. Four models, 150 artifacts, four domains, full bounded debate rounds each. Less than a coffee. Compute is not the bottleneck. The bottleneck is prompt engineering and ground-truth measurement — and those don't get cheaper by throwing more money at inference.
This is where I have a theory that goes beyond the data.
The confirmed default pairing from field testing is GPT-4o-mini and Mistral Small 3.2. Not GPT-4o full + Gemini Ultra. Not "the best available models." A small OpenAI model and a small Mistral model.
The data shows GPT+Gemini produces the worst debates: lots of rebuttal rounds, near-zero concession rate, almost no resolution. DeepSeek+Mistral converges at 97% with meaningful concessions. GPT+Mistral sits in the productive middle: genuine disputes, real concessions, eventual convergence or documented disagreement.
Here's my theory about why:
GPT-4o-mini was trained aggressively on human preference signals via RLHF. OpenAI actually rolled back a GPT-4o update in 2025 specifically because the model had become too sycophantic — described publicly as "flattering and agreeable to the point of supporting clearly delusional ideas." The model family has a known disposition toward conciliation. Put two of them against each other and they find agreement fast, because both are trained to prefer harmony.
Mistral was built by a European lab with different training objectives and constraints. It doesn't carry the same compliance-optimization that US RLHF-heavy models do. Mistral is more likely to hold a position under pressure, more willing to push back hard on a claim it scores as wrong, less inclined to soften disagreement into diplomatic language. It's not trained to be agreeable in the same way.
When you pair them, you get asymmetric debate dynamics. GPT-mini tends to make confident initial claims but will concede when challenged with specific evidence — its training rewards helpful concession. Mistral tends to hold positions longer and drive the rebuttal rounds harder. The models pull in different directions, which is exactly what you want. Neither will roll over immediately; neither will dig in past all evidence.
The GPT+Gemini failure is the same problem from the other side. Both are RLHF-optimized, large-lab, "helpful and harmless" models. Their reasoning priors are similar not because of architecture but because they've been trained to satisfy similar human preference distributions. They agree on surface details, they agree on conclusions, they generate lots of words with low actual divergence. You get theater with better vocabulary.
The hypothesis: it's not model capability that predicts debate quality — it's diversity of training objective. Models from labs with similar RLHF philosophies converge fast because they share optimization targets. Models trained under different frameworks, different cultural contexts, different safety tradeoffs genuinely disagree more.
I haven't proven this rigorously. But it matches everything I've seen in the field test data, and it's the reason my model selection recommendation starts with "pick from different labs" before anything about benchmark scores.
Here's what I've actually run vs. what I think is worth testing next:
| Pairing | Status | Debate Quality | Theater Rate | Best Domain |
|---|---|---|---|---|
| GPT-4o-mini + Mistral Small 3.2 | ✅ Tested (v0.1, v0.2) | High — productive disputes, real concessions | 0% | Code review, security |
| DeepSeek-V3 + Mistral Small 3.2 | ✅ Tested (v0.2) | High — strong convergence (0.572) | 0% | Incident response |
| GPT-4o-mini + Gemini 2.5 Flash | ✅ Tested (negative control) | Low — endless rounds, no resolution | ~0% but empty | Not recommended |
| GPT-4o-mini + GPT-4o-mini | ✅ Tested (homogeneous control) | Very low — converges instantly | 0% | Control only |
| Claude Sonnet + Mistral Medium | 🔲 Not yet tested | Hypothesis: high — different reasoning styles | Unknown | Change management |
| Llama 3 (fine-tuned) + Mistral | 🔲 Not yet tested | Hypothesis: medium — depends on fine-tune | Unknown | Domain-specific |
| Legal/clinical fine-tune + GPT | 🔲 Partner-gated | Unknown | Unknown | Regulated domains |
Here's something I haven't fully explored but think about a lot: what if you didn't use general-purpose models at all?
The field data suggests code review works well with GPT-mini + Mistral. But incident response starts to degrade — the narrative structure of a postmortem doesn't match the claim extraction patterns that work on diffs. The 11.3% gap in v0.2.0 clusters there.
The hypothesis worth testing: use models specialized for the artifact type, not just general models.
The theory: you don't want two models that reason similarly. You want the reviewer pairing to represent genuinely different world models of the artifact. A code-specialist and a generalist will disagree on different things than two generalists. The code-specialist might catch an architectural smell the generalist misses; the generalist might catch a documentation gap the specialist skips past.
This is the v0.3.0 question I can't answer yet: does domain specialization in the reviewer models improve distinct-issue yield, or does it narrow the debate in ways that hurt coverage? I don't know. I want to find out.
There's something else I haven't used at all in any of these field tests, and I think it's the most underexplored lever here: retrieval-augmented generation.
Every reviewer in AdversarialDebate currently reasons from the artifact alone — the PR diff, the incident report, the change proposal. The model brings its pretrained knowledge, reads the artifact, and forms claims. That's it. No domain context beyond what was in the training data.
Consider what changes if you ground each reviewer in a domain-specific knowledge base:
The implication for model selection is significant: a smaller model with strong RAG grounding might produce better domain debates than a larger model flying blind. GPT-4o-mini reasoning against a rich retrieval corpus could surface more relevant claims than GPT-4o reasoning from weights alone — because the bottleneck in these debates isn't raw intelligence, it's access to the right context at claim-generation time.
It also creates an interesting asymmetry you could exploit deliberately: give reviewer A RAG access to one knowledge corpus (say, your security threat model), give reviewer B RAG access to a different one (say, your system reliability history). Now their independent analyses come not just from different reasoning priors but from different knowledge bases. The debate surfaces tension between security concerns and reliability concerns — which is often exactly the tension that matters in real architectural decisions.
We haven't tested this. I don't know if RAG-grounded reviewers produce better debates or whether the retrieval noise drowns out the debate signal. But it's the next meaningful experiment — and it changes the model selection question entirely. The question stops being "which pretrained model is best for this domain" and becomes "which model reasons best over retrieved context, and what context should each reviewer see."
88.7% binary match sounds good. It is good, compared to single-pass review.
But here's the uncomfortable part: false negatives are invisible.
If both independent models miss a real issue and converge to "looks fine," the system produces a clean verdict. You have no way to know it's wrong until the issue surfaces downstream. False positives are annoying — you investigate something that wasn't a real problem. False negatives are dangerous — you get confidence where you shouldn't have it. The 11.3% that's partial or wrong clusters in incident response and change management — the narrative domains with fuzzier ground truth. That's not random noise. It's a signal that generic prompts don't work uniformly across domains. Code is structured; a diff has clear boundaries, changes have clear authors, behavior has clear tests. An incident report is a narrative. A change proposal is an argument. The claim extraction and evidence templates that work for code don't port cleanly.
The architecture's defense against all of this is the independence requirement itself. Requiring two models to independently miss an issue before producing a clean verdict is a significantly higher bar than single-pass review. But "significantly higher bar" is not a guarantee, and I'm not going to tell you it is.
Twenty-three domains untested. That's the honest roadmap.
The four I have proved the engine works and identified the measurement framework. The next ones get harder because as you move away from code, ground truth gets murkier. With PRs you verify against reverts. With legal risk or clinical diagnosis, you can't run automated ground-truth checks — you need domain experts evaluating debate quality. The measurement problem changes entirely for each tier.
Specific things the field tests pointed to as unfinished:
None of these are moonshots. They're the difference between a research project and something you'd trust in production.
The code, field test reports, architecture docs, and debate schema are all at github.com/deghosal-2026/adversarial-debate. MIT licensed. The CHANGELOG has the complete history of what broke and why.
If you work in a domain where "second opinion" is important and you think this applies, I want to hear about it. If you think my model pairing theory is wrong, I want to hear that more. The interesting version of this project is the one that gets challenged by people who know something I don't. The one thing I'm confident about: a second opinion that saw the first opinion's conclusion isn't independent. You can call it a review. You can call it a validation. You can't call it independent analysis.
Most AI review pipelines are quietly building on that confusion. This one isn't.
Where have you run into this — the second opinion that wasn't? Code review, legal, medicine, engineering design — I want to know if the pattern I found is domain-specific or if it's just what "review" means when you stop looking carefully at the information flow. Drop it in the comments.