Most AI "second opinions" are fake.
Not because there is no second model. Because the second model usually sees the first model's framing, assumptions, and conclusion before it does any real reasoning of its own. At that point, the system looks like review, but it is structurally biased toward agreement.
That bothered me enough to build AdversarialDebate, an open-source review engine where two LLMs analyze the same artifact in isolation, commit their reviews independently, and only then debate each other point by point.
I wanted to answer one practical developer question:
If you actually force independence, do you get better review quality, or do you just get a more expensive version of the same answer twice?
So I field-tested it on 70 real pull requests from projects like Kubernetes, Prometheus, Go, Rails, and Django. I ran 411 debates across 6 model pairs for a total cost of $0.53.
The result was more interesting than the original thesis:
This is the first post in a short series about what worked, what almost failed, and what I would change in v0.2.0
.
If you build agent critics, verifier loops, judge systems, or multi-model review flows, this is the part I think matters most: independence is not a prompt trick. It is a system property.
Before I get into the build, here is the shortest honest summary of the field test:
| Metric | Result |
|---|---|
| PRs in corpus | 70 |
| Debates run | 411 |
| Verdicts reached | 152 |
| Disputed outcomes preserved | 259 |
| Theater cases | 1 |
| Capitulation cascades | 80 |
| Ground-truth PRs with at least one matching claim | 49/49 |
| Total cost | $0.53 |
That mix matters. This was not a system that always converged. It was a system that usually either argued productively or told me the disagreement was still real.
The standard pattern in AI tooling is easy to recognize.
One model writes a review, classification, summary, or recommendation. A second model is then asked to validate it. But the validation prompt usually includes the first model's answer. Sometimes that's framed as context sharing. Sometimes as critique. Sometimes as self-reflection.
In all three cases, the same thing happens: the second model is no longer reasoning independently.
That matters because the failure mode we're trying to catch is often not missing information. It is prematurely accepted reasoning.
The PR that looks safe because the first explanation was clean.
The migration plan that sounds fine until someone independent asks about rollback.
The incident hypothesis that gets accepted because it is coherent, not because it is correct.
Human teams already know how to handle this. We use independent review, delayed judgment, and structured challenge. If two senior engineers review a risky change, we do not force one to read the other's opinion before writing their own first. We know that contaminates the review.
AI systems mostly ignore that lesson.
I built AdversarialDebate because I wanted an engine that preserved the useful part of multiple reviewers: separate judgment before interaction.
The non-negotiable rule in the project is simple:
Reviewer B cannot see reviewer A's output until reviewer B has fully committed its own review.
Not "should not." Not "please avoid bias." Cannot.
That meant I had to build independence into the architecture rather than rely on prompt wording.
The v0.1.0 loop looks like this:
That last output matters more than it first sounds.
I did not want a system that forced agreement because agreement is often the least trustworthy output in multi-agent systems. If both sides still disagree after structured exchange, the useful result is not a synthetic summary pretending everything is resolved. The useful result is a report that says:
That is much closer to how real engineering decisions get made.
v0.1.0
Actually Ships
The product in v0.1.0
is intentionally narrow.
It ships as:
The core components are straightforward:
I kept the MVP lean on purpose. No hosted service. No UI. No GitHub Action. No attempt to solve every review domain at once.
The only question v0.1.0
needed to answer was this:
Can two isolated LLM reviewers surface something a single reviewer would miss, or at least produce a disagreement artifact that improves a human call?
If that answer was no, the project should die early.
I did not want a benchmark made of toy examples or curated one-liners.
So I used real public PRs and documented outcomes. The final corpus covered 70 PRs across repositories like:
kubernetes/kubernetes
prometheus/prometheus
golang/go
django/django
rails/rails
I tested 4 models:
And 6 model pairs:
That produced:
The low cost matters. It let me run enough tests to discover uncomfortable truths instead of stopping at the first clean demo.
For 49 of the 70 PRs, I had a documented ground-truth outcome: revert reason, fix description, or advisory context.
I compared debate claims against those outcomes.
The top-line result:
That does not mean the system is magically correct. It does mean the debates were not wandering off into irrelevant nonsense. Even weak pairs were usually arguing about the right things.
This was the release bar that mattered most to me.
I did not need proof that debate solved review. I needed proof that independence plus debate could produce inspectably useful review artifacts on real code, not just neat demos.
It cleared that bar decisively.
Just as important, the engine was not forcing verdicts to get there:
That is the shape I wanted. If everything converges, I stop trusting the system. If everything deadlocks, the system is too expensive to justify. This landed in the middle: enough verdicts to be useful, enough disagreement to be believable.
The strongest field-test signal was not the PR itself. It was the pair.
Here was the ranking by average convergence score and verdict productivity:
| Pair | Avg Score | Verdict Rate |
|---|---|---|
| DeepSeek + Mistral | 0.982 | 97% |
| GPT + Mistral | 0.754 | 48% |
| GPT + GPT | 0.688 | 57% |
| Gemini + DeepSeek | 0.622 | 10% |
| Gemini + Mistral | 0.512 | 4% |
| GPT + Gemini | 0.357 | 4% |
The most diverse pair and the least diverse pair were not just a little different. They behaved like different systems.
On the same PRs, with the same engine, with the same prompts, one pair reached verdict after verdict while another stalled almost every time.
Here is what that looked like on real PRs:
| PR | Best Pair | Score | Worst Pair | Score |
|---|---|---|---|---|
kubernetes#140866 |
||||
| DeepSeek + Mistral | 1.00 | GPT + Gemini | 0.00 | |
kubernetes#141273 |
||||
| DeepSeek + Mistral | 1.00 | GPT + Gemini | 0.00 | |
prometheus#19492 |
||||
| DeepSeek + Mistral | 1.00 | GPT + Gemini | 0.00 | |
golang#54390 |
||||
| DeepSeek + Mistral | 1.00 | GPT + Gemini | 0.00 |
That is the part I keep coming back to. Same artifact. Same engine. Same debate rules. Completely different behavior depending on the pair.
That was the first big learning I did not fully expect: for adversarial review, model pairing is a first-order product decision, not a tuning detail.
If a user picks the wrong pair, they may conclude the system is mediocre even if the architecture is sound.
If you're building with multi-agent patterns, this should feel uncomfortably familiar.
We often think in terms of orchestration first:
Those roles matter. But this project pushed me to think harder about something lower in the stack:
What kinds of models can productively disagree?
That is a more specific question than "which model is smarter?"
An adversarial pair needs at least three things:
Miss any one of those and the debate quality degrades.
Too similar, and both sides stubbornly rebut forever.
Too different, and one side capitulates instead of engaging.
Too unstructured, and both sides perform disagreement without moving state.
I think a lot of agent builders are going to rediscover this the hard way as they move from demos to systems that people actually depend on.
Three things worked better than I expected.
The biggest architectural risk was fake independence. If reviewer sessions leaked context or if the revelation step happened too early, the whole product thesis would collapse.
The transcript structure and the control behavior held up under tests and field runs. The homogeneous GPT+GPT pair helped here too. If leakage had been happening, identical-model behavior would have looked far more suspiciously aligned.
I went in thinking verdicts would be the hero output.
By the end of the field test, I trusted structured disagreement more than fast agreement. The would_resolve_if
field is one of the best parts of the system. It gives the human an actionable next question instead of fake certainty.
The shape of the artifact is simple:
verdict: DISPUTED
unresolved:
- claim: "Expand/contract migration is safe for pr-482"
agent_a: sufficient as written
agent_b: needs lock-timeout analysis on orders table (~40GB)
would_resolve_if: load test at production row count
That is much closer to what a developer or reviewer actually needs. Not just "the agents disagree," but what they disagree about and what evidence would settle it.
411 debates for $0.53 changed how I think about evaluation scope. Cheap enough to test real behavior. Cheap enough to discover methodology mistakes. Cheap enough to iterate on design instead of protecting a fragile benchmark from bad news.
This release earned its keep mostly because the field test was willing to embarrass the original story.
I expected some pair effect. I did not expect pairing to shape the system more than the PR itself. That means user guidance for pair selection is a bigger product requirement than I originally thought.
The strongest pair by convergence later turned out to have a major quality caveat: capitulation cascades. One side would concede everything in round 1 without real rebuttal. That deserves its own article because it changes how aggregate success metrics should be interpreted.
The clean-venv install smoke test caught a runtime dependency problem: pyyaml
was only in dev dependencies even though the scripted reviewer path imported it at runtime. That is exactly the kind of issue release checks are supposed to catch, and exactly the kind of thing I should have detected earlier.
That was a good reminder: passing CI in the repo is not the same thing as being ready for pip install
in the wild.
It is also why I trust release checklists more than "it worked on my machine" confidence. The clean-venv smoke test caught something the happy-path build never would have exposed.
The field test had the same effect on the product itself. It found 14 issues during pipeline development. Twelve were real bugs I fixed. Two were not bugs at all; they were valid findings that contradicted my expectations. That is exactly what a useful release process should do.
v0.2.0
If I were restarting v0.1.0
, I would change three things.
The current result makes pair selection look obvious in hindsight, but I only learned the real diversity story by running the field test. In the next cycle, pair strategy needs to be a documented product surface, not just a research observation.
A system can produce lots of verdicts for bad reasons. That distinction should have been first-class in the initial scorecard.
The product idea, WBS, and field-test thinking were strong early. Some of the release-facing docs and artifact paths lagged behind the implementation and needed cleanup close to ship. That is survivable, but it increases release friction for no good reason.
The next version should not just add more adapters. It needs to deepen the truthfulness of the debate itself.
The highest-value directions right now look like:
I do not think the lesson of v0.1.0
is "multi-agent debate solved review."
I think the lesson is narrower and more useful:
if you want a second opinion from AI, independence must be enforced, not implied.
Without that, you're mostly building agreement with extra steps.
With it, you at least have a chance to get real adversarial pressure on a conclusion before a human trusts it.
That is a much smaller claim than the average AI launch post makes. It is also a much more defensible one.
I expect comments on this one, because the whole premise invites argument. Good.
These are the questions I think are worth debating:
If you've built critique loops, judge systems, planner/critic agents, or verifier flows, I want to hear where you've seen fake independence show up.
Because I think this problem is much more common than most agent demos admit.
AdversarialDebate
is live here:
Next in the series: the most important engineering lesson from the whole build, where three prompt lines dropped debate theater from 89% to 0.2%.