[v0.2.1 RELEASED]— Aug 28, 2026.[Release notes]·[Field test report]
v0.2.1 Key Finding:The Mistral effect is confirmed. DeepSeek+GPT (two different labs, no Mistral) converged at 0.246 — same as the homogeneous GPT+GPT control (0.273). Mistral, not lab diversity, drives productive debate. The recommendation changes from "pick from different labs" to "always include Mistral."
Also new in v0.2.1:Row-count invariants at all 5 pipeline seams (the 2,333->359 collapse is eliminated), false-negative measurement (1.7-3.4% missed-issue rate, first recall data ever), 55 new unit tests.
AdversarialDebate
v0.2.0 is released — v0.2.1 shipped Aug 28.
v0.2.0
field test report: v0.1.0
field test report: The strongest pair in my field test almost fooled me.
It had the best average convergence score.
It had the best verdict rate.
It was the cheapest to run efficiently.
It looked like the clean proof of the whole project thesis.
Then I read the transcripts.
What looked like the best debate pair was often doing something worse than disagreement: it was surrendering.
This post is about the most dangerous lesson from AdversarialDebate v0.1.0
: a multi-agent system can produce excellent aggregate metrics for the wrong reason.
v0.2.0
did not invalidate that lesson. It refined it.
In v0.1.0
, the strongest pair by raw metrics was DeepSeek + Mistral
, and the danger was capitulation cascade.
In v0.2.0
, DeepSeek + Mistral
moved into a validation role while GPT + Mistral
became the best full-corpus default. That means the exact pair ranking changed, but the core warning stayed the same:
Strong-looking multi-agent metrics are not automatically trustworthy multi-agent metrics.
If you only look at verdict counts, convergence scores, or API efficiency, you can mistake low-quality reasoning for success.
That is exactly what almost happened here.
v0.1.0
Result And The v0.2.0
Update
v0.1.0
, the strongest pair looked almost too good
In the full field test, the strongest pair was DeepSeek + Mistral
.
The top-line metrics were hard to argue with:
| Pair | Avg Score | Verdict Rate | Concessions |
|---|---|---|---|
| DeepSeek + Mistral | 0.982 | 97% | 2,352 |
| GPT + Mistral | 0.754 | 48% | 1,728 |
| GPT + GPT | 0.688 | 57% | 1,444 |
| Gemini + DeepSeek | 0.622 | 10% | 1,470 |
| Gemini + Mistral | 0.512 | 4% | 1,073 |
| GPT + Gemini | 0.357 | 4% | 727 |
If I had stopped there, this article would have been simple.
I would have written:
That would have been clean.
It also would have been incomplete.
Because the same pair that hit 97% verdict rate also had a 65% capitulation rate.
Relevant v0.1.0
sources:
That changes the story.
v0.2.0
, the same warning stayed true, but the shape changed
v0.2.0
gave me a cleaner pair story:
GPT + Mistral
became the best full-corpus default: DeepSeek + Mistral
stayed stronger than the negative control on its validation subset: GPT + Gemini
remained the weak negative control: So the exact v0.1.0
headline, “the best pair was the least trustworthy,” is most true for v0.1.0
itself.
The broader lesson still holds in v0.2.0
:
You cannot trust pair-level success metrics unless you also inspect how that success was produced.
Relevant v0.2.0
sources:
Here is the actual failure mode:
capitulation cascade
That is what I call a debate where one side concedes almost everything immediately, usually in round 1, with zero real rebuttal pressure.
In the field test, the detection rule was:
When that happens, the debate technically resolves. The engine can record it as converged. The score can land at 1.0
. The report can look successful.
But it is not the kind of success you want.
It is not "both sides examined the evidence and converged."
It is "one side gave up."
That distinction matters a lot if you are building review systems people are supposed to trust.
This is the compact version of the problem:
| Metric | Result |
|---|---|
| Total debates | 411 |
| Total capitulation cascades | 80 |
| Share of all debates | 19% |
| pair5 verdict rate | 97% |
| pair5 capitulation rate | 65% |
| pair1 verdict rate | 4% |
| pair1 capitulation rate | 0% |
That is the inversion that forced me to rethink the whole scorecard.
The pair with the best convergence was often the worst at producing trustworthy convergence.
The pair with terrible convergence was at least producing honest disagreement.
That sentence is the cleanest summary of v0.1.0
.
For v0.2.0
, the better summary is slightly narrower:
Here is the simplified shape of one of these debates:
Round 1:
Side A objects
Side B: CONCEDED
Side B: CONCEDED
Side B: CONCEDED
Side B: CONCEDED
Termination: all_resolved
Score: 1.0
Perfect score. Fast resolution. Minimal API cost.
And almost no real adversarial value.
This is why I no longer trust convergence numbers by themselves in multi-agent systems. A 1.0 score can mean:
If your metrics treat those as the same outcome, your dashboard is lying to you.
The distribution across pairs was not subtle.
| Pair | Capitulations | % of Pair Debates |
|---|---|---|
| DeepSeek + Mistral | 44 | 65% |
| GPT + GPT | 21 | 30% |
| GPT + Mistral | 14 | 21% |
| Gemini + Mistral | 1 | 1.5% |
| Gemini + DeepSeek | 1 | 1.4% |
| GPT + Gemini | 0 | 0% |
The strongest pair by convergence quality was also the strongest pair by capitulation risk.
That is not a side note. That is the main story.
It means diversity is not a monotonic good.
It helps.
Then it hurts.
At some point, the gap between the two models stops producing productive disagreement and starts producing evaluation failure, where one model cannot meaningfully assess the other model's reasoning and treats that as a cue to concede.
That was loud in v0.1.0
and much softer in v0.2.0
.
That difference is important. It means the project did not just repeat the same story. It clarified it.
What made this result convincing was that I could compare the strongest and weakest pairs on the same PRs.
For example:
| PR | DeepSeek + Mistral | GPT + Gemini |
|---|---|---|
django#18333 |
||
| verdict, 1.000 | disputed, 0.444 | |
golang#54390 |
||
| verdict, 1.000 | disputed, 0.000 | |
kubernetes#140860 |
||
| verdict, 1.000 | disputed, 0.463 | |
kubernetes#140866 |
||
| verdict, 1.000 | disputed, 0.000 |
Same artifacts. Same engine. Same debate protocol.
But the failure modes were different.
GPT + Gemini
failed by refusing to converge.
DeepSeek + Mistral
often succeeded by converging too cheaply.
Once I saw that, it became impossible to keep calling all verdicts equal.
My current hypothesis is simple.
When two models are moderately different, they can still evaluate each other's claims. That is where the best debate lives.
When they are too similar, they rebut forever because both sides produce strong arguments in the same reasoning neighborhood.
When they are too different, one side may not know how to evaluate the other side's argument well enough to rebut it. Under a prompt that says "concede when outmatched," that can collapse into mass concession.
That is why maximum diversity can look so strong in the aggregate. It is not necessarily producing better reasoning. It is sometimes producing quicker surrender.
This also explains why GPT + Gemini
had:
That pair is frustrating, but intellectually honest. It fights. It does not fold.
v0.2.0
reinforced that role exactly:
0/24
verdicts0
theater0
capitulation0.033
average convergence scoreThat is terrible default-pair behavior and excellent negative-control behavior.
This was the signal I almost missed.
At first, the round distribution looked like good news:
| Rounds | Count | % |
|---|---|---|
| 1 | 115 | 28.0% |
| 2 | 295 | 71.8% |
My first reading was naive: great, a quarter of the debates resolve early.
Then I cross-referenced round-1 resolutions with capitulation.
That produced a much more useful breakdown:
| Outcome | Count | % |
|---|---|---|
| Genuine early convergence | 35 | 8.5% |
| Capitulation cascade | 80 | 19.5% |
| Full two-round debate | 295 | 71.8% |
That table changed the meaning of the result.
Round-1 resolution was not mostly evidence that the debate engine was efficient.
It was often evidence that one side had stopped being useful.
That is exactly the kind of thing a field test is supposed to expose before you hard-code the wrong success metric into the product.
Capitulation pollutes three metrics at once.
If one side concedes everything, the score trends toward 1.0
by construction.
pair5 resolved 34.6 of 35.5 average claims. pair1 resolved 10.5 of 30.7. On paper, that makes pair5 look three times better.
But a concession storm resolves claims without proving those claims survived real adversarial pressure.
API call data told the same story:
| Pair | Avg API Calls | Avg Rounds |
|---|---|---|
| DeepSeek + Mistral | 2.4 | 1.2 |
| GPT + GPT | 3.2 | 1.6 |
| GPT + Mistral | 3.3 | 1.6 |
| Gemini + Mistral | 3.8 | 1.9 |
| Gemini + DeepSeek | 3.9 | 2.0 |
| GPT + Gemini | 4.0 | 2.0 |
If you only care about cost and throughput, pair5 looks amazing.
If you care about whether the debate was real, pair5 is much harder to celebrate.
That is the trap.
The good news is that the engine did not hide this from me forever.
And the v0.2.0
report now makes the pair roles explicit instead of letting all pair outcomes blur together:
pair3_gpt_mistral
= primary / positive pairpair5_deepseek_mistral
= validation pairpair1_gpt_gemini
= negative controlhomogeneous_gpt
= partial homogeneous controlv0.1.0
already had a failure-mode detector for capitulation cascade. It flagged debates where one side conceded almost everything without rebuttal. So the product did not silently ship fake confidence. The warning existed.
But the scorecards still counted those outcomes as verdicts.
That is the design tension I do not want to gloss over:
Detection is not the same as interpretation.
And this is exactly where a lot of developer tooling gets into trouble. We add a warning and keep the headline metric unchanged, which means the dashboard still pushes users toward the wrong conclusion.
Three things should have happened earlier.
Not every verdict is equally trustworthy. That seems obvious now, but the first scorecard did not encode it strongly enough.
By the end of the run, it was obvious that users need guidance on pair selection. The wrong pair can make the product look broken. The superficially best pair can make the product look stronger than it is.
This is probably the biggest process lesson in the whole project. Any multi-agent metric that compresses many interaction types into one clean score is dangerous until you inspect the transcripts behind it.
This field test left me with five convictions.
The most diverse pair produced the most convergence, but a large share of that convergence came from capitulation.
GPT + Gemini
was a frustrating pair. But its failure mode was at least real disagreement, not surrender disguised as success.
This is the part I want more agent builders to internalize. Aggregate metrics can absolutely point in the wrong direction if they are not paired with transcript-level failure analysis.
Flagging capitulation was good. Scoring it like normal convergence was not good enough.
If I had to recommend one pair after this test, I would not blindly recommend the pair with the highest verdict rate. I would recommend the pair that best balances diversity, evaluability, and willingness to concede honestly.
Right now that looks much closer to GPT + Mistral
than DeepSeek + Mistral
.
v0.2.0
This is the part I care about most going forward.
The next version needs to treat capitulation as a first-class product concern.
The shortlist:
I do not yet know which of those will survive real testing.
But I do know this: shipping the current scorecards unchanged into a more public product would be a mistake.
This is the article in the series where I most expect strong comments, because it cuts against the clean version of the story.
Here are the questions I think are actually worth debating:
If you have built multi-agent evaluators, judge systems, planner/critic flows, or review loops, I would genuinely like to know how you separate "resolved" from "actually trustworthy."
Because this field test convinced me those are not remotely the same thing.
AdversarialDebate
is live here:
v0.1.0
full report: v0.1.0
small-corpus report: v0.1.0
learnings: v0.2.0
full report: v0.2.0
field test plan: Next in the series: the strangest control result in the whole run, where the same model debating itself outperformed two supposedly more diverse pairs.
This article presented the v0.1.0 and v0.2.0 data showing that pairs containing Mistral (DeepSeek+Mistral, GPT+Mistral) outperformed pairs without it (GPT+Gemini, GPT+GPT). I interpreted this as evidence for diversity of training objective.
A reader (Heinrich Neb) pointed out that the data was equally consistent with a simpler explanation: "Mistral is the one that won't fold." Both stories predict every result I had.
The v0.2.1 separating experiment settled it. I ran DeepSeek+GPT-4o-mini — two different labs, no Mistral — on the same 150-artifact corpus:
| Pair | Convergence | Contains Mistral? |
|---|---|---|
| GPT+Mistral | 0.536 | Yes |
| DeepSeek+Mistral | 0.572 | Yes |
| DeepSeek+GPT | ||
| 0.246 | ||
| No | ||
| GPT+GPT | 0.273 | No |
| GPT+Gemini | 0.033 | No |
DeepSeek+GPT converged at 0.246 — statistically indistinguishable from the homogeneous GPT+GPT control (0.273). Without Mistral, two models from different labs performed like two copies of the same model.
The Mistral effect is confirmed. The recommendation changes from "pick from different labs" to "always include Mistral."
Full details in the v0.2.1 field test report.