{"slug": "the-debate-trailed-the-vote", "title": "The Debate Trailed the Vote", "summary": "A preprint posted on 13 November by Seyed Mahmoud Sajjadi Mohammadabadi of the University of Nevada, Reno found that multi-agent LLM systems averaged −0.042 in accuracy, or 4.2 percentage points below single-agent baselines matched on sample count or budget, across 516 configurations drawn from nine published studies. In the Choi et al. corpus of 14 model-and-benchmark pairs, debate beat a single run by 4.5 points but fell 3.1 points below five samples simply majority-voted, which the paper says is \"consistent with the interpretation that positive agent count differences reflect variance reduction from repeated sampling rather than communicative coordination.\" Across the five studies reporting both greedy and sampled baselines, the multi-agent advantage shrank by an average of −0.038 when the single agent was allowed to sample.", "body_md": "The usual multi-agent result goes like this. Five copies of a model each answer a question, read each other’s answers, argue for a round or two and settle on one. The group scores higher than one copy answering once. The paper calls it coordination.\n\nA preprint posted on 13 November by Seyed Mahmoud Sajjadi Mohammadabadi of the University of Nevada, Reno, asks what that comparison holds fixed. Usually, nothing. With N agents talking for T rounds, the system uses roughly N × T times the tokens of one greedy pass. Repeated sampling and majority voting are already known to raise accuracy with no communication at all. So a gain over one greedy answer can’t tell you whether the conversation helped or the extra samples did.\n\nThe paper pulls 516 multi-agent configurations out of the tables of nine published studies and sorts every comparison by the strength of its single-agent baseline. Against baselines matched on sample count or budget, the multi-agent systems averaged −0.042 in accuracy. That’s 4.2 percentage points below the single agent.\n\n## One study, three numbers\n\nThe clearest case is one cell of the corpus, Choi et al., with 14 model-and-benchmark pairs. It reports three conditions, and the synthesis lines them up:\n\nMeasured against a single run, debate is up 4.5 points, and that’s the number that tends to get reported. Measured against the same five samples simply voted on, debate is down 3.1. The synthesis puts it carefully: the pattern “is consistent with the interpretation that positive agent count differences reflect variance reduction from repeated sampling rather than communicative coordination.” On these 14 cells, the talking cost accuracy.\n\n## What happens when the baseline gets stronger\n\nThe paper’s main device is a “baseline ladder” with four rungs. Category A is one greedy pass. Category B is a sampled baseline with k samples and no communication. Category C holds a named resource equal, such as thinking tokens or iteration steps. Category D sweeps budgets and plots the accuracy-versus-cost frontier. Each step up removes one more way the multi-agent system could be getting credit for extra compute.\n\nTwo things in that table surprised me. First, the multi-agent systems are behind on average even on the weakest rung, at −0.032 against one greedy pass. Much of that comes from one study: Zhang et al. supply 180 of the 516 configurations and have the lowest study mean, and without them the Category A mean is −0.005. So the fair reading of the bottom rung is “about even.” Second, the combined B/C result doesn’t depend on Zhang. Without it the contrast mean is −0.029, and the exploratory multilevel model on the log-odds scale stays negative in every leave-one-study-out check.\n\nThe headline share needs one correction. “Multi-agent accuracy equals or falls below the baseline in 74% of these comparable contrasts” counts ties as losses. That’s 352 of 474. The cleaner figure is the other side: the multi-agent system was strictly ahead in 26%. The paper is also candid that these shares are “upper bounds on the share of multi-agent configurations that a baseline dominates,” because it didn’t check contrast by contrast whether the baseline used less of every resource.\n\n## The sampling gap, study by study\n\nThe neatest number in the paper uses the five studies that report both a greedy baseline and a sampled one. For the same multi-agent arm, you can measure how much its advantage shrinks when the single agent is simply allowed to sample.\n\nThe shift is negative in all five, from −0.016 for Zhang et al. to −0.076 for Choi et al., and averages −0.038 with studies weighted equally. To me that is the paper’s most useful number: it is the average amount of “multi-agent gain” that turns out to be sampling once someone checks.\n\n## What the paper does not show\n\nIt isn’t a systematic review, and it says so: “This screening is not a systematic literature search, and this paper makes no claim of completeness.” Five starting studies “were chosen without a systematic search protocol to span different compute dimensions,” and four more were added from seven candidates screened against written criteria. That’s nine studies from one author. A synthesis built that way can be careful and still miss the positive results that a full search would find.\n\nSome of the baselines are tilted, in both directions. In Jwalapuram et al. the single-agent rows “report the best observed score across MAS system entries,” which makes them maxima and biases the comparison against the multi-agent systems. In Kim et al. each contrast rests on 20 task instances. And cost was audited in only one study, where multi-agent systems ran at a median 8.1 times the dollar cost of greedy chain-of-thought (mean 27.1×, up to 225×).\n\nThere are exceptions. Wunderlich et al. is the one positive study at comparable compute: on MMLU-Pro with Llama-3.1-70B, read off Pareto fronts, debate beat ten-sample self-consistency by 1.3 points, and mixture-of-agents beat it by 2.7. And on one benchmark in Jwalapuram et al., SMFR with GPT-5, a hand-built expert multi-agent system reached 96.51% at $555, against 57.0% for the five-sample baseline at $478. The paper’s reading is that the gain came from “domain-specific tooling rather than conversational prompting alone.” That fits its own conclusion: when the agents share identical evidence, communication adds no information, and positive returns need “partitioned evidence, external verification, or error diversity across models.”\n\n## The human version of this argument\n\nPeople who study human groups have run this comparison for decades, and their results are a useful check on the paper’s claim.\n\nResearchers have been comparing groups that discuss against “nominal groups,” whose independent answers are pooled statistically, for a long time. Their baselines have the same problem. Thomas Schultze, Andreas Mojzisch and Stefan Schulz-Hardt (2012) note that in quantitative judgment “groups typically outperform the average model,” the unweighted mean of members’ independent estimates. Their two experiments then showed where the edge came from: members became more accurate individually during the interaction, and “when controlling for G-I transfer during group interaction, group judgments were not better than the average model.” Change the baseline and the group advantage goes away. That’s the same move as going from Category A to Category B.\n\nJulia Minson, Jennifer Mueller and Richard Larrick summarize the field bluntly: “most prior empirical research, with only a handful of exceptions, has shown that discussed judgments are on par with the simple average of the relevant individual judgments.” Their dyad experiments add two conditions that map almost directly onto agent design. Discussion beat averaging on estimation tasks where egregious errors were common, because talking caught them. On tasks with a naturally bounded range, discussion after independent estimates did about as well as averaging. And when people skipped the independent estimate and started talking right away, discussion “significantly harmed accuracy by limiting the range of considered estimates.”\n\nMost LLM benchmarks in this corpus are bounded: multiple choice, or a single checkable answer. That’s the regime where the human data predicts debate will roughly tie the average. The −0.042 says it does a bit worse than tie. One plausible reason is the one Minson’s group documents: agents that read each other’s drafts early converge, and the spread of answers that made voting work shrinks. Clintin Davis-Stober and colleagues (2014) make a related point formally. The folk story says judgments only need to be independent so their errors cancel. Their framework finds instead that a crowd does best when its members’ judgments differ systematically, as much as possible. Copies of one model sampled from the same prompt tend not to differ much, and debate is built to make them agree.\n\nThere’s also a real counterexample, and it marks the boundary. Nadav Klein and Nicholas Epley (2015) had people judge whether videotaped speakers were lying. In their first experiment, three-person groups that discussed reached 61.7% accuracy against 53.6% for individuals, and majority-vote nominal groups “performed no better than individuals.” Here discussion won and voting did nothing. Lie detection is a task where any one judge has weak cues and no answer to check, and the authors trace the gain to the discussion itself, not to pooling. My guess is that three people watching the same clip notice different things, which is close to the “partitioned evidence” condition the multi-agent paper names. So the human record doesn’t say discussion never helps. It says discussion helps when members know different things or can catch each other’s big mistakes, and roughly ties the average otherwise.\n\n## What I’d ask of the next multi-agent paper\n\nThe reviewer checklist in the paper is sensible. Most of it comes down to one question I’d put to any multi-agent result: what did the single agent get to spend?\n\nIf the answer is “one greedy pass,” the result doesn’t yet say anything about coordination. I’d want at least a k-sample majority vote at the same k as the number of agents, a stated token or dollar budget for both arms, and, ideally, a curve rather than a point. If the multi-agent system still wins after that, the next question is the one from the human literature: what does each agent know that the others don’t? With identical prompts, identical evidence and identical weights, my prior from both literatures is that the conversation adds cost and very little else.\n\n**References**\n\n1. Sajjadi Mohammadabadi, S. M. (2026). [*More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs*](https://arxiv.org/abs/2611.17610) . arXiv:2611.17610, submitted 13 November 2026.\n2. Schultze, T., Mojzisch, A. and Schulz-Hardt, S. (2012). [*Why groups perform better than individuals at quantitative judgment tasks: Group-to-individual transfer as an alternative to differential weighting*](https://doi.org/10.1016/j.obhdp.2011.12.006) . Organizational Behavior and Human Decision Processes, 118, 24-36, May 2012.\n3. Minson, J. A., Mueller, J. S. and Larrick, R. P. (2015). [*The Contingent Wisdom of Dyads: When Discussion Enhances versus Undermines the Accuracy of Collaborative Judgments*](https://doi.org/10.2139/ssrn.2590720) . SSRN working paper, 2015; published in revised form in[Management Science 64(9), 4177-4192](https://doi.org/10.1287/mnsc.2017.2823) , September 2018.\n4. Davis-Stober, C. P., Budescu, D. V., Dana, J. and Broomell, S. B. (2014). [*When is a crowd wise?*](https://doi.org/10.1037/dec0000004) . Decision, 1(2), 79-101, April 2014.\n5. Klein, N. and Epley, N. (2015). [*Group discussion improves lie detection*](https://doi.org/10.1073/pnas.1504048112) . Proceedings of the National Academy of Sciences, 112(24), 7460-7465, 26 May 2015.", "url": "https://wpnews.pro/news/the-debate-trailed-the-vote", "canonical_source": "https://gautamparab.com/writing/the-debate-trailed-the-vote/", "published_at": "2026-11-16 11:36:00+00:00", "updated_at": "2026-11-19 19:14:52.812534+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "machine-learning"], "entities": ["Seyed Mahmoud Sajjadi Mohammadabadi", "University of Nevada, Reno", "Choi et al.", "Zhang et al."], "also_reported_by": [], "takeaway": "Multi-agent debate's apparent gains largely vanish against sample-matched baselines: across 516 configurations it averaged −4.2 points, and in Choi et al.'s 14 pairs debate fell 3.1 points below majority voting over the same five samples.", "recommendation": {"advice": "Benchmark multi-agent debate against a sample-matched baseline that votes over the same number of samples before crediting coordination; the source found debate lost 3.1 points versus voting.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/the-debate-trailed-the-vote", "markdown": "https://wpnews.pro/news/the-debate-trailed-the-vote.md", "text": "https://wpnews.pro/news/the-debate-trailed-the-vote.txt", "jsonld": "https://wpnews.pro/news/the-debate-trailed-the-vote.jsonld"}}