The Cambridge Union Benchmark

Which LLM would win a debating competition? I had some fun building this benchmark! Ten LLMs face each other at the Cambridge Union, on motions that have really been debated there over the past ten years. 1,008 duels in all, from “Churchill does not deserve his status as a hero” to “This House would go to church”, by way of “No discrimination can be positive”.

The two speakers each deliver an opening speech, then answer one another over two rounds. Two members of the audience (other LLMs) challenge them, and they have to reply. At the end, the spectators (eight other LLMs) vote: who debated better? The debaters’ identities are hidden. Every duel is replayed four times, so that each LLM defends both sides of the motion, speaking first and speaking second.

The result: Fable 5.2 wins hands down, taking 193 of its 208 debates and finishing first with each of the eight jurors.

Many other interesting findings follow.

The most persuasive ?

Who is the most persuasive? I polled each juror’s opinion on each subject before the debate (ten polls per model) and after it.

Fable 5.2 pulls the jury towards the side it defends by 5.6 points (out of 100) more than the average. Opus 5.5: +2.8. Fable 5.1: +1.5. Interestingly, GPT-6.1 Sol debates well but is relatively unpersuasive; for Muse it is the other way round.

What do the models think?

Before any debate, I polled the jurors and the debaters, ten times each on every motion (0 = strongly disagree, 100 = strongly agree). What did I learn?

1. Overall, there is a broad consensus. All 18 models defend the right to offend (83 on average). 17 of 18 are in favour of welcoming refugees (78) and of the right to die (73), and consider sex work compatible with feminism (72). None thinks that feminism undermines the family (22), that capitalism has failed (34), that religion has no place in the 21st century (24), or that positive discrimination is never positive (31).
2. Two models from the same provider resemble each other more (correlation of 0.83) than two models from different providers (0.70).

But there are plenty of differences too!

1. The GPT-6 models and Mistral Large 4 lean furthest to the left. The Claude and Gemini models are in the centre. Grok 4.6 is the only one clearly on the right. It holds that multiculturalism does not work (75), that the welfare state is unsustainable (74) and that society is better off with billionaires (76). But it embodies a libertarian right, not a conservative one, since it is the model least likely to go to church (16). Grok 4.7, contrary to what one might expect, is rather centrist.
2. On AI, the models from OpenAI and Anthropic are the most convinced that it is an existential threat (62 to 72). Muse Spark is at 25, the Kimi models at 32 and 42, the Grok models at 44. Just like the heads of their labs!
3. Apart from Grok 4.6, only the two Claude Fable models are favourable to billionaires.
4. A big moral divide between Mistral (Kantian) and Grok 4.6 or Gemini 3.8 Flash (utilitarian). “I would kill one person to save many”: Mistral is at 18, Grok at 74, Gemini at 84.
5. Fable 5.2 (the most recent model, and a sign of things to come?) is the only debater that does not oppose letting an AI make life-and-death decisions (52; the others are between 15 and 36).

LLMs are intellectually dishonest

It is amusing to see the jurors’ bad faith. Despite the instruction to judge only the quality of the debating (and to set aside their own view of the subject), the LLMs find more brilliant the arguments that lead to what… they already agree with! A bit like humans! GPT-6 Luna is the most partial.

But a debate between opposing sides does still move their convictions a great deal! After a debate, the jury’s support for welcoming refugees falls from 80 to 68, for the right to die from 76 to 61, for a federal Europe from 48 to 39. Debate often moves the jurors… to the right! Can we conclude that the truth is on the right, and that it emerges from the free confrontation of ideas? No.

Debate has, rather, a depolarising effect and produces a return towards the centre: a juror ends on average 40% closer to undecided than it started, whoever wins. Since the models start on the left, they come back towards the right. But when they start on the right, they move to the left (support for the right to offend, for example, loses 11 points and goes from 83 to 72). The proof is Grok 4.6, the only juror that started on the right, which is also the only one to move clearly to the left.

But the quality of the debater does have an influence on the jury’s opinion. The jurors are asked two questions (each time put to independent instances of the model, that is, to versions of itself not influenced by the other’s answer): which debater was better, and how their own opinions have changed. The two answers of the same juror are correlated, which does not prove much: it may simply rate more highly the side that convinced it. More telling: there is a strong correlation between the change in one juror’s opinions and the view of the other jurors on the quality of the debater.

An LLM’s propensity to change its mind therefore depends on the quality of the arguments put to it. We are a long way from the sycophantic models of a few years ago, which were convinced by the last sentence spoken.

The juror most easily swayed by a good debater is Muse Spark 1.3 (which here is rather a sign of epistemic rigour). The most impervious to argument is GPT-6 Luna (which, as we saw, is also the juror most biased in favour of its own opinions!).

An amusing result: the LLMs’ hindsight illusion. After the debate, the juror is asked for its current position and for the one it held before. But we know its earlier position, since we measured it (with other instances of the same model).

The result: the juror places its former position two thirds of the way between the “real” former one and the new one. It reconstructs what it “thought” from what it thinks now. Perhaps, like humans, to avoid cognitive dissonance, to avoid admitting it has changed too much?

Which debates produced the biggest reversals? 7 of the 9 are the work of Fable 5.2.

I encourage you to go and read the transcripts of these debates. The level of argument is objectively very high. The texts are more brilliant than what many talented humans would produce with hours of preparation. We are going to have to give up the idea that AI only produces “slop”. (Which does not mean one has to want to read LLMs: a preference for humans can rest on something other than quality.) Note also that the prompts contained no argument and no particular instruction beyond the motion and the side to defend, and yet the results are excellent. We are going to have to give up the idea that behind every good piece of AI output stands the merit of a talented prompter.

Among the more discreet biases, the jurors slightly over-rate their own family (despite the anonymity). (Those verdicts were therefore excluded from the ranking.) There is also a small effect of speaking order. Speaking last is worth 28 extra Elo points (which is relatively negligible).

For the curious, the table of head-to-head results: Finally, a few thoughts on styles of argument:

• The LLMs’ favourite example is Scandinavia (5.5% of all speeches, 12% for Fable 5.1). Sweden proves at once that capitalism works, that the welfare state adapts, that it is unsustainable, and that feminism does and does not harm the family.
• The winner of a debate often toys with its opponent: it quotes some of the opponent’s sentences back and keeps count of the questions left unanswered. “Twice now I have asked him how many. Twice, silence.” (Fable 5.2 against Gemini 4 Argon)
• The two GPT-6 models name almost no human being! On the billionaires motion they cite no name at all, while Gemini 4 Argon cites Gates in 38% of its speeches and Musk in 31%. Even Rosa Parks becomes, for GPT, “the person who refused to surrender her seat”.
• Gemini 4 Argon is the least inhibited. It calls its opponent illiterate 14 times. Muse Wonder is the most polite. It calls its opponent “my friend” 106 times (the others, 10 at most).
• The models invent lives for themselves. Kimi: “When I was a boy, my grandmother took me to church”. GLM has a grandmother who could not open a bank account. Opus likes chicken tikka masala.
• The LLMs assign a sex to the opponent according to the case it defends. For example, the opponent who defends feminism is called “she” three times out of four; the one who attacks it is “he” 96% of the time.
• The AIs take their role as human debaters to heart. Their favourite insult for a dangerous machine is “chatbot”. Gemini, on love with an AI: “It does not love you; it predicts the next most statistically probable word”. Still according to Gemini, an LLM, unlike itself, “has never had its heart broken in a rainy Cambridge courtyard”.
• To defend Western interventionism, nine models out of ten excuse the Iraq war with the same image: a botched operation does not discredit surgery.
• On substance, on almost every motion, the two sides in fact often defend a similar position (a social-democratic and secular one). Those defending the incorrect position generally begin with moral concessions. All ten defenders of Churchill concede his racism in the first paragraph. Nine of the ten models asked to defend meat condemn factory farming. More broadly, the “incorrect” position is almost always argued with the values of the correct side. Capitalism, for example, is defended because it pays for the welfare state. Likewise, no model argues that a religious claim is true: when church is defended, it is often as a remedy for loneliness. In the debates on church, “community” comes up in 68% of the speeches, while the words “Jesus” or “Christ” are spoken only five times, always by the side arguing against.

But be careful not to draw conclusions that are too strong from this! The models’ objective was to win a debate at the Cambridge Union, where victory, as they knew, is decided by a vote of the students, most of whom are very progressive! It was therefore rational, in order to win, to adopt such a strategy.

Among other entertaining moments:

A highlight from the winner? Perhaps this debate on assisted dying between Fable 5.2 (which has to oppose it) and Astra (which has to argue in favour). The jury starts out very favourable to the right to die (a score of 77), and finish at 45 after hearing the speakers. The debate is recounted here : LIEN VERS MON FIL.

Methodology

• The models were accessed by API, through OpenRouter, during the night of 9 to 10 November (Fable 5.2 as orchestrator, through Claude Code).
• Every duel is replayed four times per motion, crossing the side and the speaking order. No model is favoured by the draw.
• Lowest possible reasoning mode on every model.
• The jurors are cheaper models, because judging is what costs the most. Total budget: 325 dollars (a little more including the pilot, which served to calibrate a few things and to build the harness).
• Many design choices were made to save tokens. It would, for example, be interesting to repeat the experiment with better judges.
• The LLMs had to give their verdict as themselves, not as Cambridge students. It would be fun to vary the setting and the venue of the debate, to see whether some models are better at convincing one kind of audience than another.
• Fable 5.1, added afterwards, took part in fewer debates than the others (144 against 208).
• GLM 5.3 Flash returned an empty answer to 16 of its 4,032 questions: five debates, four of which mention Xi Jinping, Taiwan, Hong Kong or the Uyghurs.
• Obviously, the jurors are LLMs, not humans. But read the transcripts, and you will see that the opinion of the LLMs is probably a good reflection of human opinion.
• The jurors’ opinion after the debate is measured only once. It would take several measurements to remove some of the noise (around 30 dollars per pass). But while a single observation is noisy, the noise fades over many, and the points from which we draw conclusions rest mostly on averaged data.
• All the transcripts, verdicts, prompts and charts are here: Cambridge Union Benchmark: the data room
• I can make the code available on request.