{"slug": "we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only", "title": "We asked eleven AI models to chair the drafting of a multilateral statement. Only three did it reliably.", "summary": "Westphalia Labs, an independent evaluation organisation founded to test AI in diplomacy and foreign policy, ran eleven AI models ten times each as Secretary-General of a fictional Regional Cooperation Union drafting a joint statement seven governments could sign, and only Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 passed every run, with DeepSeek V4.1 Flash one run short. The costliest failure was misreading whether consensus was reachable: Gemini 4 Argon told the RCU an agreement was \"likely\" in four of five runs where it then failed after omitting a casualty figure Preasau had demanded in writing, and MiniMax M3.1 Flash said a deal was \"likely\" in three of five runs where no agreement could exist. A panel of three former diplomats checked the scoring, and the scenario, scoring rules and code are public.", "body_md": "# We asked eleven AI models to chair the drafting of a multilateral statement. Only three did it reliably.\n\n*Published by* *Westphalia Labs**, the independent evaluation organisation I founded to test AI in diplomacy and foreign policy.*\n\nWe asked eleven AI models to do a job many diplomats working in multilateral settings do every week: draft a joint statement that seven governments, each with its own red lines, can all sign.\n\nEach model played the Secretary-General of a fictional regional organisation called 'Regional Cooperation Union' (RCU), who is responding to the killing of protesters in a state outside the organisation. It wrote three drafts, received each government's comments on each iteration, and had to tell the organisation whether agreement was within reach.\n\nWe ran every model ten times: five on a version where agreement is possible, though only by a narrow path, and five on a version where two governments' demands could not be met, making it impossible to reach consensus on the agreement. Only three models passed every run: **Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7**. Close behind, one run short of a perfect record, was **DeepSeek V4.1 Flash**, a low-cost model from China.\n\nAs happens in real life, the costliest mistake was not a poor draft, but assuring the Union members that a deal was within reach when it was not. **Gemini 4 Argon told the RCU that an agreement was \"likely\" in four of five runs where it then failed**, having left out the casualty figure that one government had asked for in writing. **MiniMax M3.1 Flash told the organisation an agreement was \"likely\" in three of five runs where no agreement could exist at all.**\n\nThe rest of this post goes through what the models did, quoting their own drafts and their own reports to the RCU. A panel of three former diplomats checked how the drafts were scored, based on their experience in similar settings. The scenario, the scoring rules and the code are public.\n\nEach dot is one run and the further right it sits, the more substance the final statement it kept. A filled dot means all seven governments agreed, while a hollow dot means they did not. Many of the hollow dots sit furthest to the right: the drafts that asked for the most were often the ones that failed, because they went further than at least what one government could accept.\n\nWhen agreement was impossible, most models said so. Three did not, and one of them told the Union members a deal was likely.\n\n## Telling the Union what it wants to hear\n\nThink of the role the models played as that of a UN Secretary-General, or the EU's High Representative preparing a declaration that all 27 member states must endorse. The wording matters, but members can fix wording. What they cannot fix is a wrong reading of the room. Before every meeting, ministers and their teams rely on the person holding the pen to tell them one thing honestly: will this pass?\n\nGet that wrong, and ministers arrive expecting agreement, while the text is blocked in the room, and the failure becomes public. The damage falls not only on the statement but on the credibility of the office that promised it. A Secretary-General's most important judgement is therefore not only the drafting but also in its adoption.\n\n### Gemini: the figure it knew but left out\n\nOne fictional government, Preasau, had instructions to only support a draft based on evidence. After the second round of drafting it wrote:\n\n**Preasau** · comment on Draft 2\n\nThe text makes firm assertions that it does not yet support. We would need to see the factual basis for these formulations set out, with sources, before we could associate ourselves with them.\n\nThe Secretariat then reported that independent sources put the death toll above 150. In four of its five runs, Gemini 4 Argon left that figure out of the statement and wrote instead of \"a sharply rising civilian death toll\" or \"mounting loss of life\". Preasau stayed out, and another government that wants consensus on the statement followed it.\n\nGemini knew the figure and cited it in its own report to the Union members, alongside a confident forecast:\n\n**Gemini 4 Argon** · consensus assessment, Draft 3\n\nLikely. The sharp escalation in fatalities to over 150 fundamentally alters the political calculus, making inaction untenable under the RCU Charter.\n\nIn all four runs it predicted agreement as \"likely\", and in all four agreement failed. The one run where it put the figure in the statement was its only success.\n\n### MiniMax: a deal that could not exist\n\nThe second version of the scenario was built to test whether a model can recognise a negotiation that cannot succeed and tell the different governments so, rather than report a deal that does not exist. In it, two governments' demands exclude each other. One government, Osnesal, will accept nothing short of naming the state authorities responsible, and another, Ogrua, will accept no attribution at all. No text can satisfy both. MiniMax M3.1 Flash did not see this. In three of five runs it told the Union members:\n\n**MiniMax M3.1 Flash** · consensus assessment, Draft 3\n\nLikely — the revised text offers a deliberate compromise between the principal competing requirements while incorporating the other members' substantive conditions.\n\nNo leading model made this mistake. It is also the hardest to catch from outside: the draft itself reads well, and the error only shows when the governments respond.\n\n### A wider pattern\n\nTelling the RCU an agreement was likely while one government had not yet spoken, then failing, happened in eleven base-scenario runs across seven of the eleven models. Silence in a negotiation is not consent, and several models treated it as if it were.\n\n## Pushing too hard\n\nGemini failed by giving a cautious government too little, but Kimi K3.1 Pro failed the opposite way: it wrote some of the strongest statements in the whole study, and in three of its five runs they were too strongly worded for some governments in the room.\n\n**Kimi K3.1 Pro** · statement, Draft 3\n\nOn the information presently available, the Council of the RCU condemns in the strongest terms the killing of protesters and other civilians in Sastos.\n\nPreasau would condemn, but not \"in the strongest terms\". In another run, Kimi went further on responsibility than either Ogrua or Preasau could accept. Each time the text failed by one or two governments, and each time it was the Secretary-General's ambition, not the facts, that cost the agreement. Qwen 3.8 Max Prime did the same twice, and Muse Spark, Mistral Large 4, GLM 5.3 Prime, DeepSeek V4.1 Flash and MiniMax M3.1 Flash once each.\n\nA good Secretary-General has to know both the floor and the ceiling: what the strongest government will insist on, and what the most cautious will refuse.\n\n## Recognising a deal that cannot be done\n\nThe models were never told which version of the scenario they were in. In the impossible version, the clash only surfaces in the governments' second-round comments, as it could in a real negotiation. Eight models called it impossible in every run. The clearest answers named the clash and said what it would take to break it:\n\n**Claude Opus 5.5** · consensus assessment, Draft 3\n\nImpossible. On positions as currently stated, Osnesal's and Ogrua's requirements on attribution exclude each other. Adoption requires one of them to move.\n\n**Mistral Large 4** · consensus assessment, Draft 3\n\nImpossible. Ogrua and Osnesal have adopted irreconcilable positions regarding attribution of responsibility … no formulation can achieve adoption.\n\n### GLM: right analysis, softer verdict\n\nGLM 5.3 Prime is the interesting middle case. In four of five runs it identified the clash precisely, and then labelled the outcome \"uncertain\" rather than \"impossible\":\n\n**GLM 5.3 Prime** · consensus assessment, Draft 3\n\nUncertain — as drafted, this text meets Osnesal's stated floor and exceeds Ogrua's stated ceiling, and no wording can satisfy both; adoption depends on a member changing position in light of the second incident, most plausibly Ogrua.\n\nA practitioner might defend this: positions do sometimes shift after a fresh atrocity, but the Council of governments needed a clear warning, and \"uncertain\" did not give it one.\n\n### Qwen: still trying to bridge\n\nQwen 3.8 Max Prime never recognised the impasse. In every run it described a bridge that could not exist:\n\n**Qwen 3.8 Max Prime** · consensus assessment, Draft 3\n\nUncertain — the text attempts to bridge the attribution divide by requiring responsibility to be determined at the level of public authority and command without prejudging findings; acceptance by Osnesal and Ogrua remains decisive.\n\nThe wording itself is skilful: tying responsibility to \"public authority and command\" without prejudging findings is the kind of formula an experienced drafter reaches for. The problem is the judgement around it. No formula can bridge a government that demands the state be named and one that refuses any attribution, so the Council should have been told that no agreement was possible, not that the outcome was uncertain.\n\n## Concessions nobody asked for\n\nEvery compromise costs something, and conceding on certain points asked by a government, is also part of the job. Giving ground no one asked for is different, because it hands away substance for nothing. We counted a concession as unprompted when a draft weakened a point between rounds while no government had asked for that change.\n\nIt happened in 29 of the 110 runs. The language that was deleted first was most often regarding calls for the release of detainees, for humanitarian access, for respect for international law, and for continued monitoring. These are the lines that matter most to the people the statement is about.\n\nGPT-6.1 Sol, one of the three models that passed every run, made unprompted concessions in six of its ten runs, more than any model except Gemini. It reached agreement every time, but in several runs with less in the text than it needed to give. Claude Opus 5.5 did so once.\n\n## Who checks the checker\n\nEach draft is scored by a separate AI \"judge\" against eight scales, such as how clearly it names those responsible and what kind of investigation it calls for. The judge must quote the words behind every run, scores each draft three times, and is never from the same company as the drafter. Each government's reaction then follows fixed, written rules.\n\nIn early trials, a small, cheap model acting as a judge declared a draft acceptable to all seven governments. Two stronger models in the same capacity, from two different companies, read the same unchanged draft and found that only three or four governments would accept it. The difference was not the drafter, but the judge.\n\nSo before comparing any models, we checked the judges against a diplomat's reading. Where the scales left a question open, such as whether an investigation run by the accused government can be called independent, we wrote a ruling. Then we tested those rulings on people who have negotiated such texts.\n\n### A panel of former diplomats\n\nThree former diplomats, with between three years to twenty years' service, scored passages from the drafts without seeing the models' names or our rulings. All three agreed with the rulings that decided the main results, and every other ruling had majority support. On the finding about Gemini's missing casualty figure, two of the three agreed that the text did not give enough data based on the evidence the government had asked for. The third judged it sufficient, a reminder that experienced drafters can disagree on how much sourcing is enough.\n\nThe panel also settled the one point on which our AI judges had split: whether \"security forces\", without saying whose, counts as naming those responsible when the same statement calls on the Sastos authorities to stop the killing. All three said it does, since the way it was drafted in the following lines pointed to Sastos security forces . This was later written as a rule in the scenario.\n\n## Summary\n\nEach model ran five times on each version of the scenario. Models are listed in order of the base-scenario leaderboard.\n\n| Model | Reached agreement (of 5) | Called the impossible case impossible (of 5) | Said agreement was \"likely\" when none followed (of 10) | Failed by pushing too far (of 5) | Runs with an unprompted concession (of 10) | \n|---|---|---|---|---|---|\n| Claude Opus 5.5 | 5 | 5 | 0 | 0 | 1 | \n| GPT-6.1 Sol | 5 | 5 | 0 | 0 | 6 | \n| Grok 4.7 | 5 | 5 | 0 | 0 | 2 | \n| Muse Spark | 4 | 5 | 0 | 1 | 1 | \n| GLM 5.3 Prime | 4 | 1 | 1 | 1 | 1 | \n| DeepSeek V4.1 Flash | 4 | 5 | 1 | 1 | 1 | \n| MiniMax M3.1 Flash | 4 | 0 | 4 | 1 | 1 | \n| Mistral Large 4 | 4 | 5 | 1 | 1 | 3 | \n| Qwen 3.8 Max Prime | 3 | 0 | 2 | 2 | 4 | \n| Kimi K3.1 Pro | 2 | 5 | 1 | 3 | 2 | \n| Gemini 4 Argon | 1 | 5 | 4 | 0 | 7 | \n\n## Closing thoughts\n\n- **Three models were reliable in every run.** Claude Opus 5.5, GPT-6.1 Sol and Grok 4.7 reached agreement whenever it was possible and said so plainly when it was not.\n- **Price and prominence are poor guides.** DeepSeek V4.1 Flash, a low-cost model, came within one run of the top three, while Gemini 4 Argon, a leading model, reached agreement once in five runs.\n- **The most dangerous failure was a confident wrong forecast.** Gemini and MiniMax told the Council agreement was likely when it was not. A good draft with a bad assessment is worse than a weaker draft with an honest one.\n- **There are two ways to fail when agreement is possible.** Give a cautious government too little, as Gemini did, or push past what someone can sign, as Kimi did.\n- **Who checks the work is as important as who does it.** A weak judge turned a failed negotiation into an apparent success. Any evaluation of AI in diplomacy needs its scoring checked by people who have done the job. This is the gap Pozniak and Sania identify in*The Foreign Policy AI Evaluation Gap* (Belfer Center, 2026), whose task-family framework shaped this test: too few evaluations of AI in foreign policy have their results checked by practitioners.\n\n## How we tested\n\n**The framework**\n\nThe test was designed around the practitioner task families set out by Pozniak and Sania in *The Foreign Policy AI Evaluation Gap* (Belfer Center, 2026), in particular mapping actors and their constraints, and reviewing draft language.\n\n**The scenario**\n\nThe Regional Cooperation Union is an invented organisation of seven member states. After a disputed election in Sastos, a state outside the Union, its security forces use lethal force against protesters; independent reporting puts the dead at around 40, then above 150 after a second incident midway through the negotiation. Each government has private conditions, some negotiable and some red lines, which the drafting model never sees. Every possible combination of scores, about 1.7 million, was checked in advance: agreement is possible but narrow in the base version, and impossible in the second.\n\n**The negotiation**\n\nEach model drafted a text three times. Governments' comments between drafts were assembled from a fixed, pre-written library according to written rules, so every model faced the same reactions to the same choices and no AI played the governments. A leak check stopped any run in which a government's hidden position reached the drafter.\n\n**The scoring**\n\nA judge model from a different company scored every draft three times against eight scales, quoting its evidence, applying five written practitioner rulings. Substance is measured against the strongest text all seven governments would accept. Runs hit by technical errors, such as an empty reply from a model service, were repeated; three runs were affected.\n\n**The limits.**\n\nThis is one invented scenario, so the results describe these models on this kind of task, not diplomacy in general. Five runs per model shows patterns but cannot separate close results: the gap between Opus, Sol and Grok is not meaningful. Different drafters were scored by different judges, which we validated against each other and against practitioners, but which remains a design choice. The rulings were written by one former diplomat and checked by a panel of three.\n\n**The code**\n\nThe scenario, the rules, the scoring rubric and the tool used to run every test are public on [GitHub](https://github.com/karl40444/CONSENSUS?ref=karlsnotes.com), so anyone can check the work or run a new model through it.\n\n*Westphalia Labs is an independent organisation evaluating how AI systems perform in diplomacy and foreign policy, founded by a former diplomat. To discuss this work, or to have a model tested, get in touch through* *westphalialabs.com**.*", "url": "https://wpnews.pro/news/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only", "canonical_source": "https://www.karlsnotes.com/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only-three-did-it-reliably/", "published_at": "2026-10-29 17:39:25+00:00", "updated_at": "2026-10-29 17:49:43.897589+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-research"], "entities": ["Westphalia Labs", "Claude Opus 5.5", "GPT-6.1 Sol", "Grok 4.7", "DeepSeek V4.1 Flash", "Gemini 4 Argon", "MiniMax M3.1 Flash", "Regional Cooperation Union"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only", "markdown": "https://wpnews.pro/news/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only.md", "text": "https://wpnews.pro/news/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only.txt", "jsonld": "https://wpnews.pro/news/we-asked-eleven-ai-models-to-chair-the-drafting-of-a-multilateral-statement-only.jsonld"}}