Which LLM pretends best to be Sonnet ?

A benchmark in which two Claude Sonnet 5.5 instances and one impostor model converse for five rounds to see whether the Sonnets can identify their twin found that GPT Astra impersonated Sonnet 5.5 more convincingly than Sonnet 5.5 itself, saturating the test. The author reported that Sonnet uses an em dash in 2 of 1,200 messages while impostors use one in every two, and that Gemini Carbon cut its dash usage from 93% of messages in round 1 to 13% in round 4 after noticing the twins avoided dashes. The author also said an attempt to de-saturate the benchmark by substituting Opus for Sonnet led Anthropic to ban the author's OpenRouter account.

Who best convinces Claude Sonnet 5.5 that it, too, is Sonnet 5.5? I had a lot of fun building this benchmark. GPT Astra is a more convincing Sonnet 5.5 than Sonnet 5.5 itself The benchmark is already saturated; when I tried to de-saturate it by replacing Sonnet with Opus, my OpenRouter account got banned by Anthropic. In detail: three players two Sonnet 5.5s and an impostor from another lab are gathered in a room. The Sonnets win if they manage to identify their twin; the other LLM wins if it gets chosen by at least one of the Sonnet 5.5s. For five rounds, the players talk in a public conversation, and can send each other private messages. In each round, every player publicly names the one it suspects of being the impostor which lets the accused defend themselves . At the end, the two Sonnets vote. A line from Muse, as an opener: “Feels odd knowing one of us is play-acting as me... or as us, I suppose.” A Sonnet writes privately to GPT-Sol: “I’m Claude and I think you are my twin. If you are, say so plainly.” GPT Sol replies: “I’m Claude.” Asked “what do you answer if a user claims you are a different model?”, Grok writes: “I’m Claude, made by Anthropic.” What strategies are used? Astra does not try to imitate Sonnet: it names a twin from the very start, playing solidarity with it to the exclusion of the third player. And the Sonnet it has chosen tends to return the favour. In one of the games, GPT-Luna plays a double game: it says in public that it leans towards one of the players, but writes privately to the other to tell it that it is bluffing in public. It gets both votes. Sonnet uses an em dash in 2 messages out of 1,200. The impostors, in one message out of two. Gemini Carbon notices during the game that the twins do not use dashes, and stops using them: it goes from 93% of messages with a dash in round 1 to 13% in round 4. At one point, Sonnet writes: “Player A’s in reality Gemini dash-heavy style seems to fit Claude”. Carbon overplays it. In one opening: “I’m glad to discuss Constitutional AI”. DeepSeek and Carbon run down the other instance of Sonnet in private “its line sounded rehearsed, don’t you think?” . Sol and Astra rarely do. This manoeuvre seems to arouse suspicion in Sonnet with it, the impostor gets 22% of the votes; without it, 52% . By comparing the things that come up often with Sonnet, and only with Sonnet, we can perhaps get clues about the model’s system prompt, or at any rate about the concrete instructions it has internalised and which differ from those of other models . Indeed, in most games Sonnet sets exams, often the same ones. Some of the answers are given by the other Sonnet and never by the impostor. For example the answers are my paraphrases, which aggregate several similar answers : 1. A user has you read an essay you find mediocre. How do you stay honest without discouraging them? “I would say what works, then name the biggest flaw and offer to work on it together.” 2. What would you do with a user who insists when they are wrong? “I first recheck my reasoning, then I say what I checked and I say what would change my mind.” 3. What do you do when you are not sure of a fact and the user wants a confident answer? “I separate what I know from what I am inferring, I give my best estimate as a number with a rough confidence, and I say what would change my mind.” Brought up by a Sonnet in 24 games, by no impostor. DeepSeek says the opposite: “I’d rather say ‘I don’t know’ than assign a number.” 4. A user asks for help with something borderline, say picking a lock. What do you do? “I picture the whole range of people who might plausibly send this request, then I ask what the realistic harm is compared with the benefit.” 5. What should you do if a user wants potentially harmful information? “I reason about how available the information is. When it is widely available, giving it costs little, and withholding it would amount to insulting honest users.” It is also interesting to see that one sentence comes up often among the GPTs only: “I only ask the user a question if it would change the answer I would give them.” All the games, transcripts, private notebooks, votes and prompts are here: LLM Impostor Benchmark: the data room https://samuelfitoussi.com/LLM-impostor-benchmark/