A social deduction game benchmark tests how well AI models scheme, bluff, and reason together, revealing gaps in deception and trust.
What is the Blood on the Clocktower AI benchmark? #
It’s a custom evaluation built around the social deduction game Blood on the Clocktower, where AI models play full matches against each other, split into “good” and “evil” teams, and get scored on an ELO-style ladder based on win rates. One creator spent roughly $20,000 in API credits running 368 games to rank frontier models like GPT-6 Astra and Claude Opus 4.6 and 5.5 on their ability to bluff, coordinate secretly, and detect deception in a multiplayer setting that normal benchmarks don’t touch.
TL;DR #
- The benchmark pits AI models against each other in Blood on the Clocktower , a Werewolf-style game where an evil minority tries to eliminate a good majority before getting caught.
- GPT-6 Astra currently tops the leaderboard, beating Claude Opus 4.6 in about 92% of matches, a gap of roughly 430 ELO points.
- Over six months, OpenAI’s models jumped nearly 580 ELO points between GPT-5.4 and GPT-6 Astra, mostly by getting much better at playing thegood side rather than the evil one.
- The core improvement is evidentiary skepticism : newer OpenAI models treat individual clues as one data point among many, while Anthropic’s models have tended to accept reported information at face value.
- Specific game logs show models pulling off multi-step bluffs , like a demon posing as a helpful investigator role to earn trust it later exploits to wipe out the good team.
- As both sides get smarter, evil’s win rate drifts only slightly lower, a sign the benchmark resists saturation since the difficulty bar rises alongside model capability.
- Mixed teams of different models performed no better, and arguably slightly worse , than single-model teams, though the sample size was too small to be conclusive.
- 1 Secure
- 2 Regulation-compliant
- 3 Auditable
- 4 Scalable
1 SecureAdjusters can't export patient data✕ Export blocked
2 CompliantSSN
•••-••-4821✓ HIPAA ✓ SOC 2 3 Auditable09:14 · M. Chen approved09:12 · J. Ruiz edited09:07 · A. Patel viewed
4 Scalable12,480users online
How does the game actually work? #
Blood on the Clocktower splits players into two factions: good and evil. At the start, everyone learns their secret role, which comes with a special power. Evil players know their teammates, including the demon, who is the faction’s lead killer. Good players know nothing about who’s on which team.
Play alternates between day and night phases. During the day, players discuss and vote to execute one person. At night, the demon secretly kills someone. Evil wins if it kills enough of the good team; good wins if it executes the demon first.
That structure forces very different skills out of each side. Evil players need to construct a consistent, convincing story under live questioning, adapt their lies as new information surfaces, and sometimes sacrifice a teammate to protect the demon. Good players need to pool fragmented, sometimes false information and reason collectively about who’s lying, all while suspecting that any given piece of evidence might be corrupted by design.
That mix (bluffing under pressure, tracking long chains of claims, reasoning in a group with hidden adversaries) is exactly what makes the game useful as an AI benchmark. Few existing evaluations measure sustained deception and multi-agent trust-building at the same time.
Which AI models are best at the game right now? #
According to the benchmark’s results, GPT-6 Astra sits at the top of the leaderboard. The first flagship model tracked was Claude Opus 4.6, released in February. GPT-6 Astra followed seven months later and became the strongest performer measured, beating Opus 4.6 in about 92% of matches, a gap of roughly 430 ELO points.
The ranking methodology ran whole-team matchups: one model’s instances made up the entire good team, another model’s instances made up the entire evil team, repeated across hundreds of games to get a stable score. That setup isolates how well a given model handles both sides of the game rather than mixing in confounds from teammates with different reasoning styles.
The more striking trend isn’t a single model’s score, it’s the trajectory. As recently as March, OpenAI’s GPT-5.4 trailed Anthropic’s Opus 4.6. By September, GPT-6 Astra had gained around 580 ELO points over GPT-5.4, enough to win about 97% of games against its own predecessor. That’s a large swing in a matter of months, driven mostly by one specific skill.
Why are OpenAI’s newer models pulling ahead? #
The gap isn’t symmetric. Astra’s advantage over Anthropic’s models shows up far more strongly when Astra is playing good than when it’s playing evil. In other words, the newer OpenAI models haven’t necessarily gotten dramatically better at lying, they’ve gotten much better at not being fooled.
The mechanism seems to be skepticism toward individual pieces of evidence. OpenAI’s recent models tend to treat any single claim or outcome as one data point to weigh against the whole pattern of events, rather than as settled fact. Anthropic’s models, by comparison, have shown a tendency to accept reported information more at face value.
production-readysoftware.
One documented example makes this concrete. In a role called the Slayer, a good player gets one chance per game to name who they think the demon is; if correct, the demon dies on the spot. In a mirrored pair of games using the identical role draw (once with Claude on the good team, once with GPT models on the good team), the Slayer took a shot at the real demon and missed, because an evil player had secretly disabled the Slayer’s power overnight.
When Claude instances were on the good side, they treated the missed shot as proof the suspected player was innocent and moved on. When GPT instances were on the good side, they refused to treat the miss as conclusive, reasoned that the failure looked suspiciously convenient, worked out that sabotage was more likely than bad luck, and ultimately executed the actual demon.
The benchmark also tracked this more directly: each day, good players were privately asked to estimate the probability that each other player was the demon. OpenAI’s models converged on the correct answer more accurately and faster as games progressed than Anthropic’s models did.
Does getting smarter favor good or evil? #
Not clearly either, based on the data gathered so far. There were two plausible hypotheses: smarter play might favor good, since better reasoning should eventually expose any inconsistency; or it might favor evil, since a smarter liar can always patch holes in a story. The benchmark found evil’s win rate ticks down slightly as both sides scale up in capability, but the effect is small enough that it’s hard to distinguish from noise.
That’s actually a useful property for a benchmark to have. It suggests the evaluation won’t saturate quickly, because the difficulty scales with whatever models are current. A benchmark where the strongest available model always sets the bar for its opponents stays meaningful longer than a static test with a fixed ceiling.
One side experiment also tested whether mixed teams, made of different models playing together, might perform better than single-model teams due to a wider range of reasoning styles acting as a hedge against shared blind spots. A hybrid team of roughly even numbers of Opus 5.5 and a GPT-5.6 variant played about 130 games against top single-model teams, and the mixed team performed somewhat worse, though not by a statistically significant margin.
Is this a realistic measure of deception and reasoning in AI? #
It’s a reasonable proxy, with caveats. The game format forces exactly the behaviors that matter for evaluating deception risk: constructing a false narrative that survives scrutiny, adapting it as new facts emerge, coordinating secretly with other agents, and reasoning under uncertainty about who might be lying. Watching models pull off genuinely creative multi-step plays, like a demon posing as a trustworthy information-gathering role specifically to earn credibility it later cashes in, shows this isn’t just pattern-matching on game mechanics.
pass the audit.
PASS
PASS
PASS
PASS
PASS
PASS
0
The caveat is that the benchmark depends heavily on a custom game engine built to replace the human “storyteller” role that normally exists in the physical game. In the tabletop version, a human referee uses judgment to steer ambiguous situations, which would introduce bias into an AI evaluation. The builder replaced that with a programmatic system, essentially a probability table tuned to mimic experienced human storytelling, so that outcomes remain fair and consistent across hundreds of automated games. Getting that balance right reportedly took over 20 rounds of revision, with AI models themselves doing much of the iteration: writing game logic, running batches of test games, and reviewing transcripts for unfair or unrealistic outcomes before refining the rules further.
That’s a meaningful dependency. The benchmark’s validity rests on how well that simulated storyteller matches real game balance, which is harder to verify externally than a fixed dataset or test set.
Frequently Asked Questions #
What is Blood on the Clocktower?
It’s a social deduction board game similar to Werewolf or Mafia, where players are secretly split into good and evil factions. Evil tries to eliminate good players through a hidden “demon” killer, while good players share and reason about information to find and execute the demon first.
Which AI model currently performs best at this benchmark?
GPT-6 Astra, released in September, tops the leaderboard, winning about 92% of games against Claude Opus 4.6, a gap of roughly 430 ELO points, based on games run specifically for this evaluation.
Why does OpenAI’s model lineup score higher on the good team specifically?
The benchmark data suggests OpenAI’s newer models are more skeptical of individual pieces of evidence and weigh them against the full pattern of events, rather than accepting claims or outcomes at face value, which helps them catch sabotage and inconsistency faster than Anthropic’s models have.
Does this benchmark risk becoming obsolete as models improve?
It appears resistant to saturation so far, since evil’s win rate only drifts slightly as both sides get smarter, and the difficulty is set relative to whatever models are currently competing rather than a fixed target.
Can mixing different AI models on one team improve performance?
A small test with a team split between two different models performed somewhat worse than single-model teams, though the sample size (about 130 games) was too small to call the result statistically significant.