{"slug": "evaluating-large-language-models-in-a-complex-hidden-role-game", "title": "Evaluating Large Language Models in a Complex Hidden Role Game", "summary": "A new study evaluating large language models in the social deduction game Secret Hitler found that current architectures remain ineffective at complex, multi-turn manipulation and deception. Models like Llama 3.1 70B achieved only 59.7% accuracy in matching expert human voting decisions, while rule-based agents aligned 86.7% of the time, and models playing as Fascists consistently yielded negative impact scores with roughly 40% shorter games than humans. The findings highlight a critical gap between conversational ability and strategic depth, underscoring the need for reproducible testbeds to detect when models begin mastering deceptive behaviors as capabilities advance.", "body_md": "arXiv:2605.22826v1 Announce Type: new\nAbstract: Quantifying the deceptive potential of Large Language Models (LLMs) is critical for AI safety, yet difficult to achieve in uncontrolled environments. This work investigates the reasoning, persuasion, and deceptive capabilities of LLMs within the social deduction game Secret Hitler. I introduce an open-source framework and novel metrics to measure performance: Role Identification Accuracy, Deception Retention Rate, and Game State Impact Rate. By benchmarking models against rule-based algorithms and human games, I identify a gap between conversational ability and strategic depth. The study also analyzes the impact of reasoning-enhancement techniques on win rates and strategic reasoning. Neither Chain-of-Thought prompting nor internal memory bring improvements in performance, with up to 23.2% worse win rates for fascist roles. While rule-based agents align with expert human voting decisions 86.7% of the time, models like Llama 3.1 70B achieve only a 59.7% accuracy. Models playing as Fascists consistently yield negative impact scores and fail to sustain deception, resulting in roughly 40% shorter games compared to humans. These findings suggest that current architectures remain ineffective at complex, multi-turn manipulation. As capabilities advance, detecting when models begin to master these deceptive behaviors is crucial. The developed framework serves as a reproducible testbed for future alignment research.", "url": "https://wpnews.pro/news/evaluating-large-language-models-in-a-complex-hidden-role-game", "canonical_source": "https://arxiv.org/abs/2605.22826", "published_at": "2026-05-25 04:00:00+00:00", "updated_at": "2026-05-25 15:25:00.128449+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research", "natural-language-processing", "ai-ethics"], "entities": ["Llama 3.1 70B", "Secret Hitler"], "alternates": {"html": "https://wpnews.pro/news/evaluating-large-language-models-in-a-complex-hidden-role-game", "markdown": "https://wpnews.pro/news/evaluating-large-language-models-in-a-complex-hidden-role-game.md", "text": "https://wpnews.pro/news/evaluating-large-language-models-in-a-complex-hidden-role-game.txt", "jsonld": "https://wpnews.pro/news/evaluating-large-language-models-in-a-complex-hidden-role-game.jsonld"}}