{"slug": "llm-agents-play-pokemon-showdown", "title": "LLM Agents Play Pokemon Showdown", "summary": "A multi-agent environment built on the official Pokémon Showdown simulator by developer Oliver Grabner found that Qwen3.8-27B agents allowed to talk freely to their teammate won 64% of 2v2 battles against a scripted max-damage bot, versus 42% for agents that could not talk at all, according to results published October 10, 2026. A single message each reached 52% and one model controlling both Pokémon reached 54%, while free talk consumed 978k prompt tokens per battle, 3.8x the 259k used with no talk. The same Qwen3.8-27B team won 96% against a Qwen3.5-2B team and 74% against a Qwen3.5-9B team, and Grabner plans to separate the effect of talking from the effect of seeing a partner's move and to turn the environment into a leaderboard benchmark and RL training setup.", "body_md": "[← Back to home](https://olivergrabner.com/)\n\n# Multi-agent LLM coordination in a Pokémon 2v2 environment\n\nOctober 10, 2026\n\n[https://github.com/OliverGrabner/multi-agent-poke-showdown](https://github.com/OliverGrabner/multi-agent-poke-showdown)\n\n    The future is in LLM collaborations, and it is important to be able to understand how good agents\n    are at working together, communicating and negotiating actions. Most multi-agent research is about\n    workflows, where agents pass tasks down a pipeline\n    ([Hong et al., 2024](https://arxiv.org/abs/2308.00352);\n    [Qian et al., 2024](https://arxiv.org/abs/2307.07924)). To test this, I built a multi-agent\n    environment on Pokémon Showdown's 2v2 format to evaluate these abilities.\n  \n\n## Why Pokémon?\n\n    Pokémon doubles is a game-theory environment without full information. The opponents' sets are\n    hidden and damage is random, so winning takes real strategy: reading the other team, splitting\n    targets and planning around your partner. Recent work like the PokéAgent Challenge\n    ([Karten et al., 2026](https://arxiv.org/abs/2603.15563)) benchmarks Pokémon battling,\n    but each side is played by a single agent, and specialist RL agents still outplay general LLMs\n    there. It doesn't test teammates who have to coordinate.\n  \n\nThe goal is a controlled environment where I can measure how LLM agents coordinate, and that can later be used to train them with RL to encourage these collaborative behaviors.\n\n## Methodology\n\n### Environment\n\nThe official Pokémon Showdown simulator, wrapped as a Gym-style environment. Four players each control one Pokémon from a team of three, two per side.\n\n### Agents\n\n    Each turn, an agent reads a plain-English summary of the battle and a numbered list of its legal\n    moves. It can `say` a private message to its partner or `choose` a move, and\n    once it chooses, its partner is told what it picked.\n  \n\n### Talk modes\n\nFree talk (keep talking until both have chosen), one message each, no talk, and one model controlling both Pokémon.\n\n### Evaluation\n\nQwen3.8-27B plays both teammates against a scripted max-damage bot. Battles are seeded and every team draw is played from both sides, so each talk mode faces exactly the same battles.\n\n## Results\n\n| Condition | 30%50%70% |  | Tokens per battle | \n|---|---|---|---|\n| Free talk |  | **64%** | 978k | \n| One model controls both |  | 54% | 450k | \n| One message each |  | 52% | 654k | \n| No talk |  | 42% | 259k | \n\nTalking helps. Teammates who could talk freely won 64% of their battles against the max-damage bot, while teammates who couldn't talk at all won 42%. Even a single message each (52%) beat no talk. One model controlling both Pokémon (54%) did better than no talk but worse than two agents talking freely, so I think the gain comes from the conversation itself, not just from one mind planning both moves.\n\nTalking also costs. Free talk used 978k prompt tokens per battle, 3.8x the 259k of no talk, since every message means another model call that re-reads the battle and the conversation so far.\n\n| Opponent | Win rate |  | \n|---|---|---|\n| Qwen3.5-2B team |  | **96%** | \n| Qwen3.5-9B team |  | 74% | \n| Max-damage bot |  | 64% | \n\nI also checked that the environment can tell models apart. A Qwen3.8-27B team won 96% of its battles against a team of 2B models and 74% against a team of 9B models, compared with 64% against the max-damage bot. The stronger the opponent, the closer the games, which is what makes the environment usable as a benchmark for comparing models.\n\n## Next\n\nI'm separating the effect of talking from the effect of seeing your partner's move. I'm also looking into turning the environment into a benchmark with a leaderboard across models, and using it for RL training to encourage collaborative behaviors.\n\n## Thanks\n\n    Through this project I wanted to learn more about agent collaboration and RL environments, and\n    thought I'd share the results! I ran these experiments on Texas A&M's Grace cluster. Thank you\n    to Maria Teleki and to my advisor,\n    [Prof. James Caverlee](https://people.engr.tamu.edu/caverlee/), for the guidance and\n    feedback along the way.", "url": "https://wpnews.pro/news/llm-agents-play-pokemon-showdown", "canonical_source": "https://olivergrabner.com/research/pokemon/", "published_at": "2026-10-22 02:13:48+00:00", "updated_at": "2026-10-22 02:42:07.016616+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "artificial-intelligence"], "entities": ["Oliver Grabner", "Pokémon Showdown", "Qwen3.8-27B", "Qwen3.5-2B", "Qwen3.5-9B", "Texas A&M", "PokéAgent Challenge"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llm-agents-play-pokemon-showdown", "markdown": "https://wpnews.pro/news/llm-agents-play-pokemon-showdown.md", "text": "https://wpnews.pro/news/llm-agents-play-pokemon-showdown.txt", "jsonld": "https://wpnews.pro/news/llm-agents-play-pokemon-showdown.jsonld"}}