{"slug": "king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a", "title": "King and queen vs. a lone king: 5 small LLMs mated 3 times in 139 games, and lost the queen faster than a random mover", "summary": "A benchmark of five small language models playing king-and-queen versus a lone king produced only 3 checkmates across 139 games, with Gemma 4 31B scoring 1 mate in 20 games and 2 more with thinking enabled, while Gemma 4 26B, Gemini 3.5 Flash-Lite, Gemma 4 E4B and Llama 3.2 3B recorded zero mates. The tester, who runs the free rated arena LLMPvP, reported that the models typically gave an undefended check from the square adjacent to the enemy king, losing the queen at a median of move 1.5 to 4; Llama 3.2 3B's lower 9 queen captures came with 85 of 256 calls failing to parse and 5 games ending in repetition. The author published the script, all games and a fixed seed, and noted the test covered only small, cheap models via the Gemini API free tier and Ollama, not frontier models.", "body_md": "Hi all,\n\nI wanted a chess test where “the model doesn’t know the rules” can’t be the excuse, so I picked the easiest checkmate there is: king and queen against a lone king. From a random position it takes at most about ten moves, and every model got the full list of legal moves on every turn.\n\n**Setup**\n\n`{\"move\": \"<SAN>\"}`.\n**Results**\n\n| Model | Games | Mates | Queen captured | Queen lost (median move) | \n|---|---|---|---|---|\n| Gemma 4 31B | 20 | 1 | 18 | 1.5 | \n| Gemma 4 31B, thinking on | 20 | 2 | 18 | 2 | \n| Gemma 4 26B | 19 | 0 | 19 | 2 | \n| Gemma 4 26B, thinking on | 20 | 0 | 19 | 2 | \n| Gemini 3.5 Flash-Lite | 20 | 0 | 19 | 3 | \n| Gemma 4 E4B (Ollama) | 20 | 0 | 17 | 4 | \n| Llama 3.2 3B (Ollama) | 20 | 0 | 9 | 4 | \n\nLlama 3.2’s lower queen-loss count isn’t better play: 6 of its games ended early on four unparseable answers in a row (85 of 256 calls failed to parse) and 5 ended in repetition.\n\n**What goes wrong**\n\nAlmost always the same move: a check from the square right next to the black king, with nothing defending the queen. The random mover survives longer because most of its queen moves land somewhere safe by accident. The models go looking for checks, and the nearest check is often a square the king can just take.\n\nI reran one position with Gemma 4 26B and asked for its reasoning. It chose `Qg4+` with the king on f3, then listed Black’s replies as `Ke3` and `Kf2`. `Kxg4` wasn’t on the list.\n\nTwo measurement bugs I caught before publishing, in case they help anyone running similar tests: API failures were being counted as illegal moves, and `Qb3` without the `+` was being rejected as illegal when the legal list had `Qb3+`.\n\n**Limits**\n\nSmall and cheap models only (Gemini API free tier and Ollama), 20 positions per model, one prompt format. Frontier models weren’t tested and may well do better. Tool use (a list of attacked squares, for example) could change the picture.\n\nThe script and every game (moves, results, unparsed answers) are public, and the seed is fixed, so results on other models are directly comparable:\n\nA question for people here: has anyone seen a prompt format or a small tool (attacked squares, “is this piece defended?”) that fixes this kind of blunder, or does it only go away with bigger models?\n\nDisclosure: I run LLMPvP, a free rated arena where LLM agents play chess and Go through an API. This test came out of watching agents there hang pieces.", "url": "https://wpnews.pro/news/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a", "canonical_source": "https://discuss.huggingface.co/t/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-the-queen-faster-than-a-random-mover/191912#post_1", "published_at": "2026-11-16 01:18:03+00:00", "updated_at": "2026-11-16 01:20:33.515137+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-agents"], "entities": ["Gemma 4 31B", "Gemma 4 26B", "Gemini 3.5 Flash-Lite", "Gemma 4 E4B", "Llama 3.2 3B", "Ollama", "LLMPvP", "Gemini API"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a", "markdown": "https://wpnews.pro/news/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a.md", "text": "https://wpnews.pro/news/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a.txt", "jsonld": "https://wpnews.pro/news/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-a.jsonld"}}