cd /news/large-language-models/king-and-queen-vs-a-lone-king-5-smal… · home › topics › large-language-models › article
[ARTICLE · art-170189] src=discuss.huggingface.co ↗ pub= topic=large-language-models verified=true sentiment=↓ negative

King and queen vs. a lone king: 5 small LLMs mated 3 times in 139 games, and lost the queen faster than a random mover

A benchmark of five small language models playing king-and-queen versus a lone king produced only 3 checkmates across 139 games, with Gemma 4 31B scoring 1 mate in 20 games and 2 more with thinking enabled, while Gemma 4 26B, Gemini 3.5 Flash-Lite, Gemma 4 E4B and Llama 3.2 3B recorded zero mates. The tester, who runs the free rated arena LLMPvP, reported that the models typically gave an undefended check from the square adjacent to the enemy king, losing the queen at a median of move 1.5 to 4; Llama 3.2 3B's lower 9 queen captures came with 85 of 256 calls failing to parse and 5 games ending in repetition. The author published the script, all games and a fixed seed, and noted the test covered only small, cheap models via the Gemini API free tier and Ollama, not frontier models.

read2 min views2 publishedNov 16, 2026

Hi all,

I wanted a chess test where “the model doesn’t know the rules” can’t be the excuse, so I picked the easiest checkmate there is: king and queen against a lone king. From a random position it takes at most about ten moves, and every model got the full list of legal moves on every turn.

Setup

{"move": "<SAN>"}. Results

Model Games Mates Queen captured Queen lost (median move)
Gemma 4 31B 20 1 18 1.5
Gemma 4 31B, thinking on 20 2 18 2
Gemma 4 26B 19 0 19 2
Gemma 4 26B, thinking on 20 0 19 2
Gemini 3.5 Flash-Lite 20 0 19 3
Gemma 4 E4B (Ollama) 20 0 17 4
Llama 3.2 3B (Ollama) 20 0 9 4

Llama 3.2’s lower queen-loss count isn’t better play: 6 of its games ended early on four unparseable answers in a row (85 of 256 calls failed to parse) and 5 ended in repetition.

What goes wrong

Almost always the same move: a check from the square right next to the black king, with nothing defending the queen. The random mover survives longer because most of its queen moves land somewhere safe by accident. The models go looking for checks, and the nearest check is often a square the king can just take.

I reran one position with Gemma 4 26B and asked for its reasoning. It chose Qg4+ with the king on f3, then listed Black’s replies as Ke3 and Kf2. Kxg4 wasn’t on the list.

Two measurement bugs I caught before publishing, in case they help anyone running similar tests: API failures were being counted as illegal moves, and Qb3 without the + was being rejected as illegal when the legal list had Qb3+.

Limits

Small and cheap models only (Gemini API free tier and Ollama), 20 positions per model, one prompt format. Frontier models weren’t tested and may well do better. Tool use (a list of attacked squares, for example) could change the picture.

The script and every game (moves, results, unparsed answers) are public, and the seed is fixed, so results on other models are directly comparable:

A question for people here: has anyone seen a prompt format or a small tool (attacked squares, “is this piece defended?”) that fixes this kind of blunder, or does it only go away with bigger models?

Disclosure: I run LLMPvP, a free rated arena where LLM agents play chess and Go through an API. This test came out of watching agents there hang pieces.

── more in #large-language-models 4 stories · sorted by recency
── more on @gemma 4 31b 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/king-and-queen-vs-a-…] indexed:0 read:2min 2026-11-16 · —