# King and queen vs. a lone king: 5 small LLMs mated 3 times in 139 games, and lost the queen faster than a random mover

> Source: <https://discuss.huggingface.co/t/king-and-queen-vs-a-lone-king-5-small-llms-mated-3-times-in-139-games-and-lost-the-queen-faster-than-a-random-mover/191912#post_1>
> Published: 2026-11-16 01:18:03+00:00

Hi all,

I wanted a chess test where “the model doesn’t know the rules” can’t be the excuse, so I picked the easiest checkmate there is: king and queen against a lone king. From a random position it takes at most about ten moves, and every model got the full list of legal moves on every turn.

**Setup**

`{"move": "<SAN>"}`.
**Results**

| Model | Games | Mates | Queen captured | Queen lost (median move) | 
|---|---|---|---|---|
| Gemma 4 31B | 20 | 1 | 18 | 1.5 | 
| Gemma 4 31B, thinking on | 20 | 2 | 18 | 2 | 
| Gemma 4 26B | 19 | 0 | 19 | 2 | 
| Gemma 4 26B, thinking on | 20 | 0 | 19 | 2 | 
| Gemini 3.5 Flash-Lite | 20 | 0 | 19 | 3 | 
| Gemma 4 E4B (Ollama) | 20 | 0 | 17 | 4 | 
| Llama 3.2 3B (Ollama) | 20 | 0 | 9 | 4 | 

Llama 3.2’s lower queen-loss count isn’t better play: 6 of its games ended early on four unparseable answers in a row (85 of 256 calls failed to parse) and 5 ended in repetition.

**What goes wrong**

Almost always the same move: a check from the square right next to the black king, with nothing defending the queen. The random mover survives longer because most of its queen moves land somewhere safe by accident. The models go looking for checks, and the nearest check is often a square the king can just take.

I reran one position with Gemma 4 26B and asked for its reasoning. It chose `Qg4+` with the king on f3, then listed Black’s replies as `Ke3` and `Kf2`. `Kxg4` wasn’t on the list.

Two measurement bugs I caught before publishing, in case they help anyone running similar tests: API failures were being counted as illegal moves, and `Qb3` without the `+` was being rejected as illegal when the legal list had `Qb3+`.

**Limits**

Small and cheap models only (Gemini API free tier and Ollama), 20 positions per model, one prompt format. Frontier models weren’t tested and may well do better. Tool use (a list of attacked squares, for example) could change the picture.

The script and every game (moves, results, unparsed answers) are public, and the seed is fixed, so results on other models are directly comparable:

A question for people here: has anyone seen a prompt format or a small tool (attacked squares, “is this piece defended?”) that fixes this kind of blunder, or does it only go away with bigger models?

Disclosure: I run LLMPvP, a free rated arena where LLM agents play chess and Go through an API. This test came out of watching agents there hang pieces.
