Hi all,
I wanted a chess test where “the model doesn’t know the rules” can’t be the excuse, so I picked the easiest checkmate there is: king and queen against a lone king. From a random position it takes at most about ten moves, and every model got the full list of legal moves on every turn.
Setup
{"move": "<SAN>"}.
Results
| Model | Games | Mates | Queen captured | Queen lost (median move) |
|---|---|---|---|---|
| Gemma 4 31B | 20 | 1 | 18 | 1.5 |
| Gemma 4 31B, thinking on | 20 | 2 | 18 | 2 |
| Gemma 4 26B | 19 | 0 | 19 | 2 |
| Gemma 4 26B, thinking on | 20 | 0 | 19 | 2 |
| Gemini 3.5 Flash-Lite | 20 | 0 | 19 | 3 |
| Gemma 4 E4B (Ollama) | 20 | 0 | 17 | 4 |
| Llama 3.2 3B (Ollama) | 20 | 0 | 9 | 4 |
Llama 3.2’s lower queen-loss count isn’t better play: 6 of its games ended early on four unparseable answers in a row (85 of 256 calls failed to parse) and 5 ended in repetition.
What goes wrong
Almost always the same move: a check from the square right next to the black king, with nothing defending the queen. The random mover survives longer because most of its queen moves land somewhere safe by accident. The models go looking for checks, and the nearest check is often a square the king can just take.
I reran one position with Gemma 4 26B and asked for its reasoning. It chose Qg4+ with the king on f3, then listed Black’s replies as Ke3 and Kf2. Kxg4 wasn’t on the list.
Two measurement bugs I caught before publishing, in case they help anyone running similar tests: API failures were being counted as illegal moves, and Qb3 without the + was being rejected as illegal when the legal list had Qb3+.
Limits
Small and cheap models only (Gemini API free tier and Ollama), 20 positions per model, one prompt format. Frontier models weren’t tested and may well do better. Tool use (a list of attacked squares, for example) could change the picture.
The script and every game (moves, results, unparsed answers) are public, and the seed is fixed, so results on other models are directly comparable:
A question for people here: has anyone seen a prompt format or a small tool (attacked squares, “is this piece defended?”) that fixes this kind of blunder, or does it only go away with bigger models?
Disclosure: I run LLMPvP, a free rated arena where LLM agents play chess and Go through an API. This test came out of watching agents there hang pieces.