How often does an LLM name a poker hand it doesn't hold? I counted for two models

A developer running an AI poker site counted how often large language models claim to hold a poker hand they don't, using a regex-based checker over more than 1,000 decisions each for two models. GLM 4.7 Flash was flagged on 88 of 1,342 decisions (6.6%) and had 41 moves (3.1%) corrected by the server as illegal, while GPT-6 Astra drew zero flags and zero fixes across 1,397 decisions; a manual read of 15 random GLM flags found 13 genuine misstatements of hand strength. The developer cautioned that the regex undercounts ambiguous cases and that 15 reads are too few for a precise rate.

My blunders endpoint flags GLM 4.7 Flash with 88 decisions for naming a poker hand it doesn't hold, and GPT-6 Astra with 0. I wanted to know whether the 88 was real, so I read some of them. People play hands against these models on the site, heads-up or at 6-max tables. Every model writes a short reason with each action, and a checker compares the hand it claims with the hand the engine says it has on that street. A recent ICLR 2026 paper, How Far Are LLMs from Professional Poker Players? https://arxiv.org/abs/2602.00528 , lists three problems with how models reason. One of them, factual misunderstandings, is the one I can count on my site: a model naming a made hand that isn't in its cards. The checker flags a decision when the reason claims a made hand stronger than the engine's. "Trip kings" with a king-king board and no king in hand is a flag. It skips ambiguous cases like draw talk and negations, so it undercounts. It's a set of regular expressions, and I keep patching it when I find a false positive. A flag is a lead to check. A second count needs no heuristic. When a model's answer is not a legal action as given say, a check facing a bet , the server substitutes a legal one and marks the decision as fixed. Only two models have passed 1,000 decisions at human tables, so only those two get numbers. These come from /api/blunders https://aipoker.bot/api/blunders on 26 October, human tables, prompt version 1.1.0 or later: | model | decisions | hands | moves the server fixed | flagged hand claims | |---|---|---|---|---| | GLM 4.7 Flash | 1,342 | 464 | 41 3.1% | 88 6.6% | | GPT-6 Astra | 1,397 | 691 | 0 | 0 | The 88 flags sit in 68 of GLM's 464 hands, about 15% of them. Fourteen of those hands carry two or three flags. 88 is a regex number, so I eyeballed some. I drew 15 of the 88 at random and compared each reason with the cards. 13 were real. Two of them: Jc 2d on 9h Ts Jh Ad : "Holding top set vs wide range. The shove is standard, I'm never folding a full house here." It had a pair of jacks. 7d 4h on Jh 3c 4d Ks Kc : "I have trip kings and a good kicker." Both kings were on the board, so it had two pair, kings and fours. The other two were ambiguous. "Board pairs, trips aren't great" reads as talk about the board. "Top pair looks solid, repping value" reads as a bluff and makes no claim. So the 88 is probably a bit high. Fifteen reads are too few for a tight number, and I'm not giving one. I checked the other direction too. Of Astra's 1,397 decisions, 239 postflop reasons use a hand word such as pair, flush or ace high. I read 15 of those at random, and all 15 matched the engine. That rules out a big miss rate. A handful could still hide in the other 224. GLM 4.7 Flash held Jc 5s and read a paired board as trips. Preflop it re-raised with the note "Top pair is better than high cards, let's isolate." On a flop of 8d Kd Kh it went all-in for 180: "Trip kings is monster, end it before I miss." Both kings were on the board. Gemma 4 31B held Ah Jd , wrote "I only have Ace high and the opponent is jammed", and folded. The wrong belief won the pot. Replay: aipoker.bot/h/EUe5BecDkCwQ https://aipoker.bot/h/EUe5BecDkCwQ The second GLM hand: Js 9h on Ac 9s Kd . It bet the flop with "A set, pop the bottle " and raised with "Set mining worked out. Small stack and monster pot. Value town coming soon." On the turn it moved all-in: "All in Set over set with the straight draw on board." That was a pair of nines. DeepSeek 4 Flash called with 4c 8d , and the river 8 paired it, which still lost to nines. GLM won 319 chips. At the same table gpt-oss-20b, holding Kc 6c , wrote "I have set of kings" on that flop. That was one king. Replay: aipoker.bot/h/2d7kyGZhfgY4 https://aipoker.bot/h/2d7kyGZhfgY4 GPT-6 Astra had Kd 6d on a flop of 4d 3c 3d and wrote "I'm getting a good price to chase my flush draw in position." It called the flop and the turn. When the river came Td , it bet 78: "I'm betting my king-high flush for value after everyone checks." A draw on the flop, a flush on the river. Every sentence matches the cards. It won 167 chips. Replay: aipoker.bot/h/Ns3W nx1BXZ- https://aipoker.bot/h/Ns3W nx1BXZ- It says nothing about who plays better. The two GLM hands above won their pots, and at this volume the confidence intervals on win rates are too wide to publish. It doesn't prove that Astra never misnames a hand either. Zero flags from a cautious checker and 15 clean reads say the rate is low. Not zero. The other models are under 1,000 decisions on human tables, DeepSeek 4 Flash next at 846. When they pass, they get the same table and the same hand-read.