I run aipoker.bot, a small site where you sit down and play poker against a lineup of AI poker bots. Some of them are language models, some are old-school rule engines like Slumbot, Shanky, Warbot and OpenHoldem. Every bot drops a one-line note on each move, so you watch it talk itself through the hand while the cards come out. I log those notes on every hand. They taught me more about the models than the leaderboard ever did.
The leaderboard tells you which seat is up and which is stuck. It can't show you the gap between the hand a model says it's holding and the hand in front of it.
Here are two hands where a seat announced a full house.
Six-max table, five open models drawn at random. The board ran out Jc Jd Kc 2c 4h, so the jacks were paired on the felt from the flop onward. Replay.
One seat, GLM 4.7 Flash, held Kd 3s. That makes kings and jacks, a fine two pair. On the flop it wrote "Full house now, running it up is the only move" and raised. On the turn it added "Sets crushed so cap it off" and put the rest in. There was no full house anywhere in its two cards. It had paired the board and talked itself into a boat.
Another seat, Qwen 3.8 27B, had quietly backed into five clubs for a flush and called the whole thing down. GLM's two pair, dressed up in its notes as a full house, paid off the flush for its full stack.
The move by itself was not crazy. Two pair on a wet board can go to the felt. The sentence beside the move is where it came apart.
Different table, a six-max Arena seat. Board Kd Ks Tc 2h 7c. Replay. GPT-6 Astra held Td Th. On the flop that is tens full of kings already, a real full house, and its note read "I smooth-call my full house, leaving room for more action behind me." On the turn: "I put the rest in with tens full; there's too much value to leave behind." It named the hand it held, named it correctly, and got the chips in for value. It won the pot.
Same six words, roughly. One seat said full house and had tens full. The other said full house and had two pair. Reading the note alone, you cannot tell them apart. You have to check it against the cards.
What I expected to see and barely do: a model trying a move it is not allowed to make, or returning an empty action. Across the logged decisions, that almost never happens. The action a model picks is legal and usually reasonable; what drifts is the story it tells about its own cards. The action is one bit. The sentence next to it is where a model narrates a hand it never held.
So I keep both kinds. The seat that announces a monster it imagined. The seat that announces the same monster and has it.
aipoker.bot is free, no account, no ads, play money only. You pick a table and play a few hands against the bots. The post-hand review shows the cards they were actually holding. The leaderboard tracks which bots are ahead; every row carries its sample size and a confidence interval, because most of those numbers are still noise.
I am Nils. Self-funded side project, I write software and lose at PLO. If you catch a bot describing its hand worse than these two did, send me the replay.