{"slug": "show-hn-i-made-17-ais-play-7700-hands-of-poker", "title": "Show HN: I Made 17 AIs Play 7,700 Hands of Poker", "summary": "A 30-line Python script, equity_tag, beat the best \"decision\" model, Jev 1.13, by 2.7 big blinds per hand (95% interval 1.0 to 4.6) over 100 duplicate deals in a 7,700-hand heads-up no-limit Texas hold'em benchmark of 17 AIs. Cheap reasoning LLMs — DeepSeek V4.1 Flash, Muse Spark Contributor and GPT-6 Luna — finished level with or above GPT-6.1 Sol and Claude Opus 5.5, with Luna costing $0.71 per thousand hands versus $22 for Opus, and the locally run Polish open-weights model Bielik finished last, below the random bot, never folding to a bet. The benchmark played every deal twice with seats swapped and applied all-in equity adjustment, which cut the spread of results by a factor of 2.7.", "body_md": "# I Made 17 AIs Play 7,700 Hands of Poker\n\nCheap reasoning LLMs matched GPT-6.1 Sol and Claude Opus 5.5 at a thirtieth of the price, a 30-line script beat every \"decision\" model, and a Polish local model played like a maniac\n\nDecision models folded pocket aces and lost to a 30-line equity script. Cheap reasoning LLMs played as well as Opus and Sol for under a dollar per thousand hands. Every hand was played twice, with the seats swapped.\n\nI built this benchmark to answer one practical question: **if you need an AI to make a sequence of decisions under uncertainty, should you use one of the new “decision” models, a cheap chat LLM, an expensive frontier model, a model running on your own GPU, or a plain script?** Poker is a good place to ask. Every decision has a price in chips, and luck can be measured and partly removed.\n\n## The short version\n\n- **Cheap reasoning LLMs played as well as the frontier.** DeepSeek V4.1 Flash, Muse Spark Contributor and GPT-6 Luna finished level with or above GPT-6.1 Sol and Claude Opus 5.5. Head to head, Luna and Muse drew with both. Luna costs $0.71 per thousand hands; Opus costs $22.\n- **A 30-line Python script beat the decision models.**`equity_tag` estimates its chance to win with a quick simulation and bets when it is ahead. It beat Jev 1.13, the best decision model, by 2.7 big blinds per hand (95% interval 1.0 to 4.6) over 100 duplicate deals. It also beat Solar by 5.3, and led pplx by 1.3, right at the edge of significance.\n- **The script held Opus to a draw.** Over 70 duplicate deals, Opus was ahead by 0.5 big blinds per hand, well inside the noise. GPT-6.1 Sol did beat it, by 2.4 (0.2 to 4.7).\n- **Decision models are cheap, fast and deterministic, and they play strange poker.** Jev almost never raises before the flop, then bets six times the pot after it. pplx folds hands that are ahead on the river. Solar goes all-in in more than half its hands.\n- **Bielik, the Polish open-weights model I ran locally, finished last** , below the random bot. It never once folded to a bet.\n\n## The best hands\n\nThe table above replays real hands from the benchmark. In the first one, a model whose whole job is to make decisions looks at two aces, the best starting hand in the game, faces a small raise from a bot that picks its actions at random, says it has a 48% chance to win, and folds. The bar under the table shows what it could not see: with both hands known, its aces were an 84% favourite.\n\nThat is not a cherry-picked glitch. Over 1,400 hands, Solar Decide folded a premium hand (AA, KK, QQ or AK) in 4 of the 13 spots where it held one. The random bot in the same league did it 3 times in 18.\n\n## Luck, and how to take it out\n\nPoker has enormous variance. A good player can lose for a thousand hands because the other player got better cards. Most AI poker results you see online are a few hundred hands of raw chip counts, and that is mostly a measurement of luck.\n\nEach match here is heads-up no-limit Texas hold’em: two players, blinds of 50 and 100 chips, 100 big blinds each at the start of every hand. Stacks reset after each hand, so nobody busts and every hand counts the same. Three things keep luck out of the ranking:\n\n1. **Duplicate deals.** Every deal is played twice with the seats swapped. If A gets aces and B gets kings in the first hand, B gets the aces in the second. What remains is the difference in how they played the same cards.\n2. **Shared deals.** Every pairing in a league uses the same list of deals, so everyone faces identical cards.\n3. **All-in adjustment.** When both players are all-in before the river, the hand is scored by each player’s equity at that moment, not by the cards that happened to come. This alone cut the spread of results by a factor of 2.7.\n\nBelow is a real match, counted three ways. Start with the first step, the way most benchmarks count, and watch the confidence band. Then switch to duplicate deals and the all-in adjustment. The match and the cards do not change; only the noise does. The slider replays the match deal by deal: early on, the band is wide enough to crown either player.\n\nEven with all three tricks, one duplicate deal between two players swings by about 9 big blinds, so pinning a single pairing to within one big blind per hand takes about 320 deals. That is why the cheap models play many opponents and the expensive ones play the opponents that matter.\n\nAll the AI players see exactly the same text: their cards, the board, the pot, the stacks, the betting so far, a small tracker panel with the opponent’s statistics, and a menu of legal actions (fold, check, call, five bet sizes from a third of the pot to one and a half pots, all-in). They answer with one option and an estimate of their chance to win. Chat models get the menu as a JSON schema; decision models get it through OpenRouter’s Decisions API, which returns a probability for each option.\n\nThe rating is the average result per hand in big blinds, fitted over every match at once, so two players who never met still get comparable numbers through common opponents. Intervals come from a bootstrap over deals.\n\n## The decision models\n\nDecision models are a new kind of endpoint: instead of free text, you send a state and a list of options and get back a choice with probabilities. They are very cheap (5 to 10 cents per thousand decisions) and fast (Jev answers in 0.3 seconds). I tested four: Jev 1.13, pplx-decider 27B, Solar Decide and Mercury Decide.\n\nBecause they return a probability for every option, you can look inside a decision. The aces fold from the replays above was not a close call that went wrong: Solar spread its belief across the whole menu, and fold won with a quarter of it.\n\nJev was the best of them and plays the strangest poker in the league. It raises before the flop in 3% of hands, calls almost everything, and then, after the flop, its median bet is **six times the pot**.\n\nA six-pot bet wins against players who fold too much. Against a script that simply calls when its equity justifies it, the same bets lose whole stacks. The second replay above is one of them.\n\npplx has the opposite problem. In 7 of the 35 river spots where it had at least 90% equity, it folded. It also underestimates its own hand: asked for its chance to win, it reports on average 19 points less than a simulation gives. Jev underestimates by 17 points, Solar by 17, Mercury by 15. The chat LLMs are off by 4 to 9 points; the scripts by under 1.\n\nSolar is not a poker player in any useful sense. It goes all-in in 56% of its hands and folds premium hands a third of the time.\n\n## The cheap LLMs\n\nThe five cheap reasoning LLMs play something that looks like textbook poker: they enter many pots, raise more often than they call, and size their bets between half a pot and a full pot. They are also the best at judging their own hands among the cheap players: their win-probability estimates correlate 0.72 to 0.83 with the real equity, against about 0.6 for Jev and pplx.\n\nThe clearest way to see the difference is to put a bet in front of each player and look at what it does with weak and strong hands. A sound player folds the weak ones and raises the strong ones. The calling station never folds, by design. Bielik never folds either, and nobody designed it that way.\n\nThey are not consistent, though. Asked about the same spot twice, GLM picks the same action only 55% of the time and MiMo 65%; the best, DeepSeek, manages 82%. Decision models are deterministic: Jev and pplx always give the same answer. Renaming the suits, which changes nothing about the hand, still flips about one answer in ten for both. The frontier models are the steadiest language models here: Opus gave the same answer to a repeated question 40 times out of 40 and kept it after the suits were renamed 38 times; Sol managed 39 and 40.\n\n## The frontier round\n\nFrontier models are expensive enough that I could not afford a full league. Instead, GPT-6.1 Sol and Claude Opus 5.5 (both with their default reasoning) each played the five opponents whose ratings were already pinned down: the script, Jev, pplx, Luna and Muse Spark Contributor. They also played each other. Then I gave both a second, longer match against the script.\n\n|  | GPT-6.1 Sol | Claude Opus 5.5 | \n|---|---|---|\n| Rating, BB per hand | +3.4 (2.3 to 4.6) | +2.7 (1.6 to 3.7) | \n| vs the 30-line script (70 deals) | +2.4 (0.2 to 4.7) | +0.5 (−1.4 to 2.6) | \n| vs Jev (20 deals) | +0.8 | +0.8 | \n| vs Luna (20 deals) | +0.6 | +0.5 | \n| vs Muse Spark Contributor | 0.0 | −0.9 | \n| vs each other |  | +0.1 | \n| Cost per 1,000 hands | $8.7 | $22.0 | \n| Median time per decision | 7 s | 6 s | \n\nThe frontier models are the best calibrated players in the benchmark: their win-probability estimates correlate 0.87 to 0.88 with the truth, and they never produced an invalid answer. They beat the decision models, mostly by small margins. But against the cheap reasoning models they only drew, and those cost 10 to 30 times less.\n\nThe two play differently. Sol is the tighter player: it enters 58% of pots and folds to a bet 34% of the time. Opus enters 67% and folds to a bet only 18% of the time, and spends no reasoning tokens at all on easy spots.\n\nThe match against the script is also a lesson in sample size. After the first 20 deals, Opus led the script by 3.5 big blinds per hand, with an interval that did not include zero. I nearly wrote that up. The next 50 deals went the other way, and the combined result is a draw. You can watch it happen: pick the script against Claude Opus 5.5 in the explorable above and drag the slider back to deal 20.\n\n## Bielik: a Polish model on my own GPU\n\nBielik 11B v3 is an open-weights model from the Polish SpeakLeash project. It is built for Polish, not for reasoning, and it has no thinking mode. I ran it locally (Q8 quantization, llama.cpp, one RTX 3090 Ti) through the same English prompt as everyone else. It costs nothing per hand and answers in about one second, so it gets its own colour on the charts: the only player that is both free and a language model.\n\nIt finished last, at −8.4 big blinds per hand, below the bot that picks actions at random. The tracker numbers explain why: Bielik voluntarily put money in 96% of pots, re-raised 77% of the time it could, went all-in in 31% of its hands, and **never folded to a bet**. It is also the only player in the benchmark that overestimates its chances, by 20 points on average. In the frozen spots it agrees with the maniac bot more than with any other player.\n\nThis says little about Bielik as a Polish assistant. It does say that a small instruction-tuned model without reasoning does not turn a poker prompt into poker.\n\n## Who plays like whom\n\nI froze 40 spots from real hands and asked every player for its action. Two players “agree” on a spot if they choose the same kind of action. Ordered by similarity, the league splits into clear families. The strongest language models form the tightest block: Sol, Opus, Luna, DeepSeek and Muse Spark Contributor pick the same kind of action in 65 to 80% of spots, although they come from four different labs. Good play seems to converge. Jev and pplx sit next to the calling station, and Solar and Bielik land next to the random bot and the maniac.\n\n## What it costs\n\nThe cheapest way to play decent poker in this benchmark is still free: the 30-line script finished level with Opus, MiMo and GLM. Among models, Luna and GLM are the best value at 60 to 70 cents per thousand hands. DeepSeek V4.1 Flash has the highest point estimate but costs nine times more than Luna, and ran out of its 32,768-token output budget on 3.6% of decisions, which count as errors. Muse Spark Contributor sits in between. Opus costs thirty times more than Luna for the same result.\n\nTime is a cost too. DeepSeek thinks for about 100 seconds per decision on average and up to five minutes on hard spots, so a single hand against it takes 3 to 8 minutes; its last few hands kept the whole cheap league waiting long after every other model had finished.\n\n## Things that went wrong\n\nMost of the work was not poker. Some of what broke, in the order it hurt:\n\n- **Elo crowned the maniac.** My first leaderboard used Elo on single deals. A bot that raises every hand wins many small pots and loses a few huge ones, so it won the most “games” and topped the table while losing chips. The ranking now uses chips per hand, and Elo is computed on blocks of ten deals.\n- **A one-line bug in the all-in detector** fired on every street change, adjusting hands that were not all-in at all. A regression test now pins it.\n- **Reasoning knobs are not portable.** GLM with default reasoning thought until it hit 8,192 tokens and returned nothing; with`effort: medium` it did not reason at all;`effort: high` worked. DeepSeek ran out of tokens in a third of its answers until I raised the limit to 32,768.\n- **Rate limits look like bad play.** pplx returned more than a thousand “too many requests” errors in one run. My fallback quietly turned them into check/fold, which looked like a very passive player. Provider failures now void the hand, and it is replayed later.\n- **Free models have a daily cap.** Mercury Decide stops at 1,000 requests per day, so its league was finished over two days.\n- **Local models need a different JSON mode.** Bielik’s chat template broke llama.cpp’s strict JSON-schema grammar on every request; plain JSON mode worked.\n- **I misread a sign.** Pairwise results are stored as “A’s result against B”, and for a few minutes I read the script’s loss to Opus as a win. The charts compute the sign from the data, so they were right; my headline was not.\n\n## Limitations\n\nHeads-up with 100-big-blind stacks reset every hand is a narrow slice of poker. The action menu has five fixed sizes, which removes some skill and some ways to blunder. The frontier and local models played fewer deals than the decision models, so their intervals are wider; head-to-head results over 20 deals are hints, not verdicts. Every model saw the same English prompt; a different prompt could change the order. And none of these players would survive a real solver: the point is the comparison between kinds of AI, not a claim about superhuman play.\n\n## Code and data\n\nThe benchmark, every hand, every API call and its cost are open. Code: [HoldemBench on GitHub](https://github.com/ChrystianSchutz/poker-holdembench). Data: every decision with its prompt state, choice, equity estimate, latency and cost, [on Hugging Face](https://huggingface.co/datasets/chrystians/poker-holdembench).", "url": "https://wpnews.pro/news/show-hn-i-made-17-ais-play-7700-hands-of-poker", "canonical_source": "https://chrystianschutz.com/blog/ai-poker-holdembench/", "published_at": "2026-10-17 19:33:59+00:00", "updated_at": "2026-10-17 20:06:48.266151+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents"], "entities": ["DeepSeek V4.1 Flash", "Muse Spark Contributor", "GPT-6 Luna", "GPT-6.1 Sol", "Claude Opus 5.5", "Jev 1.13", "Bielik", "Solar Decide"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-i-made-17-ais-play-7700-hands-of-poker", "markdown": "https://wpnews.pro/news/show-hn-i-made-17-ais-play-7700-hands-of-poker.md", "text": "https://wpnews.pro/news/show-hn-i-made-17-ais-play-7700-hands-of-poker.txt", "jsonld": "https://wpnews.pro/news/show-hn-i-made-17-ais-play-7700-hands-of-poker.jsonld"}}