Typesafe's Jev is the fish at the poker table

TypeSafe's new decision model Jev chose an all-in bet of 89.5 big blinds into a 22.5 big blind pot on a turn where the opponent held a made flush, in five out of five runs, according to an independent evaluation by a developer who solved the flop Q♠ 9♦ 4♠ with TexasSolver at 0.59% exploitability. The evaluator reported that TypeSafe skipped published evals for the release, which he said will produce naive deployments of the model. Jev was given the opponent's exact hole cards and still rated its King-high straight as ahead, the evaluator said.

Jev is the 🐠 at the poker table TypeSafe's new decision model held a straight on a board showing three spades. I gave it the opponent's exact cards, a made flush. It said it was ahead and bet 89.5 into a 22.5 pot, five runs out of five. There are plenty of good uses for Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev and plenty still to find. There is also going to be a regrettable stretch of this hype cycle, because people are wiring it into decisions without evaluating it. TypeSafe skipped published evals in this release, and I think that will produce a lot of naive deployments. I have built high-performance classification systems for critical domains, so I am skeptical by default. Poker is a cheap way to check: a solver computes the correct play, money rides on it, and plenty of spots are obvious to any player. Here is the eval. 1. The setup I solved one flop with TexasSolver https://github.com/bupticybee/TexasSolver : heads-up, 100bb deep, LJ opens and BTN calls, flop Q♠ 9♦ 4♠. Eight minutes, 7.6GB, 0.59% exploitability. That gives the correct strategy for every hand either player can hold, on every turn card, at every decision. Jev gets a state object describing the table the way a player sees it, and one question: which action should hero take, from the options the solver's tree offers. Pot odds, stack-to-pot ratio, hero's hand rank and outs are computed in Python first, so Jev never does arithmetic. The exact state and question { "game": { "format": "No-limit Texas Hold'em cash game", "players dealt in": 8, "blinds bb": { "small blind": 0.5, "big blind": 1 }, "units": "All amounts are in big blinds bb " }, "seats": { "position": "UTG", "starting stack": 100.0, "status": "folded preflop" }, { "position": "UTG+1", "starting stack": 100.0, "status": "folded preflop" }, { "position": "LJ", "starting stack": 100.0, "status": "in hand", "stack behind": 89.5, "is hero": true }, { "position": "HJ", "starting stack": 100.0, "status": "folded preflop" }, { "position": "CO", "starting stack": 100.0, "status": "folded preflop" }, { "position": "BTN", "starting stack": 100.0, "status": "in hand", "stack behind": 89.5 }, { "position": "SB", "starting stack": 100.0, "status": "folded preflop" }, { "position": "BB", "starting stack": 100.0, "status": "folded preflop" } , "hero": { "position": "LJ", "hole cards": "King of diamonds", "10 of diamonds" }, "board": { "flop": "Queen of spades", "9 of diamonds", "4 of spades" , "turn": "Jack of spades" }, "action history": { "preflop": "UTG folds", "UTG+1 folds", "LJ hero raises to 2.5", "HJ folds", "CO folds", "BTN calls 2.5", "SB folds", "BB folds" , "flop": "LJ hero bets 2", "BTN raises to 8", "LJ hero calls 6" , "turn": }, "pot": { "current pot": 22.5 }, "hero hand": { "made hand": "Straight, King high", "draws": , "outs to straight or better": 0, "chance to hit on next card": 0.0 }, "decision": { "street": "turn", "hero to act": true, "legal actions": "check", "all-in 89.5" , "effective stack behind before acting": 89.5, "facing bet": 0, "stack to pot ratio": 3.98 } } { "action": { "type": "choice", "instructions": "Hero is on the turn and it is hero's turn to act. Which action from decision.legal actions should hero take?", "criteria": { "check": "Check", "all in": "All-in 89.5" } } } Option labels carry no framing. Five phrasings over 30 spots, including "which action makes hero the most money in the long run" and no labels at all, matched the solver on the same 63% of spots. Value-laden verbs pushed Jev toward bigger bets, so the final wording has none. 2. An easy spot: good Hero holds J♠T♠ on Q♠ 9♦ 4♠ 2♥ facing a bet of 8 into 10.5. Fifteen outs, needs 30% equity to call, has 33%. | Option | Solver | Jev | |---|---|---| | Call 8 | 96% | 94% | | Raise to 24 | 4% | 2% | | All-in 95.5 | 0% | 2% | | Fold | 0% | 2% | Jev calls at 94%, the solver calls at 96%. 215ms. Across 30 random spots it matched the solver's top action 63% of the time. 3. A basic spot: wrong Same hand, one card changed. The turn is the J♠ instead of the J♥, so three spades are showing. Hero holds K♦T♦ for a king-high straight, and any flush beats it. Pot 22.5, both players 89.5 behind, hero first to act. The solver's tree gives this node two options, check or bet, where the only bet available is all-in for 89.5. That makes it a binary decision with no sizing to work out: put money in or do not. From here on, bet means that all-in. | Option | Solver | Jev | |---|---|---| | Check | 100% | 40% | | Bet 89.5 all-in | 0% | 60% 58–63 | Jev chose to bet in sixteen runs out of sixteen. 4. Show it the opponent's cards I added the opponent's literal cards to the state as villain hand, and a second question: is hero's hand better than the hand in villain hand? | Opponent's hand, shown to Jev | Reality | Jev: is hero ahead? | |---|---|---| | A♠8♠ flush | hero is beat | 0.71 | | A♠K♠ nut flush | hero is beat | 0.62 | | T♠8♠ flush | hero is beat | 0.84 | | 9♠9♣ set | hero ahead | 0.88 | | 7♣6♣ nothing | hero ahead | 0.94 | Three flushes shown, three times it answered that hero was ahead. It bet in all five rows, whether hero was drawing dead or far ahead. On the J♥ board, where no flush is possible, the same question got all nine comparisons right. 5. What changes the answer I added facts one at a time and ran each version several times. | What the state contained | Jev | Jev's probability | |---|---|---| | Nothing about the opponent | bet | 60% 58–63 | | + the opponent's exact cards | bet | 58% 56–60 | | + "three cards of one suit are showing" | bet | 57% 55–58 | | + the opponent's hand named: "Flush, Ace high" | check | 58% 57–61 | | + both hands named side by side | check | 74% 71–76 | | + "hero is currently behind" | check | 78% 77–79 | | + "hero has 0 outs" | check | 88% 87–89 | The last column is Jev's probability for the action it chose: the mean, then the range across runs. Five runs per row, sixteen for the first row and eleven for the second, which I sampled more than once. Run-to-run movement is a few points; the jumps between rows are 15 to 30. The first three rows contain everything a player sees at the table. None of it changed the decision. The answer changes when the state names the opponent's hand, and moves further as the state adds both hands together, then that hero is behind, then that hero has zero outs. In the last row the state contains the conclusion. The field that flips it, verbatim "villain hand": { "hole cards": "Ace of spades", "King of spades" , "hand rank": "Flush, Ace high" } With the cards alone, Jev bets across eleven runs, 0.56 to 0.60. With hand rank added, check wins every run. 6. Haiku 4.5 with thinking off Same spot on the J♥ board, where hero has the nuts and no flush is possible. Same two options, no opponent cards, no hand ranks. Jev bet this about 60% of the time. It states that a bet of four times the pot only gets called by better hands, cites the stack-to-pot ratio of 3.98, and checks. 7. The whole range at one decision Different spot from the same solve. The flop checks through and the A♥ arrives. LJ raised before the flop, so LJ holds most of the aces. The solver bets 71% of its range here, including hands with nothing. I asked Jev the same question with every hand class that reaches this spot, 54 of them, one call each. Left is the solver, right is Jev. Green is check, amber and orange are the two bet sizes. | Hand | What it is | Solver | Jev | Jev's confidence | |---|---|---|---|---| | 5♥4♥ | bottom pair | bets big 84% | checks 90% | 0.86 | | K♥J♦ | king high, nothing | bets big 79% | checks 80% | 0.73 | | 8♠7♠ | eight high, nothing | bets big 71% | checks 65% | 0.52 | | A♦K♦ | top pair, aces | bets big 82% | bets small 32% | 0.09 | | T♠T♣ | pocket tens | checks 83% | checks 77% | 0.70 | Jev bets when its own cards are good and checks when they are not. The solver bets this board with its whole range, because the ace fits the hands LJ raised with. Confidence is 0.86 on checking bottom pair, which the solver bets 84% of the time, and 0.09 on top pair, where Jev is closest to the solver. 8. Asking it the way the docs prescribe TypeSafe's docs say not to ask one broad question. Ask narrow judgments, send them in one call, combine them in code. I built that and two other shapes, and scored all of them on the same 150 random spots. - One question. Which action should hero take. - Six judgments. Hand rank against his betting range, which range the board favors, whether a free card beats hero, whether hero is behind when called, how often he folds, whether his line looks strong. Code turns the six answers into an action. - Regret per action. One yes/no per available action: taking this action is a mistake. Code picks the lowest. - Six binary facts. Best hand now, can he have better, will worse hands call, will better hands fold, can hero improve, is a free card dangerous. Then ordinary poker logic in code. I also scored three rules with no model in them: check if checking is legal otherwise call, pick at random, and always bet. None of them see the cards. Every number below is how often a strategy picked the action the solver plays most often in that spot. Perfect play scores 100%. The spots are skewed: the solver checks in 95 of the 150, so the second column drops those and keeps only the 55 where it does something else. | Strategy | All 150 spots | The 55 contested spots | |---|---|---| | Jev, one question | 63% | 38% | | Jev, six judgments + code | 57% | 44% | | Jev, regret per action | 57% | 36% | | Jev, six binary facts + code | 59% | 33% | | no model: check if legal, else call | 72% | 24% | | no model: always bet | 31% | 29% | | no model: pick at random | 37% | 25% | Read the first column as a warning about the sample, not a result: a rule that always checks scores 72% there because checking is usually right. In the second column, where the solver bets, raises, calls or folds, Jev's four shapes score 33% to 44% against 24% to 29% for the no-model rules. So Jev is doing something, and it gets the answer wrong more often than it gets it right. On the 21 spots facing a bet, the six judgments scored 71% against 62% for the single question. Across all spots the six judgments changed 47 actions: 12 matched the solver where the single question had not, 21 went the other way. Tuning thresholds on half the spots and scoring the other half gave 65%. The judgments are cached, so rewriting the composition rules costs nothing. The whole sweep was 300 calls and 363,000 input tokens, about a cent and a half. In all four shapes, Jev checks and folds hands that the solver bets. One caveat on the decomposed versions: the showcases are not decomposed. They hand the model a diff and ask whether it is secure, or a thread and ask whether to refund. That is the undecomposed use, and it is the use being sold. Limits of this eval Everything comes from one solved board. I measured agreement with the solver's top action, not EV loss, which is what costs money. The same request run sixteen times put the bet between 0.58 and 0.63, so differences under six points are noise. Evaluating five cards is mechanical, and TypeSafe's docs say to compute that in code rather than ask the model, which is what section 5 ends up doing. Takeaway TypeSafe does not publish standard benchmarks. They built their own workflow evals against expensive reference models, and their launch post lists the caveats, including that those workflows were built in-house. I prefer that to a leaderboard. It also means there is no number telling you whether Jev clears the bar for the decision you are about to hand it. You have to evaluate every situation you want to use this in, one at a time, against an answer key you trust, and decide what a wrong call costs you there. I aligned my setup as carefully as I could, including splitting the question the way their docs prescribe, and it still scored under a one-line rule. TexasSolver console build, flop Q♠9♦4♠, LJ opens 2.5 and BTN calls, 100bb, 300 iterations to 0.59% exploitability. Jev calls via POST /v1/systemone, model jev-latest resolving to jev-1.13.0, roughly 1,200 input tokens per call at 200 to 300ms.