TypeSafe Jev Played Chess — And Landed Next to Reasoning Models

A developer integrated TypeSafe's Jev model into the LLM Chess benchmark harness, adapting the classifier-style API to return UCI chess moves and landing the model at roughly #59 on the leaderboard with an Elo near 243, alongside qwen3.6-27b and o4-mini-medium. Jev achieved a flat 50% draw rate against Dragon at levels L1 through L3 while costing about $0.0015 per game and 119 tokens, far below chat-model competitors. The developer noted that Jev's fixed-label design suits selecting from a list of legal moves better than free-form text generation, which produced near-words like 'bue' and 'helhoh' in a spelling test.

TypeSafe's Jev https://docs.typesafe.ai/introduction is an odd one. It isn't a chat model - you send a state plus typed questions Choice / Score / Noul and get labels with probabilities — like a classifier, except the input can be free text and you choose the labels each time. I've been running models through LLM Chess https://maxim-saplin.github.io/llm chess/ for a while. LLMs play in an agentic setting with a multi-turn protocol: get current board , get legal moves , make move <uci . Jev doesn't do that. Can you still put it on the same leaderboard? Apparently yes — if you talk to it the way it expects. TypeSafe is openly anti-benchmaxxing https://typesafe.ai/blog/antibenchmaxxing . In the System One / Jev launch post https://typesafe.ai/blog/introducing-system-one-models-and-jev they say existing LLM benchmarks don't really apply here, and they skip the usual public-eval scoreboard putting it's model next to other models. They prefer dated internal snapshots instead. I put Jev on LLM Chess https://maxim-saplin.github.io/llm chess/ anyway — same harness, same opponents as the chat models. It reminded me of talking to Stephen Hawking: smart, but no free speech. Answers do not come out in a free from Outside chess I tried a silly next-character loop — Choice over a–z plus <eos , one API call per letter. Asked to spell blue or finish hel… , it spat out almost-words bue , helhoh . Started from pari… and it finished paris . Generating from scratch was weak; finishing a partial worked better. A fixed list of legal chess moves fits this model much better than spelling English letter by letter. Each ply: { fen, side to move } Jev returns a UCI string. It turns into make move e7e5 — same action chat models send to the eval runner, on the outside seems almost like normal eval interaction. Pricing: $0.042 / 1M input tokens , output free. On the current leaderboard jev-latest is around 59 , Elo ~243 , sitting next to qwen3.6-27b and o4-mini-medium : | | Player | Elo | Game Duration | Tokens | Cost/Game | Time/Game | Games | |---|---|---|---|---|---|---|---| | 58 | qwen3.6-27b@q4 k s | 270.0±148.6 | 100% | 5956 | $1.46 | 7.54h | 27 | | 59 | jev-latest | 242.9±117.5 | 100% | 119 | $0.0015 | 35.6s | 80 | | 60 | o4-mini-medium | 240.3±64.3 | 100% | 2156 | $1.11 | 41.5m | 199 | Dragon white vs Jev black , 10 games per level: | Dragon | Dragon wins | Draws | Jev wins | |---|---|---|---| | L1 | 5 | 5 | 0 | | L2 | 5 | 5 | 0 | | L3 | 5 | 5 | 0 | 50% draws at L1, L2, and L3. Flat. With most chat models, stronger Dragon usually means fewer draws — the engine finishes more games. Here the draw rate doesn't drop across three levels. That helps the Elo. TypeSafe calls these System One models: fast decisions, no deep thinking. Chess still needs real choices — what to capture, when to trade, when to aim for a draw. Jev has no chat, costs fractions of a cent per game, and still lands near mid-pack reasoning models on Elo. So the skill isn't "writing about chess." It's picking a move from a list. TypeSafe doesn't want a public leaderboard as their proof. This is one outside eval. bue / helhoh . Leaderboard: LLM Chess https://maxim-saplin.github.io/llm chess/ Harness: maxim-saplin/llm chess https://github.com/maxim-saplin/llm chess feat/typesafe-jev-player / TypeSafe Jev player