TypeSafe's Jev is an odd one. It isn't a chat model - you send a state plus typed questions (Choice / Score / Noul) and get labels with probabilities — like a classifier, except the input can be free text and you choose the labels each time.
I've been running models through LLM Chess for a while. LLMs play in an agentic setting with a multi-turn protocol: get_current_board, get_legal_moves, make_move <uci>. Jev doesn't do that. Can you still put it on the same leaderboard?
Apparently yes — if you talk to it the way it expects.
TypeSafe is openly anti-benchmaxxing. In the System One / Jev launch post they say existing LLM benchmarks don't really apply here, and they skip the usual public-eval scoreboard putting it's model next to other models. They prefer dated internal snapshots instead.
I put Jev on LLM Chess anyway — same harness, same opponents as the chat models.
It reminded me of talking to Stephen Hawking: smart, but no free speech. Answers do not come out in a free from
Outside chess I tried a silly next-character loop — Choice over a–z plus <eos>, one API call per letter. Asked to spell blue or finish hel…, it spat out almost-words (bue, helhoh). Started from pari… and it finished paris. Generating from scratch was weak; finishing a partial worked better. A fixed list of legal chess moves fits this model much better than spelling English letter by letter.
Each ply:
{ fen, side_to_move }
Jev returns a UCI string. It turns into make_move e7e5 — same action chat models send to the eval runner, on the outside seems almost like normal eval interaction.
Pricing: $0.042 / 1M input tokens, output free.
On the current leaderboard jev-latest is around #59, Elo ~243, sitting next to qwen3.6-27b and o4-mini-medium:
| # | Player | Elo | Game Duration | Tokens | Cost/Game | Time/Game | Games |
|---|---|---|---|---|---|---|---|
| 58 | qwen3.6-27b@q4_k_s | 270.0±148.6 | 100% | 5956 | $1.46 | 7.54h | 27 |
| 59 | jev-latest | 242.9±117.5 | 100% | 119 | $0.0015 | 35.6s | 80 |
| 60 | o4-mini-medium | 240.3±64.3 | 100% | 2156 | $1.11 | 41.5m | 199 |
Dragon (white) vs Jev (black), 10 games per level:
| Dragon | Dragon wins | Draws | Jev wins |
|---|---|---|---|
| L1 | 5 | 5 | 0 |
| L2 | 5 | 5 | 0 |
| L3 | 5 | 5 | 0 |
50% draws at L1, L2, and L3. Flat. With most chat models, stronger Dragon usually means fewer draws — the engine finishes more games. Here the draw rate doesn't drop across three levels. That helps the Elo.
TypeSafe calls these System One models: fast decisions, no deep thinking.
Chess still needs real choices — what to capture, when to trade, when to aim for a draw. Jev has no chat, costs fractions of a cent per game, and still lands near mid-pack reasoning models on Elo. So the skill isn't "writing about chess." It's picking a move from a list.
TypeSafe doesn't want a public leaderboard as their proof. This is one outside eval.
bue / helhoh).
Leaderboard: [LLM Chess](https://maxim-saplin.github.io/llm_chess/)
Harness: [maxim-saplin/llm_chess](https://github.com/maxim-saplin/llm_chess) (`feat/typesafe-jev-player` / TypeSafe Jev player)