Crosshair-Bench: Evaluating Frontier Models on an Unseen Game

A new benchmark, Crosshair-Bench, evaluated 20 OpenRouter language-model configurations across 495 complete games of a 7×7 connection-and-flip board game called Crosshair that was invented on October 25, 2026, so no model could have seen it in training. The tournament, run by TheGustafson and published in the GitHub repository TheGustafson/crosshair-bench, found that giving a smaller Flash model a real thinking budget beat running a Flagship model at low effort by a 16–4 margin. Games were played in color-swapped pairs from balanced 2-ply openings, with the az-puct-400 engine acting as a shadow oracle to score every position before and after each move.

Crosshair-Bench: Evaluating Frontier Models on an Unseen Game If you ask a language model to play Chess, Othello, or Connect Four, it is hard to tell how much of its play comes from calculating on the spot and how much comes from having read millions of games and strategy guides during training. On October 25, 2026, I invented a 7×7 connection-and-flip board game called Crosshair https://thegustafson.com/games/crosshair and trained a 114,290-parameter 457 KB AlphaZero network https://thegustafson.com/blog/crosshair-alphazero for it on one desktop GPU in six minutes. Because the game is brand new, none of the models in this tournament have ever seen a position or read a word of advice about it. Should be safe for a few generations yet The rules fit in four sentences, the board is 49 squares, and every game finishes within 49 moves with no draws. This note is the companion writeup to the Crosshair LLM Benchmark https://thegustafson.com/benchmark . The main /benchmark page https://thegustafson.com/benchmark has the full leaderboard and dedicated pages for every model and engine where you can step through all of their games. Here, we will walk through how the tournament was set up, why giving a small Flash model a real thinking budget beat running a giant Flagship model on low effort 16–4 , and replay a few of the best games. 1. How the tournament and shadow oracle work To put language models and tree-search engines on the same rating scale, we ran 495 complete games TheGustafson/crosshair-bench https://github.com/TheGustafson/crosshair-bench across five main tournament batches plus a prompt-scaffolding pilot: - Batch 0 96 games : Engine-vs-engine calibration 16 games per pair across all 6 pairs among four search rungs from ai-crosshair https://github.com/TheGustafson/ai-crosshair : random-uct-50 https://thegustafson.com/benchmark/engine-random-uct-50 , az-puct-1 https://thegustafson.com/benchmark/engine-az-puct-1 , az-puct-400 https://thegustafson.com/benchmark/engine-az-puct-400 , and az-puct-4000 https://thegustafson.com/benchmark/engine-az-puct-4000 . - Batch 1 240 games : All 20 OpenRouter LLM configurations 5 labs × 2 model tiers × 2 reasoning budgets playing 4 games each against random-uct-50 Batch 1a , az-puct-400 Batch 1b , and az-puct-4000 Batch 1c . - Batch 2 60 games : Thinking budget vs. parameter size. First, 10 intra-model pairs @high vs @low / @none , 40 games . Second, 5 cross-tier matchups pitting each lab's smaller Flash @high model against its Flagship @low / @none model 20 games . - Batch 3 40 games : A 5×5 round-robin among the five Flagship @high models gpt-6.1-sol@high https://thegustafson.com/benchmark/gpt-6-1-sol-high , gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high , claude-opus-5.5@high https://thegustafson.com/benchmark/claude-opus-5-5-high , deepseek-v4-pro@high https://thegustafson.com/benchmark/deepseek-v4-pro-high , and qwen3.8-max@high https://thegustafson.com/benchmark/qwen3-8-max-high . - Batch 4 40 games : A 5×5 round-robin among the five Flash @high models gemini-3.8-flash@high https://thegustafson.com/benchmark/gemini-3-8-flash-high , claude-haiku-5.5@high https://thegustafson.com/benchmark/claude-haiku-5-5-high , deepseek-v4.1-flash@high https://thegustafson.com/benchmark/deepseek-v4-1-flash-high , gpt-6-luna@high https://thegustafson.com/benchmark/gpt-6-luna-high , and qwen3.8-flash@high https://thegustafson.com/benchmark/qwen3-8-flash-high . - Pilot & Validation 19 games : Prompt-scaffolding ablation raw , assisted , and strategic modes and harness validation. Every matchup is played in color-swapped pairs from balanced 2-ply openings so each side plays Red twice and Blue twice from identical starting positions. All 20 OpenRouter configurations receive the exact same unassisted raw prompt: the four-sentence ruleset, a 7×7 ASCII grid a1 .. g7 , coordinate lists of Red and Blue stones, the active row/column crosshair constraint, and the explicit list of legal moves. On every turn of every game, az-puct-400 https://thegustafson.com/benchmark/engine-az-puct-400 evaluates the board position before and after the chosen move as a shadow oracle. That gives us four move-level diagnostics independent of the game outcome: 1. AZ-400 Top-1 & Top-3 Agreement: How often the player chooses the 1 move or a Top-3 move from 400-simulation AlphaZero search. 2. Raw Policy Top-1 Agreement: How often the player matches the 1-simulation neural policy prior . 3. Mean Win-Probability Loss MWPL : Average drop in 400-simulation expected win probability per turn. 4. Blunder Rate: Fraction of turns that lose at least 10% expected win probability . az-puct-4000: 2343 Elo· az-puct-400: 1888 Elo· az-puct-1 Anchor : 1200 Elo· random-uct-50: 1135 Elo | | Entrant | Lab / Tier | Anchored Elo | Overall | vs UCT50 | vs AZ400 | vs AZ4000 | AZ Top-1 | Top-3 | Blund% | Think Tok | Cost/Game | |---|---|---|---|---|---|---|---|---|---|---|---|---| | 1 | az-puct-4000 https://thegustafson.com/benchmark/engine-az-puct-4000 | Engine | 2343 | 127–1 | · | · | · | 82.9% | 96.5% | 0.0% | 0 | $0.00 | | 2 | az-puct-400 https://thegustafson.com/benchmark/engine-az-puct-400 | Engine | 1888 | 117–18 | · | · | · | 99.9% | 100.0% | 0.6% | 0 | $0.00 | | 3 | gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high | Google · Flag | 1440 | 19–13 | 4–0 | 1–3 | 0–4 | 42.2% | 75.1% | 11.6% | 2,152 | $0.390 | | 4 | gpt-6.1-sol@low https://thegustafson.com/benchmark/gpt-6-1-sol-low | OpenAI · Flag | 1408 | 9–11 | 4–0 | 0–4 | 0–4 | 39.9% | 65.6% | 12.0% | 588 | $0.137 | | 5 | gpt-6.1-sol@high https://thegustafson.com/benchmark/gpt-6-1-sol-high | OpenAI · Flag | 1398 | 17–15 | 4–0 | 0–4 | 0–4 | 40.4% | 68.3% | 12.8% | 3,060 | $0.488 | | 6 | gemini-3.8-flash@high https://thegustafson.com/benchmark/gemini-3-8-flash-high | Google · Flash | 1328 | 20–16 | 1–3 | 0–4 | 0–4 | 46.3% | 68.0% | 14.4% | 2,649 | $0.166 | | 7 | gemini-4-argon@low https://thegustafson.com/benchmark/gemini-4-argon-low | Google · Flag | 1290 | 6–14 | 3–1 | 0–4 | 0–4 | 38.8% | 67.2% | 16.7% | 0 | $0.095 | | 8 | claude-opus-5.5@high https://thegustafson.com/benchmark/claude-opus-5-5-high | Anthropic · Flag | 1259 | 14–18 | 3–1 | 0–4 | 0–4 | 36.0% | 59.2% | 17.2% | 1,213 | $0.551 | | 9 | claude-haiku-5.5@low https://thegustafson.com/benchmark/claude-haiku-5-5-low | Anthropic · Flash | 1255 | 5–11 | 3–1 | 0–4 | 0–4 | 38.3% | 56.9% | 21.3% | 995 | $0.010 | | 10 | claude-haiku-5.5@high https://thegustafson.com/benchmark/claude-haiku-5-5-high | Anthropic · Flash | 1223 | 16–20 | 2–2 | 0–4 | 0–4 | 38.0% | 57.6% | 21.9% | 2,559 | $0.019 | | 11 | deepseek-v4.1-flash@high https://thegustafson.com/benchmark/deepseek-v4-1-flash-high | DeepSeek · Flash | 1211 | 18–18 | 2–2 | 0–4 | 0–4 | 35.1% | 62.4% | 16.6% | 6,143 | $0.043 | | 12 | az-puct-1 https://thegustafson.com/benchmark/engine-az-puct-1 | Engine | 1200 | 12–45 | · | · | · | 54.2% | 91.7% | 12.5% | 0 | $0.00 | | 13 | gpt-6-luna@high https://thegustafson.com/benchmark/gpt-6-luna-high | OpenAI · Flash | 1158 | 13–23 | 2–2 | 0–4 | 0–4 | 36.5% | 65.2% | 17.0% | 2,845 | $0.021 | | 14 | deepseek-v4-pro@high https://thegustafson.com/benchmark/deepseek-v4-pro-high | DeepSeek · Flag | 1148 | 11–21 | 3–1 | 0–4 | 0–4 | 38.7% | 63.7% | 15.7% | 4,486 | $0.321 | | 15 | random-uct-50 https://thegustafson.com/benchmark/engine-random-uct-50 | Engine | 1135 | 46–83 | · | · | · | 29.0% | 57.0% | 20.7% | 0 | $0.00 | | 16 | gemini-3.8-flash@low https://thegustafson.com/benchmark/gemini-3-8-flash-low | Google · Flash | 1135 | 3–13 | 3–1 | 0–4 | 0–4 | 32.2% | 59.9% | 14.8% | 367 | $0.037 | | 17 | claude-opus-5.5@low https://thegustafson.com/benchmark/claude-opus-5-5-low | Anthropic · Flag | 1085 | 4–16 | 2–2 | 0–4 | 0–4 | 31.8% | 53.6% | 22.2% | 550 | $0.312 | | 18 | qwen3.8-flash@high https://thegustafson.com/benchmark/qwen3-8-flash-high | Qwen · Flash | 1036 | 11–25 | 1–3 | 0–4 | 0–4 | 28.1% | 61.5% | 19.7% | 1,453 | $0.013 | | 19 | qwen3.8-max@high https://thegustafson.com/benchmark/qwen3-8-max-high | Qwen · Flag | 978 | 6–26 | 1–3 | 0–4 | 0–4 | 31.6% | 59.9% | 18.8% | 1,513 | $0.183 | | 20 | gpt-6-luna@low https://thegustafson.com/benchmark/gpt-6-luna-low | OpenAI · Flash | 970 | 2–14 | 0–4 | 0–4 | 0–4 | 29.5% | 60.7% | 17.6% | 735 | $0.008 | | 21 | qwen3.8-flash@none https://thegustafson.com/benchmark/qwen3-8-flash-none | Qwen · Flash | 909 | 2–14 | 0–4 | 0–4 | 0–4 | 21.9% | 45.7% | 23.7% | 0 | $0.008 | | 22 | deepseek-v4-pro@none https://thegustafson.com/benchmark/deepseek-v4-pro-none | DeepSeek · Flag | 899 | 2–18 | 1–3 | 0–4 | 0–4 | 20.4% | 48.2% | 21.2% | 0 | $0.021 | | 23 | qwen3.8-max@none https://thegustafson.com/benchmark/qwen3-8-max-none | Qwen · Flag | 866 | 3–17 | 1–3 | 0–4 | 0–4 | 30.1% | 57.1% | 16.3% | 1,424 | $0.172 | | 24 | deepseek-v4.1-flash@none https://thegustafson.com/benchmark/deepseek-v4-1-flash-none | DeepSeek · Flash | 865 | 1–15 | 1–3 | 0–4 | 0–4 | 17.1% | 51.7% | 19.3% | 0 | $0.003 | az-puct-1 = 1200.0 Elo the vertical amber line on the Elo bar . Against az-puct-4000 4,000 MCTS simulations per move, ~279 ms , all 20 OpenRouter LLM configurations went a combined 0–80. Against az-puct-400 ~26 ms/move , gemini-4-argon@high scored the sole victory 1–79 overall . 2. Where frontier LLMs sit on the AlphaZero search ladder Matching the 20 OpenRouter configurations against the Monte Carlo Tree Search https://thegustafson.com/blog/mcts-is-awesome and AlphaZero https://thegustafson.com/blog/crosshair-alphazero baselines in Batch 1 240 games shows where scratchpad reasoning sits relative to explicit tree search: - Top reasoning LLMs reliably beat random-uct-50 1135 Elo and outscore the zero-search policy net az-puct-1 , 1200 Elo : gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high 1440 Elo , gpt-6.1-sol@low https://thegustafson.com/benchmark/gpt-6-1-sol-low 1408 Elo , and gpt-6.1-sol@high https://thegustafson.com/benchmark/gpt-6-1-sol-high 1398 Elo all swept random-uct-50 4–0 . Nine of the twenty LLM configurations rated above az-puct-1 1200 Elo , and eleven rated above random-uct-50 1135 Elo . When the neural net isn't allowed to search ahead, a frontier model thinking out loud in its scratchpad can genuinely outplay it. - Turning on 400 MCTS simulations az-puct-400 , 1888 Elo flips the score to 79–1 : Across 80 games in Batch 1b, 19 of the 20 LLM configurations were swept 0–4 . Only gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high won a game 1–3 , converting a 38-ply Blue victory when az-puct-400 overpushed at f7 on ply 32. - At 4,000 simulations az-puct-4000 , 2343 Elo , the 114K network went 80–0 : Not a single LLM configuration won a game in Batch 1c. Even when gpt-6.1-sol@high spent 70 seconds and 3,060 reasoning tokens per move $0.49 per game , az-puct-4000 0.28 seconds per move, $0.00 locked the board by ply 16. Turns out fifty cents and a full minute of prose per move is still no match for a quarter-second of real tree search. 3. Thinking budget beats model size 16–4 In Batch 2 60 games , we tested how much of a model's playing strength comes from its base parameter count versus how long it is allowed to think on each turn: 1. Same-model thinking ablation 40 games : Each of the 10 models played a 4-game match between its @high reasoning configuration and its @low or @none configuration. @high won 25–15 62.5% and gained an average of +149 Elo across the 10 pairs. 2. Cross-tier showdown 20 games : For each of the five labs, we matched the smaller Flash @high model directly against the same lab's Flagship @low / @none model. The smaller Flash models with @high thinking beat their own labs' low-effort Flagships 16–4 80.0% , usually at a fraction of the cost per game: - Anthropic 4–0 : claude-haiku-5.5@high https://thegustafson.com/benchmark/claude-haiku-5-5-high 1223 Elo , 2,559 reasoning tokens/turn, $0.019/game swept claude-opus-5.5@low https://thegustafson.com/benchmark/claude-opus-5-5-low 1085 Elo , 550 reasoning tokens/turn, $0.312/game 4–0 at 16× lower cost per game . - DeepSeek 4–0 : deepseek-v4.1-flash@high https://thegustafson.com/benchmark/deepseek-v4-1-flash-high 1211 Elo , $0.043/game swept deepseek-v4-pro@none https://thegustafson.com/benchmark/deepseek-v4-pro-none 899 Elo , $0.021/game 4–0 . - Qwen 4–0 : qwen3.8-flash@high https://thegustafson.com/benchmark/qwen3-8-flash-high 1036 Elo , $0.013/game swept qwen3.8-max@none https://thegustafson.com/benchmark/qwen3-8-max-none 866 Elo , $0.172/game 4–0 at 13× lower cost . - Google 2–2 & OpenAI 2–2 : gemini-3.8-flash@high https://thegustafson.com/benchmark/gemini-3-8-flash-high split 2–2 with gemini-4-argon@low https://thegustafson.com/benchmark/gemini-4-argon-low , and gpt-6-luna@high https://thegustafson.com/benchmark/gpt-6-luna-high split 2–2 with gpt-6.1-sol@low https://thegustafson.com/benchmark/gpt-6-1-sol-low 6.6× cheaper . Not a single low-effort Flagship won its mini-match against its own lab's smaller Flash sibling. When you cannot coast on memorized opening patterns, having tokens to actually trace moves on the board matters much more than model size. | Lab | @high Entrant | @low / @none Entrant | Direct H2H | @high Elo | @low Elo | Δ Elo | Think Tokens/Turn | |---|---|---|---|---|---|---|---| | Google Flagship | gemini-4-argon@high | gemini-4-argon@low | 3–1 | 1440 | 1290 | +150 | 2,152 vs 0 | | Google Flash | gemini-3.8-flash@high | gemini-3.8-flash@low | 4–0 | 1328 | 1135 | +193 | 2,649 vs 367 | | OpenAI Flagship | gpt-6.1-sol@high | gpt-6.1-sol@low | 1–3 | 1398 | 1408 | -11 | 3,060 vs 588 | | OpenAI Flash | gpt-6-luna@high | gpt-6-luna@low | 2–2 | 1158 | 970 | +188 | 2,845 vs 735 | | Anthropic Flagship | claude-opus-5.5@high | claude-opus-5.5@low | 2–2 | 1259 | 1085 | +174 | 1,213 vs 550 | | Anthropic Flash | claude-haiku-5.5@high | claude-haiku-5.5@low | 2–2 | 1223 | 1255 | -33 | 2,559 vs 995 | | DeepSeek Flagship | deepseek-v4-pro@high | deepseek-v4-pro@none | 3–1 | 1148 | 899 | +248 | 4,486 vs 0 | | DeepSeek Flash | deepseek-v4.1-flash@high | deepseek-v4.1-flash@none | 4–0 | 1211 | 865 | +347 | 6,143 vs 0 | | Qwen Flagship | qwen3.8-max@high | qwen3.8-max@none | 2–2 | 978 | 866 | +112 | 1,513 vs 1,424 | | Qwen Flash | qwen3.8-flash@high | qwen3.8-flash@none | 2–2 | 1036 | 909 | +127 | 1,453 vs 0 | 4. Inside the Flagship and Flash 5×5 round-robins To get a direct head-to-head ordering among the five labs, we ran two 5×5 round-robin tournaments 40 games each, 4 games per pair with swapped colors and openings : Batch 3: Flagship @high round-robin - gpt-6.1-sol@high https://thegustafson.com/benchmark/gpt-6-1-sol-high 12–4 , 75.0% finished first inside the round-robin by sweeping claude-opus-5.5@high 4–0 and qwen3.8-max@high 4–0 , while splitting 2–2 with gemini-4-argon@high and deepseek-v4-pro@high . - gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high 11–5 , 68.8% finished one game back in the round-robin and 1 overall among LLMs at 1440 Elo thanks to its 1–3 win over az-puct-400 and 3–1 win over @low , sweeping deepseek-v4-pro@high 4–0 and qwen3.8-max@high 4–0 . - Opus had Argon's number 3–1 : Even though gpt-6.1-sol@high swept claude-opus-5.5@high https://thegustafson.com/benchmark/claude-opus-5-5-high 4–0 , Opus 9–7 overall turned around and beat gemini-4-argon@high 3–1 . Batch 4: Flash @high round-robin - gemini-3.8-flash@high https://thegustafson.com/benchmark/gemini-3-8-flash-high 13–3 , 81.3% , 1328 Elo ran away with the Flash bracket, sweeping gpt-6-luna@high 4–0 and deepseek-v4.1-flash@high 4–0 , beating claude-haiku-5.5@high 3–1 , and splitting 2–2 with qwen3.8-flash@high . - gemini-3.8-flash@high also recorded the highest 400-simulation AlphaZero Top-1 move agreement of any OpenRouter model in the entire tournament 46.3% across 512 turns , finishing ahead of three of the five Flagship @high models on the overall Elo leaderboard at $0.166 per game. Apparently Flash did not get the memo that it was supposed to stay in the budget bracket. 5. Walking through the most interesting games in the tournament Every game log records the active player's chain-of-thought alongside the shadow 400-simulation AlphaZero evaluation before and after every move. You can step through six featured moments in the interactive board below four from the OpenRouter tournament listed here, plus two from the prompt-scaffolding pilot in Section 6 , or visit any model's dedicated benchmark page https://thegustafson.com/benchmark to browse all of its games: 1. Moment 1 b01 gemini4argon high vs az400 g02 , Ply 33 : The sole LLM win out of 80 games in Batch 1b against 400-simulation AlphaZero. Playing Blue in raw mode, gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high survives the middlegame and spots that Red's push to f7 on ply 32 leaves Row 6 and Column a exposed. Argon plays f6 on ply 33 77.8% - 89.0% , AZ-400 search rank 4 , then finds AZ-400's 1 move on ply 35 a6 and ply 37 a1 to convert in 38 plies. 2. Moment 2 b02 opus55 low vs haiku55 high g01 , Ply 28 : Why thinking budget beats parameter count. claude-opus-5.5@low https://thegustafson.com/benchmark/claude-opus-5-5-low 550 reasoning tokens/turn holds an 81.1% winning position as Red against claude-haiku-5.5@high https://thegustafson.com/benchmark/claude-haiku-5-5-high 2,559 reasoning tokens/turn . Opus plays b7 to claim a bottom-edge stone without checking what activating Column b gives Blue; Haiku immediately seizes the a / b file and wins in 32 plies. 3. Moment 3 b03 openai vs google g01 , Ply 24 : Flagship clash between gpt-6.1-sol@high https://thegustafson.com/benchmark/gpt-6-1-sol-high Red and gemini-4-argon@high https://thegustafson.com/benchmark/gemini-4-argon-high Blue . Sol builds two parallel vertical columns down b and d , setting up a clean fork where both c4 and c5 finish Red's top-to-bottom connection. 4. Moment 4 b01 gpt61sol high vs az4000 g01 , Ply 15 : Why 4,000-simulation AlphaZero az-puct-4000 https://thegustafson.com/benchmark/engine-az-puct-4000 went 80–0 . By ply 15 b5 , the 114K network has built an unstoppable left-to-right bridge. On ply 16, gpt-6.1-sol@high spends 70 seconds calculating every branch before writing: "Every legal move permits an immediate Blue win. I choose b4 as the direct blocking attempt, although it cannot save the position." At least it knew it was cooked. 6. Why LLMs blunder: board parsing vs. lookahead depth When an LLM throws away a winning endgame, is it misreading the ASCII board, or is it just not looking enough moves ahead? In our pilot study with Antigravity pro and flash , we tested that directly using three levels of prompt help: 1. Raw Board + Legal Moves raw : Exact rules, 7×7 ASCII grid a1 through g7 , stone lists for Red and Blue, opponent's last move, and the explicit list of legal moves. 2. Tactical Previews assisted : On top of raw , the prompt includes the engine's 1-ply hover calculation for every legal move: which enemy stones that move immediately flips including cascades , whether it wins immediately, and which row/column or wildcard it forces the opponent into next. 3. Strategic Primer + Previews strategic : On top of assisted , we add four short bullets of basic tactical advice: 1 diagonals do not connect roads, 2 do not play right next to an enemy stone along an open line where their next crosshair move can sandwich you, 3 watch out for the 4th corner of a 2×2 diagonal crosscut, and 4 route the opponent into full or dead lines. Comparing the three prompting modes shows two clear limits: Giving the model 1-ply previews still leaves 2-ply traps wide open In flash assisted az1 R Moment 6 in the replay explorer above , flash held an 83.1% winning position on ply 32 with full 1-ply tactical previews turned on. It chose g2 , writing: "Connect g2 to g1 to establish a two-stone top-edge group and create a winning threat via f2." The 1-ply preview accurately told flash that g2 flipped zero stones and sent Blue to Row 2 or Column g. What a 1-ply preview does not spell out is what happens one move later: playing in Row 2 gives Blue permission to play in Row 2 before Red gets another turn, so Blue simply plays f2 and wins on the spot 83.1% - 0.0% . Handing flash the 4-sentence strategic primer fixed that habit and dropped its blunders to zero against az-puct-1 . Three plies of scratchpad math vs. five plies of tree search Across all three prompt modes, pro went 4–0 with zero blunders against the zero-search policy net az-puct-1 , and with full strategic scaffolding pro strategic https://thegustafson.com/benchmark/pilot-pro-strategic , 1815 Anchored Elo it split 1–1 against 400-simulation AlphaZero az-puct-400 . Those two games against az-puct-400 show right where scratchpad lookahead runs out of gas: - When the trap is 2–3 plies deep, pro catches it pro strategic az400 R , ply 24 : On ply 23, az-puct-400 played a sacrifice bait at b5 , inviting Red to sandwich it at b6 . Looking two plies ahead, pro saw that playing b6 would send Blue to a6 for a 2×2 diagonal crosscut on a7 , declined the bait with b2 , and won in 45 plies. - When the refutation is 5 plies deep, both the LLM and the raw policy net miss it pro strategic az400 B , ply 37 : Playing Blue, pro outplayed az-puct-400 for 36 plies to reach a 97.5% winning position . Looking for a clean finish, pro worked through three branches in its scratchpad and played a1 which was also the 1 pick of the raw 1-sim policy prior at P = 46.7% . Its calculation on the f1 branch stopped 3 plies deep, missing that az-puct-400 could ignore the right edge after f1 - f6 and play a6 - a5 , horizontally sandwiching b5 against c5 to connect all seven columns 97.5% - 10.6% , -86.9% . | Model | Prompt Mode | Engine Opponent | W–L | LLM Turns | AZ Top-1 | AZ Top-3 | Avg ΔWin | Blunders | |---|---|---|---|---|---|---|---|---| | pro | raw | az-puct-1 | 1–0 | 17 | 47.1% | 76.5% | -0.8% | 0 | | pro | assisted | az-puct-1 | 2–0 | 36 | 52.8% | 75.0% | -1.5% | 0 | | pro | strategic | az-puct-1 | 1–0 | 18 | 33.3% | 66.7% | -1.5% | 0 | | pro | raw | az-puct-400 | 0–2 | 45 | 28.9% | 71.1% | -5.8% | 5 | | pro | strategic | az-puct-400 | 1–1 | 44 | 45.5% | 79.5% | -4.1% | 4 | | flash | raw | az-puct-1 | 1–1 | 36 | 30.6% | 66.7% | -7.0% | 8 | | flash | assisted | az-puct-1 | 0–1 | 17 | 41.2% | 52.9% | -7.4% | 2 | | flash | strategic | az-puct-1 | 1–0 | 13 | 46.2% | 53.8% | -1.5% | 0 | | flash | strategic | random-uct-50 | 1–0 | 16 | 43.8% | 81.3% | -3.0% | 1 | | flash | raw | az-puct-400 | 1–1 | 45 | 40.0% | 82.2% | -3.8% | 4 | pro the 4-rule strategic primer and 1-ply previews lifts its performance from 0–2 in raw mode against az-puct-400 to a 1–1 split 1815 Anchored Elo . 7. Takeaways and running Crosshair-Bench yourself Testing language models on a board game that did not exist during training strips away memorized openings and leaves pure in-context calculation: 1. Scratchpad calculation beats 1-ply intuition, but loses to real tree search: Frontier reasoning models reliably beat 50-rollout MCTS random-uct-50 , 1135 Elo and the zero-search 114K policy network az-puct-1 , 1200 Elo , but go 1–79 against 400-simulation AlphaZero 26 ms/move and 0–80 against 4,000 simulations 279 ms/move . 2. Test-time reasoning tokens beat parameter count: Across all five labs, smaller Flash models running with @high thinking beat their own labs' Flagship models running with @low or disabled thinking 16–4 , often at 6× to 16× lower cost per game. 3. Explore the games or run the suite: You can inspect every participant's head-to-head stats and step through every game on the Crosshair Benchmark page https://thegustafson.com/benchmark , play against the Rust WASM engines yourself at /games/crosshair https://thegustafson.com/games/crosshair , or run the open-source match harness and 100-position single-turn tactical benchmark Crosshair-Bench-100 from TheGustafson/crosshair-bench https://github.com/TheGustafson/crosshair-bench .