{"slug": "crosshair-bench-evaluating-frontier-models-on-an-unseen-game", "title": "Crosshair-Bench: Evaluating Frontier Models on an Unseen Game", "summary": "A new benchmark, Crosshair-Bench, evaluated 20 OpenRouter language-model configurations across 495 complete games of a 7×7 connection-and-flip board game called Crosshair that was invented on October 25, 2026, so no model could have seen it in training. The tournament, run by TheGustafson and published in the GitHub repository TheGustafson/crosshair-bench, found that giving a smaller Flash model a real thinking budget beat running a Flagship model at low effort by a 16–4 margin. Games were played in color-swapped pairs from balanced 2-ply openings, with the az-puct-400 engine acting as a shadow oracle to score every position before and after each move.", "body_md": "# Crosshair-Bench: Evaluating Frontier Models on an Unseen Game\n\nIf you ask a language model to play Chess, Othello, or Connect Four, it is hard to tell how much of its play comes from calculating on the spot and how much comes from having read millions of games and strategy guides during training.\n\nOn October 25, 2026, I invented a 7×7 connection-and-flip board game called [Crosshair](https://thegustafson.com/games/crosshair) and [trained a 114,290-parameter (`457 KB`) AlphaZero network](https://thegustafson.com/blog/crosshair-alphazero) for it on one desktop GPU in six minutes. Because the game is brand new, none of the models in this tournament have ever seen a position or read a word of advice about it. (Should be safe for a few generations yet!) The rules fit in four sentences, the board is 49 squares, and every game finishes within 49 moves with no draws.\n\nThis note is the companion writeup to the **[Crosshair LLM Benchmark](https://thegustafson.com/benchmark)**. The [main `/benchmark` page](https://thegustafson.com/benchmark) has the full leaderboard and dedicated pages for every model and engine where you can step through all of their games. Here, we will walk through how the tournament was set up, why giving a small Flash model a real thinking budget beat running a giant Flagship model on low effort (`16–4`), and replay a few of the best games.\n\n## 1. How the tournament and shadow oracle work\n\nTo put language models and tree-search engines on the same rating scale, we ran **495 complete games** ([`TheGustafson/crosshair-bench`](https://github.com/TheGustafson/crosshair-bench)) across five main tournament batches plus a prompt-scaffolding pilot:\n\n- **Batch 0 (`96 games`):** Engine-vs-engine calibration (` 16 games` per pair across all 6 pairs) among four search rungs from[`ai-crosshair`](https://github.com/TheGustafson/ai-crosshair) :[` random-uct-50`](https://thegustafson.com/benchmark/engine-random-uct-50) ,[` az-puct-1`](https://thegustafson.com/benchmark/engine-az-puct-1) ,[` az-puct-400`](https://thegustafson.com/benchmark/engine-az-puct-400) , and[` az-puct-4000`](https://thegustafson.com/benchmark/engine-az-puct-4000) .\n- **Batch 1 (`240 games`):** All 20 OpenRouter LLM configurations (` 5 labs × 2 model tiers × 2 reasoning budgets` ) playing 4 games each against`random-uct-50` (Batch 1a),`az-puct-400` (Batch 1b), and`az-puct-4000` (Batch 1c).\n- **Batch 2 (`60 games`):** Thinking budget vs. parameter size. First, 10 intra-model pairs (`@high` vs`@low` /`@none` , 40 games). Second, 5 cross-tier matchups pitting each lab's smaller**Flash `@high`** model against its**Flagship `@low`/`@none`** model (20 games).\n- **Batch 3 (`40 games`):** A 5×5 round-robin among the five Flagship`@high` models ([`gpt-6.1-sol@high`](https://thegustafson.com/benchmark/gpt-6-1-sol-high) ,[` gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high) ,[` claude-opus-5.5@high`](https://thegustafson.com/benchmark/claude-opus-5-5-high) ,[` deepseek-v4-pro@high`](https://thegustafson.com/benchmark/deepseek-v4-pro-high) , and[` qwen3.8-max@high`](https://thegustafson.com/benchmark/qwen3-8-max-high) ).\n- **Batch 4 (`40 games`):** A 5×5 round-robin among the five Flash`@high` models ([`gemini-3.8-flash@high`](https://thegustafson.com/benchmark/gemini-3-8-flash-high) ,[` claude-haiku-5.5@high`](https://thegustafson.com/benchmark/claude-haiku-5-5-high) ,[` deepseek-v4.1-flash@high`](https://thegustafson.com/benchmark/deepseek-v4-1-flash-high) ,[` gpt-6-luna@high`](https://thegustafson.com/benchmark/gpt-6-luna-high) , and[` qwen3.8-flash@high`](https://thegustafson.com/benchmark/qwen3-8-flash-high) ).\n- **Pilot & Validation (`19 games`):** Prompt-scaffolding ablation (` raw` ,`assisted` , and`strategic` modes) and harness validation.\n\nEvery matchup is played in color-swapped pairs from balanced 2-ply openings so each side plays Red twice and Blue twice from identical starting positions. All 20 OpenRouter configurations receive the exact same unassisted `raw` prompt: the four-sentence ruleset, a 7×7 ASCII grid (`a1`..` g7`), coordinate lists of Red and Blue stones, the active row/column crosshair constraint, and the explicit list of legal moves.\n\nOn every turn of every game, **[`az-puct-400`](https://thegustafson.com/benchmark/engine-az-puct-400)** evaluates the board position before and after the chosen move as a shadow oracle. That gives us four move-level diagnostics independent of the game outcome:\n\n1. **AZ-400 Top-1 & Top-3 Agreement:** How often the player chooses the #1 move (or a Top-3 move) from 400-simulation AlphaZero search.\n2. **Raw Policy Top-1 Agreement:** How often the player matches the 1-simulation neural policy prior .\n3. **Mean Win-Probability Loss (`MWPL`):** Average drop in 400-simulation expected win probability per turn.\n4. **Blunder Rate:** Fraction of turns that lose at least`10%` expected win probability ().\n\n**az-puct-4000:** 2343 Elo·\n\n**az-puct-400:** 1888 Elo·\n\n**az-puct-1 (Anchor):** 1200 Elo·\n\n**random-uct-50:** 1135 Elo\n\n| # | Entrant | Lab / Tier | Anchored Elo | Overall | vs UCT50 | vs AZ400 | vs AZ4000 | AZ Top-1 | Top-3 | Blund% | Think Tok | Cost/Game | \n|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| 1 | [az-puct-4000](https://thegustafson.com/benchmark/engine-az-puct-4000) | Engine | 2343 | 127–1 | · | · | · | 82.9% | 96.5% | 0.0% | 0 | $0.00 | \n| 2 | [az-puct-400](https://thegustafson.com/benchmark/engine-az-puct-400) | Engine | 1888 | 117–18 | · | · | · | 99.9% | 100.0% | 0.6% | 0 | $0.00 | \n| 3 | [gemini-4-argon@high](https://thegustafson.com/benchmark/gemini-4-argon-high) | Google · Flag | 1440 | 19–13 | 4–0 | 1–3 | 0–4 | 42.2% | 75.1% | 11.6% | 2,152 | $0.390 | \n| 4 | [gpt-6.1-sol@low](https://thegustafson.com/benchmark/gpt-6-1-sol-low) | OpenAI · Flag | 1408 | 9–11 | 4–0 | 0–4 | 0–4 | 39.9% | 65.6% | 12.0% | 588 | $0.137 | \n| 5 | [gpt-6.1-sol@high](https://thegustafson.com/benchmark/gpt-6-1-sol-high) | OpenAI · Flag | 1398 | 17–15 | 4–0 | 0–4 | 0–4 | 40.4% | 68.3% | 12.8% | 3,060 | $0.488 | \n| 6 | [gemini-3.8-flash@high](https://thegustafson.com/benchmark/gemini-3-8-flash-high) | Google · Flash | 1328 | 20–16 | 1–3 | 0–4 | 0–4 | 46.3% | 68.0% | 14.4% | 2,649 | $0.166 | \n| 7 | [gemini-4-argon@low](https://thegustafson.com/benchmark/gemini-4-argon-low) | Google · Flag | 1290 | 6–14 | 3–1 | 0–4 | 0–4 | 38.8% | 67.2% | 16.7% | 0 | $0.095 | \n| 8 | [claude-opus-5.5@high](https://thegustafson.com/benchmark/claude-opus-5-5-high) | Anthropic · Flag | 1259 | 14–18 | 3–1 | 0–4 | 0–4 | 36.0% | 59.2% | 17.2% | 1,213 | $0.551 | \n| 9 | [claude-haiku-5.5@low](https://thegustafson.com/benchmark/claude-haiku-5-5-low) | Anthropic · Flash | 1255 | 5–11 | 3–1 | 0–4 | 0–4 | 38.3% | 56.9% | 21.3% | 995 | $0.010 | \n| 10 | [claude-haiku-5.5@high](https://thegustafson.com/benchmark/claude-haiku-5-5-high) | Anthropic · Flash | 1223 | 16–20 | 2–2 | 0–4 | 0–4 | 38.0% | 57.6% | 21.9% | 2,559 | $0.019 | \n| 11 | [deepseek-v4.1-flash@high](https://thegustafson.com/benchmark/deepseek-v4-1-flash-high) | DeepSeek · Flash | 1211 | 18–18 | 2–2 | 0–4 | 0–4 | 35.1% | 62.4% | 16.6% | 6,143 | $0.043 | \n| 12 | [az-puct-1](https://thegustafson.com/benchmark/engine-az-puct-1) | Engine | 1200 | 12–45 | · | · | · | 54.2% | 91.7% | 12.5% | 0 | $0.00 | \n| 13 | [gpt-6-luna@high](https://thegustafson.com/benchmark/gpt-6-luna-high) | OpenAI · Flash | 1158 | 13–23 | 2–2 | 0–4 | 0–4 | 36.5% | 65.2% | 17.0% | 2,845 | $0.021 | \n| 14 | [deepseek-v4-pro@high](https://thegustafson.com/benchmark/deepseek-v4-pro-high) | DeepSeek · Flag | 1148 | 11–21 | 3–1 | 0–4 | 0–4 | 38.7% | 63.7% | 15.7% | 4,486 | $0.321 | \n| 15 | [random-uct-50](https://thegustafson.com/benchmark/engine-random-uct-50) | Engine | 1135 | 46–83 | · | · | · | 29.0% | 57.0% | 20.7% | 0 | $0.00 | \n| 16 | [gemini-3.8-flash@low](https://thegustafson.com/benchmark/gemini-3-8-flash-low) | Google · Flash | 1135 | 3–13 | 3–1 | 0–4 | 0–4 | 32.2% | 59.9% | 14.8% | 367 | $0.037 | \n| 17 | [claude-opus-5.5@low](https://thegustafson.com/benchmark/claude-opus-5-5-low) | Anthropic · Flag | 1085 | 4–16 | 2–2 | 0–4 | 0–4 | 31.8% | 53.6% | 22.2% | 550 | $0.312 | \n| 18 | [qwen3.8-flash@high](https://thegustafson.com/benchmark/qwen3-8-flash-high) | Qwen · Flash | 1036 | 11–25 | 1–3 | 0–4 | 0–4 | 28.1% | 61.5% | 19.7% | 1,453 | $0.013 | \n| 19 | [qwen3.8-max@high](https://thegustafson.com/benchmark/qwen3-8-max-high) | Qwen · Flag | 978 | 6–26 | 1–3 | 0–4 | 0–4 | 31.6% | 59.9% | 18.8% | 1,513 | $0.183 | \n| 20 | [gpt-6-luna@low](https://thegustafson.com/benchmark/gpt-6-luna-low) | OpenAI · Flash | 970 | 2–14 | 0–4 | 0–4 | 0–4 | 29.5% | 60.7% | 17.6% | 735 | $0.008 | \n| 21 | [qwen3.8-flash@none](https://thegustafson.com/benchmark/qwen3-8-flash-none) | Qwen · Flash | 909 | 2–14 | 0–4 | 0–4 | 0–4 | 21.9% | 45.7% | 23.7% | 0 | $0.008 | \n| 22 | [deepseek-v4-pro@none](https://thegustafson.com/benchmark/deepseek-v4-pro-none) | DeepSeek · Flag | 899 | 2–18 | 1–3 | 0–4 | 0–4 | 20.4% | 48.2% | 21.2% | 0 | $0.021 | \n| 23 | [qwen3.8-max@none](https://thegustafson.com/benchmark/qwen3-8-max-none) | Qwen · Flag | 866 | 3–17 | 1–3 | 0–4 | 0–4 | 30.1% | 57.1% | 16.3% | 1,424 | $0.172 | \n| 24 | [deepseek-v4.1-flash@none](https://thegustafson.com/benchmark/deepseek-v4-1-flash-none) | DeepSeek · Flash | 865 | 1–15 | 1–3 | 0–4 | 0–4 | 17.1% | 51.7% | 19.3% | 0 | $0.003 | \n\n`az-puct-1 = 1200.0 Elo` (the vertical amber line on the Elo bar). Against `az-puct-4000` (4,000 MCTS simulations per move, ~279 ms), all 20 OpenRouter LLM configurations went a combined 0–80. Against `az-puct-400` (~26 ms/move), `gemini-4-argon@high` scored the sole victory (1–79 overall).\n## 2. Where frontier LLMs sit on the AlphaZero search ladder\n\nMatching the 20 OpenRouter configurations against the [Monte Carlo Tree Search](https://thegustafson.com/blog/mcts-is-awesome) and [AlphaZero](https://thegustafson.com/blog/crosshair-alphazero) baselines in Batch 1 (`240 games`) shows where scratchpad reasoning sits relative to explicit tree search:\n\n- **Top reasoning LLMs reliably beat `random-uct-50` (`1135 Elo`) and outscore the zero-search policy net (` az-puct-1`, `1200 Elo`):**[` gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high) (` 1440 Elo` ),[`gpt-6.1-sol@low`](https://thegustafson.com/benchmark/gpt-6-1-sol-low) (` 1408 Elo` ), and[`gpt-6.1-sol@high`](https://thegustafson.com/benchmark/gpt-6-1-sol-high) (` 1398 Elo` ) all swept`random-uct-50`` 4–0` . Nine of the twenty LLM configurations rated above`az-puct-1` (`1200 Elo` ), and eleven rated above`random-uct-50` (`1135 Elo` ). When the neural net isn't allowed to search ahead, a frontier model thinking out loud in its scratchpad can genuinely outplay it.\n- **Turning on 400 MCTS simulations (`az-puct-400`, `1888 Elo`) flips the score to `79–1`:** Across 80 games in Batch 1b, 19 of the 20 LLM configurations were swept`0–4` . Only**[`gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high)** won a game (` 1–3` ), converting a 38-ply Blue victory when`az-puct-400` overpushed at`f7` on ply 32.\n- **At 4,000 simulations (`az-puct-4000`, `2343 Elo`), the 114K network went `80–0`:** Not a single LLM configuration won a game in Batch 1c. Even when`gpt-6.1-sol@high` spent 70 seconds and`3,060` reasoning tokens per move (`$0.49` per game),`az-puct-4000` (`0.28 seconds` per move,`$0.00` ) locked the board by ply 16. Turns out fifty cents and a full minute of prose per move is still no match for a quarter-second of real tree search.\n\n## 3. Thinking budget beats model size (`16–4`)\n\nIn Batch 2 (`60 games`), we tested how much of a model's playing strength comes from its base parameter count versus how long it is allowed to think on each turn:\n\n1. **Same-model thinking ablation (`40 games`):** Each of the 10 models played a 4-game match between its`@high` reasoning configuration and its`@low` (or`@none` ) configuration.`@high` won**`25–15` (`62.5%`)** and gained an average of**`+149 Elo`** across the 10 pairs.\n2. **Cross-tier showdown (`20 games`):** For each of the five labs, we matched the smaller**Flash `@high`** model directly against the same lab's**Flagship `@low`/`@none`** model.\n\nThe smaller Flash models with `@high` thinking beat their own labs' low-effort Flagships **`16–4` (`80.0%`)**, usually at a fraction of the cost per game:\n\n- **Anthropic (`4–0`):**[` claude-haiku-5.5@high`](https://thegustafson.com/benchmark/claude-haiku-5-5-high) (` 1223 Elo` ,`2,559` reasoning tokens/turn,`$0.019/game` ) swept[`claude-opus-5.5@low`](https://thegustafson.com/benchmark/claude-opus-5-5-low) (` 1085 Elo` ,`550` reasoning tokens/turn,`$0.312/game` )`4–0` at**16× lower cost per game** .\n- **DeepSeek (`4–0`):**[` deepseek-v4.1-flash@high`](https://thegustafson.com/benchmark/deepseek-v4-1-flash-high) (` 1211 Elo` ,`$0.043/game` ) swept[`deepseek-v4-pro@none`](https://thegustafson.com/benchmark/deepseek-v4-pro-none) (` 899 Elo` ,`$0.021/game` )`4–0` .\n- **Qwen (`4–0`):**[` qwen3.8-flash@high`](https://thegustafson.com/benchmark/qwen3-8-flash-high) (` 1036 Elo` ,`$0.013/game` ) swept[`qwen3.8-max@none`](https://thegustafson.com/benchmark/qwen3-8-max-none) (` 866 Elo` ,`$0.172/game` )`4–0` at**13× lower cost** .\n- **Google (`2–2`) & OpenAI (` 2–2`):**[` gemini-3.8-flash@high`](https://thegustafson.com/benchmark/gemini-3-8-flash-high) split` 2–2` with[`gemini-4-argon@low`](https://thegustafson.com/benchmark/gemini-4-argon-low) , and[` gpt-6-luna@high`](https://thegustafson.com/benchmark/gpt-6-luna-high) split` 2–2` with[`gpt-6.1-sol@low`](https://thegustafson.com/benchmark/gpt-6-1-sol-low) (` 6.6×` cheaper). Not a single low-effort Flagship won its mini-match against its own lab's smaller Flash sibling. When you cannot coast on memorized opening patterns, having tokens to actually trace moves on the board matters much more than model size.\n\n| Lab | @high Entrant | @low / @none Entrant | Direct H2H | @high Elo | @low Elo | Δ Elo | Think Tokens/Turn | \n|---|---|---|---|---|---|---|---|\n| Google (Flagship) | gemini-4-argon@high | gemini-4-argon@low | 3–1 | 1440 | 1290 | +150 | 2,152 vs 0 | \n| Google (Flash) | gemini-3.8-flash@high | gemini-3.8-flash@low | 4–0 | 1328 | 1135 | +193 | 2,649 vs 367 | \n| OpenAI (Flagship) | gpt-6.1-sol@high | gpt-6.1-sol@low | 1–3 | 1398 | 1408 | -11 | 3,060 vs 588 | \n| OpenAI (Flash) | gpt-6-luna@high | gpt-6-luna@low | 2–2 | 1158 | 970 | +188 | 2,845 vs 735 | \n| Anthropic (Flagship) | claude-opus-5.5@high | claude-opus-5.5@low | 2–2 | 1259 | 1085 | +174 | 1,213 vs 550 | \n| Anthropic (Flash) | claude-haiku-5.5@high | claude-haiku-5.5@low | 2–2 | 1223 | 1255 | -33 | 2,559 vs 995 | \n| DeepSeek (Flagship) | deepseek-v4-pro@high | deepseek-v4-pro@none | 3–1 | 1148 | 899 | +248 | 4,486 vs 0 | \n| DeepSeek (Flash) | deepseek-v4.1-flash@high | deepseek-v4.1-flash@none | 4–0 | 1211 | 865 | +347 | 6,143 vs 0 | \n| Qwen (Flagship) | qwen3.8-max@high | qwen3.8-max@none | 2–2 | 978 | 866 | +112 | 1,513 vs 1,424 | \n| Qwen (Flash) | qwen3.8-flash@high | qwen3.8-flash@none | 2–2 | 1036 | 909 | +127 | 1,453 vs 0 | \n\n## 4. Inside the Flagship and Flash 5×5 round-robins\n\nTo get a direct head-to-head ordering among the five labs, we ran two 5×5 round-robin tournaments (`40 games` each, 4 games per pair with swapped colors and openings):\n\n### Batch 3: Flagship `@high` round-robin\n\n- **[`gpt-6.1-sol@high`](https://thegustafson.com/benchmark/gpt-6-1-sol-high) (` 12–4`, `75.0%`)** finished first inside the round-robin by sweeping` claude-opus-5.5@high` (`4–0` ) and`qwen3.8-max@high` (`4–0` ), while splitting`2–2` with`gemini-4-argon@high` and`deepseek-v4-pro@high` .\n- **[`gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high) (` 11–5`, `68.8%`)** finished one game back in the round-robin (and`#1` overall among LLMs at`1440 Elo` thanks to its`1–3` win over`az-puct-400` and`3–1` win over`@low` ), sweeping`deepseek-v4-pro@high` (`4–0` ) and`qwen3.8-max@high` (`4–0` ).\n- **Opus had Argon's number (`3–1`):** Even though` gpt-6.1-sol@high` swept[`claude-opus-5.5@high`](https://thegustafson.com/benchmark/claude-opus-5-5-high)` 4–0` , Opus (`9–7` overall) turned around and beat`gemini-4-argon@high`` 3–1` .\n\n### Batch 4: Flash `@high` round-robin\n\n- **[`gemini-3.8-flash@high`](https://thegustafson.com/benchmark/gemini-3-8-flash-high) (` 13–3`, `81.3%`, `1328 Elo`)** ran away with the Flash bracket, sweeping` gpt-6-luna@high` (`4–0` ) and`deepseek-v4.1-flash@high` (`4–0` ), beating`claude-haiku-5.5@high` (`3–1` ), and splitting`2–2` with`qwen3.8-flash@high` .\n- `gemini-3.8-flash@high` also recorded the highest 400-simulation AlphaZero Top-1 move agreement of any OpenRouter model in the entire tournament (**`46.3%`** across 512 turns), finishing ahead of three of the five Flagship`@high` models on the overall Elo leaderboard at`$0.166` per game. (Apparently Flash did not get the memo that it was supposed to stay in the budget bracket.)\n\n## 5. Walking through the most interesting games in the tournament\n\nEvery game log records the active player's chain-of-thought alongside the shadow 400-simulation AlphaZero evaluation before and after every move. You can step through six featured moments in the interactive board below (four from the OpenRouter tournament listed here, plus two from the prompt-scaffolding pilot in Section 6), or visit any model's [dedicated benchmark page](https://thegustafson.com/benchmark) to browse all of its games:\n\n1. **Moment 1 (`b01_gemini4argon_high_vs_az400_g02`, Ply 33):** The sole LLM win out of 80 games in Batch 1b against 400-simulation AlphaZero. Playing Blue in`raw` mode,[`gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high) survives the middlegame and spots that Red's push to`f7` on ply 32 leaves Row 6 and Column`a` exposed. Argon plays`f6` on ply 33 (`77.8% -> 89.0%` , AZ-400 search rank #4), then finds AZ-400's #1 move on ply 35 (`a6` ) and ply 37 (`a1` ) to convert in 38 plies.\n2. **Moment 2 (`b02_opus55_low_vs_haiku55_high_g01`, Ply 28):** Why thinking budget beats parameter count.[` claude-opus-5.5@low`](https://thegustafson.com/benchmark/claude-opus-5-5-low) (` 550` reasoning tokens/turn) holds an`81.1%` winning position as Red against[`claude-haiku-5.5@high`](https://thegustafson.com/benchmark/claude-haiku-5-5-high) (` 2,559` reasoning tokens/turn). Opus plays`b7` to claim a bottom-edge stone without checking what activating Column`b` gives Blue; Haiku immediately seizes the`a` /`b` file and wins in 32 plies.\n3. **Moment 3 (`b03_openai_vs_google_g01`, Ply 24):** Flagship clash between[` gpt-6.1-sol@high`](https://thegustafson.com/benchmark/gpt-6-1-sol-high) (Red) and[` gemini-4-argon@high`](https://thegustafson.com/benchmark/gemini-4-argon-high) (Blue). Sol builds two parallel vertical columns down`b` and`d` , setting up a clean fork where both`c4` and`c5` finish Red's top-to-bottom connection.\n4. **Moment 4 (`b01_gpt61sol_high_vs_az4000_g01`, Ply 15):** Why 4,000-simulation AlphaZero ([` az-puct-4000`](https://thegustafson.com/benchmark/engine-az-puct-4000) ) went` 80–0` . By ply 15 (`b5` ), the 114K network has built an unstoppable left-to-right bridge. On ply 16,`gpt-6.1-sol@high` spends 70 seconds calculating every branch before writing:*\"Every legal move permits an immediate Blue win. I choose b4 as the direct blocking attempt, although it cannot save the position.\"* At least it knew it was cooked.\n\n## 6. Why LLMs blunder: board parsing vs. lookahead depth\n\nWhen an LLM throws away a winning endgame, is it misreading the ASCII board, or is it just not looking enough moves ahead?\n\nIn our pilot study with Antigravity `pro` and `flash`, we tested that directly using three levels of prompt help:\n\n1. **Raw Board + Legal Moves (`raw`):** Exact rules, 7×7 ASCII grid (` a1` through`g7` ), stone lists for Red and Blue, opponent's last move, and the explicit list of legal moves.\n2. **Tactical Previews (`assisted`):** On top of` raw` , the prompt includes the engine's 1-ply hover calculation for every legal move: which enemy stones that move immediately flips (including cascades), whether it wins immediately, and which row/column (or wildcard) it forces the opponent into next.\n3. **Strategic Primer + Previews (`strategic`):** On top of` assisted` , we add four short bullets of basic tactical advice: (1) diagonals do not connect roads, (2) do not play right next to an enemy stone along an open line where their next crosshair move can sandwich you, (3) watch out for the 4th corner of a 2×2 diagonal crosscut, and (4) route the opponent into full or dead lines.\n\nComparing the three prompting modes shows two clear limits:\n\n### Giving the model 1-ply previews still leaves 2-ply traps wide open\n\nIn `flash_assisted_az1_R` (Moment 6 in the replay explorer above), `flash` held an **83.1% winning position** on ply 32 with full 1-ply tactical previews turned on. It chose `g2`, writing: *\"Connect g2 to g1 to establish a two-stone top-edge group and create a winning threat via f2.\"*\n\nThe 1-ply preview accurately told `flash` that `g2` flipped zero stones and sent Blue to Row 2 or Column g. What a 1-ply preview does not spell out is what happens one move later: playing in Row 2 gives Blue permission to play in Row 2 *before* Red gets another turn, so Blue simply plays `f2` and wins on the spot (`83.1% -> 0.0%`). Handing `flash` the 4-sentence `strategic` primer fixed that habit and dropped its blunders to zero against `az-puct-1`.\n\n### Three plies of scratchpad math vs. five plies of tree search\n\nAcross all three prompt modes, `pro` went `4–0` with zero blunders against the zero-search policy net (`az-puct-1`), and with full `strategic` scaffolding ([`pro (strategic)`](https://thegustafson.com/benchmark/pilot-pro-strategic), `1815 Anchored Elo`) it split **` 1–1`** against 400-simulation AlphaZero (` az-puct-400`).\n\nThose two games against `az-puct-400` show right where scratchpad lookahead runs out of gas:\n\n- **When the trap is 2–3 plies deep, `pro` catches it (`pro_strategic_az400_R`, ply 24):** On ply 23,` az-puct-400` played a sacrifice bait at`b5` , inviting Red to sandwich it at`b6` . Looking two plies ahead,`pro` saw that playing`b6` would send Blue to`a6` for a 2×2 diagonal crosscut on`a7` , declined the bait with`b2` , and won in 45 plies.\n- **When the refutation is 5 plies deep, both the LLM and the raw policy net miss it (`pro_strategic_az400_B`, ply 37):** Playing Blue,` pro` outplayed`az-puct-400` for 36 plies to reach a**97.5% winning position** . Looking for a clean finish,`pro` worked through three branches in its scratchpad and played`a1` (which was also the #1 pick of the raw 1-sim policy prior at`P = 46.7%` ). Its calculation on the`f1` branch stopped 3 plies deep, missing that`az-puct-400` could ignore the right edge after`f1 -> f6` and play`a6 -> a5` , horizontally sandwiching`b5` against`c5` to connect all seven columns (`97.5% -> 10.6%` ,`-86.9%` ).\n\n| Model | Prompt Mode | Engine Opponent | W–L | LLM Turns | AZ Top-1 | AZ Top-3 | Avg ΔWin | Blunders | \n|---|---|---|---|---|---|---|---|---|\n| pro | raw | az-puct-1 | 1–0 | 17 | 47.1% | 76.5% | -0.8% | 0 | \n| pro | assisted | az-puct-1 | 2–0 | 36 | 52.8% | 75.0% | -1.5% | 0 | \n| pro | strategic | az-puct-1 | 1–0 | 18 | 33.3% | 66.7% | -1.5% | 0 | \n| pro | raw | az-puct-400 | 0–2 | 45 | 28.9% | 71.1% | -5.8% | 5 | \n| pro | strategic | az-puct-400 | 1–1 | 44 | 45.5% | 79.5% | -4.1% | 4 | \n| flash | raw | az-puct-1 | 1–1 | 36 | 30.6% | 66.7% | -7.0% | 8 | \n| flash | assisted | az-puct-1 | 0–1 | 17 | 41.2% | 52.9% | -7.4% | 2 | \n| flash | strategic | az-puct-1 | 1–0 | 13 | 46.2% | 53.8% | -1.5% | 0 | \n| flash | strategic | random-uct-50 | 1–0 | 16 | 43.8% | 81.3% | -3.0% | 1 | \n| flash | raw | az-puct-400 | 1–1 | 45 | 40.0% | 82.2% | -3.8% | 4 | \n\n`pro` the 4-rule strategic primer and 1-ply previews lifts its performance from 0–2 in `raw` mode against `az-puct-400` to a 1–1 split (1815 Anchored Elo).\n## 7. Takeaways and running Crosshair-Bench yourself\n\nTesting language models on a board game that did not exist during training strips away memorized openings and leaves pure in-context calculation:\n\n1. **Scratchpad calculation beats 1-ply intuition, but loses to real tree search:** Frontier reasoning models reliably beat 50-rollout MCTS (`random-uct-50` ,`1135 Elo` ) and the zero-search 114K policy network (`az-puct-1` ,`1200 Elo` ), but go`1–79` against 400-simulation AlphaZero (`26 ms/move` ) and`0–80` against 4,000 simulations (`279 ms/move` ).\n2. **Test-time reasoning tokens beat parameter count:** Across all five labs, smaller Flash models running with`@high` thinking beat their own labs' Flagship models running with`@low` or disabled thinking`16–4` , often at 6× to 16× lower cost per game.\n3. **Explore the games or run the suite:** You can inspect every participant's head-to-head stats and step through every game on the**[Crosshair Benchmark page](https://thegustafson.com/benchmark)** , play against the Rust WASM engines yourself at**[`/games/crosshair`](https://thegustafson.com/games/crosshair)** , or run the open-source match harness and 100-position single-turn tactical benchmark (`Crosshair-Bench-100` ) from**[`TheGustafson/crosshair-bench`](https://github.com/TheGustafson/crosshair-bench)** .", "url": "https://wpnews.pro/news/crosshair-bench-evaluating-frontier-models-on-an-unseen-game", "canonical_source": "https://thegustafson.com/blog/llms-vs-crosshair", "published_at": "2026-10-28 13:45:24.213979+00:00", "updated_at": "2026-10-28 13:45:27.018747+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence", "machine-learning"], "entities": ["Crosshair-Bench", "Crosshair", "TheGustafson", "OpenRouter", "az-puct-400", "gpt-6.1-sol", "gemini-4-argon", "claude-opus-5.5"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/crosshair-bench-evaluating-frontier-models-on-an-unseen-game", "markdown": "https://wpnews.pro/news/crosshair-bench-evaluating-frontier-models-on-an-unseen-game.md", "text": "https://wpnews.pro/news/crosshair-bench-evaluating-frontier-models-on-an-unseen-game.txt", "jsonld": "https://wpnews.pro/news/crosshair-bench-evaluating-frontier-models-on-an-unseen-game.jsonld"}}