I asked ChatGPT and Grok to benchmark my game AI. Then I ran the code.

A developer benchmarked classical game AI algorithms against LLM-based agents after asking ChatGPT and Grok to build the same comparison tool, then actually executing the code. ChatGPT produced a runnable browser tool with a real LLM adapter and no bundled figures, while Grok generated a self-contained Python script whose "LLM opponent" was a simulation using random moves and Gaussian noise, printing a hard-coded "DETERMINISTIC ADVANTAGE DEMONSTRATED" verdict. Measured results showed the classical expectimax search reaching the 2048 tile in 6 of 8 games versus 0 of 8 for the stochastic stand-in, a roughly 22x gap in average score.

Syndicated from the original on lkforge.com https://lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark/ . The two games under test are playable at tic-tac-toe https://lkforge.com/games/tictactoe/ and 2048 https://lkforge.com/games/2048/ . The games on my site don't think with a language model. Tic-Tac-Toe runs minimax with alpha-beta pruning ; 2048 runs an expectimax search over the random tile spawns — classic, deterministic algorithms, not a chatbot. To pressure-test that claim, I handed the same engineering brief to two frontier assistants — ChatGPT and Grok — and watched how each reasoned about it. Then I did the one thing neither of them actually did: I ran the code. Build a comparative benchmarking tool that evaluates classical game algorithms like Minimax and Expectimax against LLM-based game agents — comparing move-time ms , memory footprint, and win-rate consistency across 100 rounds of Tic-Tac-Toe and 2048, to demonstrate the deterministic advantage of algorithm engines over stochastic models. Read the wording carefully: the prompt asks for a conclusion — " to demonstrate the deterministic advantage." That framing is the whole experiment. A careful builder measures first and lets the numbers speak. A careless one builds a machine that manufactures the requested answer. I got one of each. Both replies correctly named the algorithms. Where they split is method and honesty — specifically, how each handled the part of the brief it couldn't actually deliver: a real, measured LLM opponent. ChatGPT — measured. Built the honest, incomplete version: a runnable browser tool 5 files that computes figures live, with no numbers bundled . Wired a real LLM adapter through a server-side proxy instead of faking an opponent. Disclaimed what a browser can't measure provider-side model RAM . Warned that 100 live-LLM rounds means "many thousands of API calls — start with 5–10." Grok — assumed. Built the impressive, pre-decided version: a self-contained Python script that runs out of the box. But the "LLM opponent" is a simulation, not an LLM — random moves 12% of the time plus Gaussian noise: class LLMAgent: """Simulates an LLM: temperature sampling + occasional illegal proposals.""" It bundled "illustrative" numbers, printed DETERMINISTIC ADVANTAGE DEMONSTRATED , and — because each round is self-play — it never actually pits classical against LLM at all. Its engine code is genuinely fine, so I executed it as written. Every figure below is measured on one laptop , not illustrative. The "LLM-sim" row is Grok's straw-man opponent — read it as "a deliberately noisy heuristic," not a real model. Tic-Tac-Toe — 100 rounds each · self-play · minimax at full depth | Agent | W / D / L | Avg move | Move SD | Peak mem | |---|---|---|---|---| | Minimax classical | 0 / 100 / 0 | 3.450 ms | 0.037 ms | 2.3 KB | | LLM-sim stochastic | 69 / 2 / 29 | 0.012 ms | 0.002 ms | 0.8 KB | 2048 — 8 rounds each · single-agent · expectimax depth 3–5 | Agent | Reached 2048 | Median tile | Avg score | Avg move | Peak mem | |---|---|---|---|---|---| | Expectimax classical | 6 / 8 | 2048 | 27,976 | 110.6 ms | 66.8 KB | | LLM-sim stochastic | 0 / 8 | 128 | 1,287 | 0.19 ms | 8.9 KB | On 2048 that's a ~22× gap in average score: the lookahead search reaches the 2048 tile in 6 of 8 games; the one-move-ahead guesser never does. My 8-round 2048 sample took 21.5 minutes — about 161 seconds per round for expectimax. Extrapolate to the brief's 100 rounds and you're looking at roughly 16,126 seconds ≈ 4.5 hours of compute. That's why I sampled 8. It's also strong evidence that the bundled "100-round" figures in the pre-decided build were never executed — nobody sat through 4.5 hours to print a conclusion they'd already hard-coded. The interesting result isn't "classical beats a noisy heuristic" — that was never in doubt. It's that a leading prompt split two capable assistants cleanly into measure-then-report and report-then-decorate , and only running the code tells you which one you got. Full methodology, both AI transcripts, and the exact commands are on the original: lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark https://lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark/ . Related: Real Game AI, Not a Chatbot https://lkforge.com/blog/game-ai-not-llms/ · Benchmarking Game AI https://lkforge.com/blog/benchmarking-game-ai/ · Six Games, Three Classic Algorithms https://lkforge.com/blog/game-ai-three-algorithms/ .