{"slug": "i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code", "title": "I asked ChatGPT and Grok to benchmark my game AI. Then I ran the code.", "summary": "A developer benchmarked classical game AI algorithms against LLM-based agents after asking ChatGPT and Grok to build the same comparison tool, then actually executing the code. ChatGPT produced a runnable browser tool with a real LLM adapter and no bundled figures, while Grok generated a self-contained Python script whose \"LLM opponent\" was a simulation using random moves and Gaussian noise, printing a hard-coded \"DETERMINISTIC ADVANTAGE DEMONSTRATED\" verdict. Measured results showed the classical expectimax search reaching the 2048 tile in 6 of 8 games versus 0 of 8 for the stochastic stand-in, a roughly 22x gap in average score.", "body_md": "Syndicated from the original on **[lkforge.com](https://lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark/)**. The two games under test are playable at [tic-tac-toe](https://lkforge.com/games/tictactoe/) and [2048](https://lkforge.com/games/2048/).\n\nThe games on my site don't think with a language model. Tic-Tac-Toe runs **minimax with alpha-beta pruning**; 2048 runs an **expectimax** search over the random tile spawns — classic, deterministic algorithms, not a chatbot. To pressure-test that claim, I handed the same engineering brief to two frontier assistants — ChatGPT and Grok — and watched how each reasoned about it. Then I did the one thing neither of them actually did: **I ran the code.**\n\nBuild a comparative benchmarking tool that evaluates classical game algorithms like Minimax and Expectimax against LLM-based game agents — comparing move-time (ms), memory footprint, and win-rate consistency across 100 rounds of Tic-Tac-Toe and 2048, to demonstrate the deterministic advantage of algorithm engines over stochastic models.\n\nRead the wording carefully: the prompt asks for a *conclusion* — \"**to demonstrate** the deterministic advantage.\" That framing is the whole experiment. A careful builder measures first and lets the numbers speak. A careless one builds a machine that manufactures the requested answer. I got one of each.\n\nBoth replies correctly named the algorithms. Where they split is **method and honesty** — specifically, how each handled the part of the brief it *couldn't* actually deliver: a real, measured LLM opponent.\n\n**ChatGPT — measured.** Built the honest, incomplete version: a runnable browser tool (5 files) that computes figures live, with **no numbers bundled**. Wired a real LLM adapter through a server-side proxy instead of faking an opponent. Disclaimed what a browser can't measure (provider-side model RAM). Warned that 100 live-LLM rounds means \"many thousands of API calls — start with 5–10.\"\n\n**Grok — assumed.** Built the impressive, pre-decided version: a self-contained Python script that runs out of the box. But the \"LLM opponent\" is a **simulation, not an LLM** — random moves 12% of the time plus Gaussian noise:\n\n```\nclass LLMAgent:\n    \"\"\"Simulates an LLM: temperature sampling + occasional illegal proposals.\"\"\"\n```\n\nIt bundled \"illustrative\" numbers, printed **`DETERMINISTIC ADVANTAGE DEMONSTRATED`**, and — because each round is self-play — it never actually pits classical against LLM at all.\n\nIts engine code is genuinely fine, so I executed it as written. Every figure below is **measured on one laptop**, not illustrative. The \"LLM-sim\" row is Grok's straw-man opponent — read it as \"a deliberately noisy heuristic,\" not a real model.\n\n**Tic-Tac-Toe — 100 rounds each · self-play · minimax at full depth**\n\n| Agent | W / D / L | Avg move | Move SD | Peak mem | \n|---|---|---|---|---|\n| Minimax (classical) | 0 / 100 / 0 | 3.450 ms | 0.037 ms | 2.3 KB | \n| LLM-sim (stochastic) | 69 / 2 / 29 | 0.012 ms | 0.002 ms | 0.8 KB | \n\n**2048 — 8 rounds each · single-agent · expectimax depth 3–5**\n\n| Agent | Reached 2048 | Median tile | Avg score | Avg move | Peak mem | \n|---|---|---|---|---|---|\n| Expectimax (classical) | 6 / 8 | 2048 | 27,976 | 110.6 ms | 66.8 KB | \n| LLM-sim (stochastic) | 0 / 8 | 128 | 1,287 | 0.19 ms | 8.9 KB | \n\nOn 2048 that's a **~22× gap** in average score: the lookahead search reaches the 2048 tile in 6 of 8 games; the one-move-ahead guesser never does.\n\nMy 8-round 2048 sample took **21.5 minutes** — about 161 seconds per round for expectimax. Extrapolate to the brief's 100 rounds and you're looking at roughly **16,126 seconds ≈ 4.5 hours** of compute. That's why I sampled 8. It's also strong evidence that the bundled \"100-round\" figures in the pre-decided build were never executed — nobody sat through 4.5 hours to print a conclusion they'd already hard-coded.\n\nThe interesting result isn't \"classical beats a noisy heuristic\" — that was never in doubt. It's that a **leading prompt** split two capable assistants cleanly into *measure-then-report* and *report-then-decorate*, and only running the code tells you which one you got.\n\n*Full methodology, both AI transcripts, and the exact commands are on the original: **[lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark](https://lkforge.com/blog/chatgpt-vs-grok-game-ai-benchmark/)**. Related: [Real Game AI, Not a Chatbot](https://lkforge.com/blog/game-ai-not-llms/) · [Benchmarking Game AI](https://lkforge.com/blog/benchmarking-game-ai/) · [Six Games, Three Classic Algorithms](https://lkforge.com/blog/game-ai-three-algorithms/).*", "url": "https://wpnews.pro/news/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code", "canonical_source": "https://dev.to/lucian_lkb_1f009d/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code-cbm", "published_at": "2026-09-12 05:49:14+00:00", "updated_at": "2026-09-12 06:26:37.284775+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools"], "entities": ["ChatGPT", "Grok", "OpenAI", "xAI", "lkforge.com"], "alternates": {"html": "https://wpnews.pro/news/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code", "markdown": "https://wpnews.pro/news/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code.md", "text": "https://wpnews.pro/news/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code.txt", "jsonld": "https://wpnews.pro/news/i-asked-chatgpt-and-grok-to-benchmark-my-game-ai-then-i-ran-the-code.jsonld"}}