{"slug": "baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost", "title": "Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost?", "summary": "Fable 5 and GPT-5.6 Sol solved nearly all levels of the first two stages of the puzzle game Baba Is You, but took significantly longer than humans and cost over $2,000 in experiments. The AI models' performance highlights the gap between capability and cost-effectiveness in software development tasks.", "body_md": "# Baba Is Solved by Fable 5 and GPT-5.6 Sol, but at what cost?\n\nWe tested GPT-5.6 Sol and Claude Fable 5, among other models, on the first two stages of Baba Is You. While they are able to solve almost all levels, it took them significantly longer than humans.\n\nWhile AI might [overtake humans in tasks](https://x.com/FakePsyho/status/2075128093891801305), it is not always cost-effective.\nThat is the opposite of everyday software development, where agents tend to overestimate their own effort — see [I am sorry Dave, it will take 3 days](https://www.linkedin.com/pulse/all-agents-useless-estimating-own-work-michal-przadka-vox5e/).\n\nThere were surprises — for example, Gemini 3.5 Flash was 2.4x more expensive than Fable 5 to solve The Intro stage. Or that a more “budget” GPT-5.6 Terra was actually 2.9x more costly than Sol.\n\nOverall, we burned well over $2k on experiments and runs (not including our subscriptions), so you don’t have to.\n\n## The game\n\n[Baba Is You](https://en.wikipedia.org/wiki/Baba_Is_You) is a lovely indie puzzle game where you rewrite the rules by pushing text blocks around. `[WALL] [IS] [STOP]`\n\nmeans walls block you — break the sentence, and you walk right through. It’s the childhood game *the floor is lava*, made literal.\n\nYou are a cute bunny. Like [another game with a cute bunny](https://en.wikipedia.org/wiki/Ori_and_the_Blind_Forest), this cuteness is deceiving. A few levels into the game and your `[BRAIN] [IS] [MELT]`\n\n. I highly recommend it.\n\n## Prior art\n\nThis game has already attracted a lot of attention, e.g. with the [Baba Is AI](https://arxiv.org/abs/2407.13729) paper from 2024.\nA year ago, in [Baba Is Eval](https://fi-le.net/baba/), Sonnet 4 could only finish its first, introductory level. This May, in [Baba Is Agent](https://meffmadd.github.io/samplesurium/posts/baba_is_agent/), some models were able to beat the first stage consisting of 8 levels, with Gemini 3.1 Pro and GPT-5.5 in the lead.\nSimilar geometrical puzzles are at the center of the [ARC-AGI-3](https://arcprize.org/arc-agi/3) benchmark, aimed at measuring abstract reasoning and intelligence of LLMs.\n\nNow there are a few more models, including the famed (and feared) Claude Fable 5, GPT-5.5 Pro solving open mathematical problems, but also open-weight GLM-5.2 and fast-and-cheap Gemini 3.5 Flash.\n\n## Making it a benchmark\n\nPrevious projects either re-made parts of the game, or had an AI again play it on Steam.\nWe did it differently — in our open-source [Baba Is Harbor](https://github.com/stared/baba-is-harbor) repository, we extracted both levels and game logic into the [Harbor](https://www.harborframework.com/) agent evaluation framework.\nWe did it with the help of Claude Code, and it was surprisingly easy — both to decode levels and to extract actual game logic from Lua scripts.\n\nThat way we can run it like any other [benchmark](https://quesma.com/benchmarks/), easily with various models and harnesses. We provided proper tools to allow the same actions a player could use, while providing a text interface:\n\n```\nYou play the level interactively with the `baba` command. Your moves accumulate\ninto a running game (state persists between calls):\n\n    baba rrd          apply moves r, r, d; show what CHANGED (object diff)\n    baba undo 2       undo the last 2 moves (default 1); show what changed\n    baba restart      reset to the start of the level; show the full board\n    baba show         reprint the full current state without moving\n```\n\nIt took a few iterations to polish tools and prompt, so that the focus is on the puzzles themselves, rather than deciphering data formats or other repetitive work. To run it, we just need to\n\n```\nharbor run -p tasks/ -i \"stage-00-level-0[0-7]\" -a terminus-2 \\ \n  -m openrouter/openai/gpt-5.6-luna \\ \n  -m openrouter/z-ai/glm-5.2 \\ \n  -k 3 -n 18 -y\n```\n\nWe initially ran it with Terminus-2 harness (to use the same one for all models), with 3 attempts per model per level.\n\n## Results for Stage 0: The Intro\n\nHere is an example of how an AI agent solves a level (apologies to Hempuli, the game’s author, for such a simplified widget). Even the same model can arrive at different solutions across runs.\n\nIn the case of level `01`\n\n*Where do I go?*, Baba is encircled by a wall. But blocks `[WALL] [IS] [STOP]`\n\ncan be dismantled, allowing us to pass through the wall. From there, two solutions exist: compose `[FLAG] [IS] [WIN]`\n\nand reach the flag, or think outside the box again and compose `[WALL] [IS] [WIN]`\n\n.\nExplore other levels if you’re fine with getting spoilers.\n\n### Solving The Intro\n\nThe first set of levels is an introduction. While the first level requires just a few moves in the same direction, subsequent ones require learning some of the game’s mechanics.\n\n| Model | 00 baba is you | 01 where do i go? | 02 now what is this? | 03 out of reach | 04 still out of reach | 05 volcano | 06 off limits | 07 grass yard | pass@1 ↓ | pass@3 ↓ | Turns ↓ | Output tokens ↓ | Total cost ↑ |\n|---|---|---|---|---|---|---|---|---|---|---|---|---|---|\n| GPT-5.5 | 100% | 100% | 8 | 15k | $13.62 | ||||||||\n| Claude Fable 5 | 100% | 100% | 6 | 29k | $40.98 | ||||||||\n| GPT-5.6 Sol | 100% | 100% | 11 | 10k | $11.91 | ||||||||\n| Claude Opus 4.8 | 96% | 100% | 35 | 46k | $63.14 | ||||||||\n| GLM-5.2 | 96% | 100% | 12 | 75k | $6.20 | ||||||||\n| Gemini 3.1 Pro | 92% | 100% | 8 | 38k | $12.41 | ||||||||\n| Gemini 3.5 Flash | 88% | 100% | 111 | 74k | $98.31 | ||||||||\n| GPT-5.6 Terra | 88% | 100% | 41 | 55k | $34.28 | ||||||||\n| Claude Sonnet 5 | 67% | 88% | 22 | 151k | $42.21 | ||||||||\n| GPT-5.6 Luna | 63% | 88% | 118 | 129k | $59.32 | ||||||||\n| Hy3 | 54% | 75% | 25 | 76k | $3.23 | ||||||||\n| MiniMax M3 | 46% | 63% | 96 | 173k | $34.43 | ||||||||\n| Qwen3.6 27B | 29% | 38% | 24 | 82k | $5.59 | ||||||||\n| DeepSeek V4 Pro | 29% | 38% | 37 | 183k | $11.49 | ||||||||\n| Qwen3.7 Max | 25% | 38% | 21 | 132k | $16.98 |\n\nThere was a split: strong models solved most levels, weaker ones struggled even on simple ones.\nStill, even a generous 60 min time limit per attempt was not enough for some models.\nWhile [Qwen3.6 27B might be good enough for website dev](https://quesma.com/blog/qwen-36-is-awesome/), it fails at atypical cognitive tasks.\n\nBut our biggest shock was that Gemini 3.5 Flash cost us 2.4x more than Claude Fable 5. And it did 111 turns on average — a trigger-happy approach, compared to that of frontier models.\n\nHere are the models that solved every intro level at least once (pass@3 = 100%), by how many turns they took and what they cost:\n\n## A refined approach to The Lake\n\nNow, we wanted to run Stage 1: The Lake, consisting of 14 levels, including two more difficult bonus ones.\n\nWe inspected previous solutions. Frontier models like GPT-5.6 Sol or Fable 5 would often spend several minutes reasoning about the puzzle. In most extreme cases, the models spent nearly 9 minutes on a single LLM call, producing tens of thousands of reasoning tokens — and burning a lot of our budget. Many correct solutions landed just short of the time limit.\n\nAlso, we saw that Terminus-2 harness is leaking — does not preserve reasoning tokens between agent calls — the model reasoned through the puzzle essentially from scratch each turn, burning budget and the time fuse.\n\nThough, before proceeding, we wanted a few improvements:\n\n- Using only models that were able to solve the initial stage.\n- Effectively no time limit — 60 min can still be harsh, especially if the API is slow.\n- Using dedicated harnesses.\n\nWe therefore realized that frontier models are probably capable of solving more levels, given a better harness and more compute budget. We picked:\n\n- Claude Code for Claude models and GLM-5.2.\n- Codex for GPT models.\n- A fixed version of Terminus-2 for Gemini models.\n\nThe last part might be surprising, but [Gemini CLI is discontinued](https://developers.googleblog.com/an-important-update-transitioning-gemini-cli-to-antigravity-cli/), and Antigravity CLI works only with subscription, thus would break our OpenRouter setup. We tried to use Claude Code and Codex, but Gemini models were not too collaborative either, with getting ominous `400: Corrupted thought signature`\n\n.\n\nWe set the reasoning effort to `high`\n\nfor each thinking budget mode. This time just one run per model, to save at least some of the budget.\n\nHere is the more generous run, and what we learned.\n\n### Claude Fable 5 is the fastest\n\nThe best models, Claude Fable 5 and GPT-5.6 Sol, managed to solve 13 out of 14 levels — all except one bonus level, *Sunken Temple*.\n\nWhen it comes to speed, Claude Fable 5 was the clear winner, solving the levels 3.1x faster than GPT-5.6 Sol.\n\nWe were happy to see that the newest generations of GPT and Claude models improved considerably in efficiency compared to their respective predecessors: Fable 5 solved levels 5x faster than Opus 4.8, and GPT-5.6 Sol was 1.6x faster than GPT-5.5.\n\nThe open-weight GLM-5.2 model was the slowest model we tested, but managed to solve 10 tasks after considerable time. It is a reasonably smart one, if we are patient.\n\n### GPT-5.6 Sol is the most cost-efficient\n\nNext we looked at the cost of the models solving the levels. GPT-5.6 Sol was 47% less expensive than Claude Fable 5. To our surprise, it was cheaper than the open-weight GLM-5.2.\n\nPeople often look at token prices — but they are essentially meaningless unless we know how many tokens are burnt, for example:\n\n- GLM-5.2’s effective token price is 4.1x lower than GPT-5.6 Sol’s, but it generated so many more tokens that the total cost was 1.7x higher.\n- Claude Opus 4.8 tokens are 2x cheaper than Claude Fable 5’s, yet the total cost was 2.4x higher.\n\n### GLM-5.2 beats Gemini 3.1 Pro and 3.5 Flash\n\nGoogle’s Gemini models lag behind considerably and couldn’t make any progress on a large portion of levels. Gemini 3.1 Pro repeatedly gave up and submitted an incorrect solution without verifying it. Gemini 3.5 Flash couldn’t make any progress on the remaining levels even after 1.5 hours per level.\n\nThe only runs we stopped manually were Gemini 3.5 Flash runs. This model is really fast, but fairly expensive — running it can cost a lot very quickly. We ultimately spent ~$220 on Gemini 3.5 Flash runs; the model managed to build up 500k-800k context windows and perform hundreds of attempts at solving the problem. But it was far from the correct solution.\n\nWhile Gemini models are in the lead on many benchmarks, in Baba Is You they were overtaken by GLM-5.2 by Z.ai.\n\n### Claude Fable 5 vs. a Twitch streamer\n\nAnd now a reality check — how does it compare to a human? Even though The Lake levels are hard for LLMs, this is an early and relatively easy stage of the game.\n\nAs a comparison, we watched [a playthrough of Baba Is You by the Twitch streamer Vinny](https://youtu.be/XNPeOl53qPc?t=870) and tracked how long it took him to beat the levels. The playthrough was done live on the release day of the game, so we think it accurately represents a realistic time to solve the puzzles by a first-time player.\n\n[Vinny](https://www.twitch.tv/vinesauce) was 4x faster than Claude Fable 5 and 13x faster than GPT-5.6 Sol. He effortlessly solved most of the levels in under 3 minutes, while LLMs often spend tens of minutes on them. He also managed to solve the bonus level, one too hard for the frontier models.\n\n## Caveats\n\nThese results come with multiple caveats.\n\nFirst and most important — in principle, an LLM can be trained on Baba Is You. Nothing prevents that — it is not an official benchmark, so training would be fair game. So far, it has not been the case, at least, not in a way that would impact the results.\n\nWe analyzed all 522 winning runs with Claude Fable 5, searching for memorized levels and solutions — see [the full audit](https://github.com/stared/baba-is-harbor/blob/main/recall_audit.md). Our tasks never name the game or the level, yet models recognize Baba Is You from the board alone. Only one winning run retrieved a walkthrough — for the wrong level, with a solution that did not fit the board — and won by ordinary search anyway.\nModels do try to recall level names, and get them wrong. Stuck on the *Jelly Throne* level, Gemini 3.5 Flash announced:\n\nYes! This level is called ‘Prison’!\n\nand applied the solution of that other level, failing. What does pay is some hints on the game mechanics — the same Gemini 3.5 Flash recalled a hint:\n\nIn Baba Is You, if an object is both YOU and WIN, you win immediately!\n\nThe containers have network access, so a model could `curl`\n\na walkthrough. Luckily, none ever tried — models assume they are offline. MiniMax M3, stuck on a level:\n\nLet me search online for baba-is-you puzzle solutions… wait, I can’t. Let me think.\n\nSecond, we are a bit unfair to AI agents. People play one level at a time, learning the patterns and meta-patterns of the game’s logic. We instead handed models a single level to reason about from scratch, not as part of a sequence — easier to run as a benchmark, but harder for the model. If someone wants to create an agent that can explore the board the same way as human players do, we would love to hear about it.\n\nIn several runs of Claude models, we observed that they tried to find the solution solely “mentally”, without making any moves in the actual game. This is how the only failed Claude Fable 5 run ended: it exceeded the reasoning token limit and couldn’t progress further. We observed similar behavior in several Claude Opus 4.8 runs.\n\nWe had numerous failed attempts. The initial approach was with 10 min time limit, which was fine for some models and some levels, but overall skewed results a lot. We tried to use the famed GPT-5.5 Pro on some hard tasks (solvable by other top models), but it surprisingly struggled. Fast iteration beats overthinking.\n\nAlso, needless to say, while the title is “Baba Is Solved”, it is a play on words. We solved the first two stages — waaay above what was possible a few months ago, it is not solving the whole game. The current models may or may not be strong enough to beat the whole game… but bring your own tokens, for it is a `[TOKEN] [IS] [MELT]`\n\nkind of affair.\n\n## Conclusions\n\nAI is getting smarter — puzzle games that were out of reach a year ago are now easily solved. But it also gets more and more expensive — we need to know not only if AI can solve a problem, but also if it is worth it. We shouldn’t look at token price anymore, but at price per task — since models differ in how verbose they are.\n\nFor video games the answer is easy — we don’t want AI, we want to have fun ourselves. For any other tasks, it seems that we are heading toward a token economy — taking for granted that agents can solve a task, but optimizing budget.\n\nWhich puzzles are still too hard for AI? And which practical tasks, while solvable by AI, make no economic sense so far?\n\nDiscuss on Hacker News, X, Reddit, and LinkedIn.\n\nStay tuned for future posts and releases", "url": "https://wpnews.pro/news/baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost", "canonical_source": "https://quesma.com/blog/baba-is-bench/", "published_at": "2026-07-16 00:00:00+00:00", "updated_at": "2026-07-16 14:59:42.709269+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-tools", "ai-agents"], "entities": ["Fable 5", "GPT-5.6 Sol", "Gemini 3.5 Flash", "GPT-5.6 Terra", "Claude Fable 5", "GPT-5.5 Pro", "GLM-5.2", "Baba Is You"], "alternates": {"html": "https://wpnews.pro/news/baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost", "markdown": "https://wpnews.pro/news/baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost.md", "text": "https://wpnews.pro/news/baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost.txt", "jsonld": "https://wpnews.pro/news/baba-is-solved-by-fable-5-and-gpt-5-6-sol-but-at-what-cost.jsonld"}}