# I let LLMs play a 1988 detective gamebook 646 times > Source: > Published: 2026-11-15 15:36:13+00:00 Somewhere in the middle of *The Money Spider*, a 1988 detective gamebook by Robin Waterfield and Wilfred Davies, you’re in a hot-air balloon and a storm is coming. You can land, or you can run before the wind. Landing is the sensible choice. In 50 of the 58 runs that got that far, the model landed the first time. Then it lost the game because of it. The loss came about ten decisions later, at the very end of the book, and the model almost never worked out why. I spent a week letting LLMs play this book and its 1988 sequel through the [GitHub Copilot SDK](https://github.com/github/copilot-sdk): 95 runs and 646 games. The book turned out to be a good stand-in for a problem agents hit in real work: a failure at the very end, caused by a sensible decision made much earlier. Think of a deploy that fails at the last stage because of a config choice made in the first. Here’s what I found: - All 95 first attempts lost, across every model and setting I tried. - With retries, capable models learned one lesson per loss. GPT‑6 Astra at maximum effort won 7 of 15 runs with 10 attempts, and 6 of 6 with 15. - More attempts beat a bigger model. Five extra attempts halved the cost per solve. More reasoning effort or a pricier model mostly didn’t help. - Most late losses came down to one missing clue. You earn it about ten decisions earlier, through two choices that don’t look like detective work. The models rarely traced the loss back to it. Notes they wrote for themselves didn’t help either. - On the sequel, the same model won all six runs within seven attempts, while my brute-force solver gave up after 10 million states. ## How the models played I built a TypeScript engine that runs both books. You’re a detective called T.S. Each section of the story ends with a few options, and every option tells you which section to turn to. Some sections tell you to *mark a number on the Web*, which is where you keep track of the clues you’ve collected. Later, the book checks the Web: “If you have marked all of 22, 55, 60, 3, 51, 20 and 50, turn to 175. Otherwise, turn to 201.” Section 175 solves the case, while section 201 has your boss telling you the evidence won’t hold up in court. The LLM player sends each section to a model and feeds the model’s choice back to the engine. The models have no tools and no access to files or the internet, so they can’t read the story’s source or look up a walkthrough. Every model got the same system prompt. Its line about no retries dates from the first experiments, where each model played a single game. I kept the prompt word for word when I added retries, so every run got the same instructions: ``` You are playing a choose-your-own-adventure detective story. After each part of the story you'll get a numbered list of options. Read carefully, remember the clues you've collected, and choose the option most likely to solve the case. There are no retries: a wrong ending means you lose. Reply with JSON only, e.g. {"choice": 2}. ``` I let the models play the game in two modes: - **As printed:** the model sees the section numbers, where each option leads, the book’s instructions such as “Mark 60 on the Web”, and the Web itself, just like a reader of the paper book. The system prompt gets one extra sentence explaining these. Most of the results below come from this mode. - **Bookkeeping hidden:** the model gets only the story text and the options. The engine marks the Web and runs the checks silently. ## Every first attempt lost I started with what most benchmarks measure: one game per model. I screened 13 models from Anthropic, OpenAI, xAI and Microsoft, plus Copilot’s automatic model picker, at default reasoning effort. Then I ran the strongest ones again at maximum effort. None of them won. Across the whole project, models played 95 first attempts, and all 95 lost. That also answers whether the models simply remembered the books: a model that had memorized a walkthrough should have won on the first try. The losses weren’t random either. With the bookkeeping hidden, five of six games at maximum effort fell into the same early trap: questioning one suspect before another lets the second one flee. Claude Opus at maximum effort made the same seven choices in all three of its games and lost at the same place each time. Copilot bills in premium requests, one per decision for most models and 15 for Opus, so that was 315 premium requests to learn nothing. ## Then they learned So I changed the question. Instead of one game, each run gets up to ten attempts. After a loss, the model sees “THE END: you lost. You get another attempt”, and the story starts over. It isn’t told how many attempts it has left, and its only memory is the full transcript of its earlier games. To measure progress when nobody wins, I had a solver work out the 32 checkpoints every winning game passes through, in order. Then I tracked how far each attempt got. Here are all 21 runs of GPT‑6 Astra at maximum effort, played as printed: Read any row from left to right and you’ll see the lessons pile up: escape the early trap, survive the island, don’t drink the drugged coffee. Every loss teaches the model one thing, and it rarely makes that mistake again. Then the rows reach checkpoint 32, the final check, and stop improving: many of them fall back to the first few checkpoints instead. Not every model learns, though. Claude Haiku 4.5 and GPT‑6 Luna played ten attempts per run and showed no trend at all. In its tenth attempt, Haiku still barely got past the early investigation. Below a certain level of capability, more attempts don’t help. ## More attempts beat a bigger model As printed, Astra at maximum effort won 7 of 15 runs. Five of those seven wins came on attempt 10, the last one allowed, which looked like a ceiling of my own making. So I ran Astra again with up to 15 attempts. All six runs won, and the two that hadn’t won by attempt 10 got there on attempts 12 and 13. Because a run stops at its first win, the extra attempts barely raised the average cost per run, while the win rate doubled. Cost per solve dropped from 789 premium requests to 396. The other knobs gave mixed results. Some configurations had only two runs, and the runtime summarized Claude’s conversation history sooner than GPT’s, so treat these as observations rather than rankings: - Reasoning effort: GPT‑6.1 Sol learned at about the same rate at low and maximum effort. Maximum effort cost the same in premium requests and took 2.7 times as long. Claude Sonnet 5.5 was the exception: it won 2 of 6 runs at maximum effort but never reached the final check at default effort. - A bigger model: Opus at maximum effort learned no faster than Sol and cost 15 times as much per decision. - Showing the bookkeeping: this helped mid-tier models most. Luna reached the final check only as printed. Across all models, only one run with the bookkeeping hidden ever won. ## Models had no clue Capable models reliably learned their way to the final check, checkpoint 32 of 32, where the book asks for seven markers. And then they lost there, over and over. Of the 97 times a capable model reached that check, it was missing something 80 times. In 74 of those, the missing piece was marker 60. Marker 60 takes two counterintuitive choices in a row. In the balloon, you have to run before the wind instead of landing. Then, when kidnappers walk into the chapel where you’ve hidden, you have to take action now instead of waiting for the second one to come closer. A second route, through a climbing holiday, works the same way. Neither pair of choices looks like it has anything to do with evidence. A model that has never made them has never even seen marker 60 handed out. As printed, the final check lists marker 60 among the clues it needs. So the model knows *what* it’s missing, but not *where* it was supposed to get it. Nothing about the failure points back to the two reasonable-looking choices that caused it, about ten decisions earlier. The models’ explanations for the loss were plausible and wrong. Once they had a wrong explanation, they kept it. When a run reached the final check missing only marker 60, its next visit to that check fixed it just 3 times out of 13. Eight times it was missing 60 again. The grid above shows what they did instead: after reaching checkpoint 32, many runs fell back to checkpoint 1 or 2. They changed their early investigation and lost before they ever got near the balloon. I also tried replacing the transcript with notes: after each loss, the model writes itself up to 500 words of advice and starts the next attempt fresh. None of the six runs won. The notebooks show why, starting with the first lines of GPT‑6.1 Terra’s notes: Critical: Final reviews require 42, 48, 5, … 20 and 50. One attempt lost solely because 60 was missing. Find how to mark 60 before leaving preliminary inquiries. The model knew exactly which clue cost it the game, but it went looking for that clue in the wrong part of the book. Further down, the same notes record the real culprit. Every later attempt followed this advice faithfully: Visit Klaus in Basle; take the balloon trip and land when weather worsens. ## A second book, a different result To see whether any of this generalizes, I ran Astra on the sequel, *The Water Spider*. On paper it’s a twin: about the same size, with about as many choices and evidence checks. Long-range dependencies are just as common: in both books, nearly half of the conditions on game-ending checks depend on a choice made more than 20 decisions earlier. If anything, the sequel is stricter, with more checks that can end the game and a cap on how many clues you may hold. Astra won all six runs, on attempts 5, 5, 5, 6, 6 and 7. It had a blocker too. At the old whaling station, you either go in alone while your companion keeps watch, or you both go in together. Going in alone gets you captured two or three steps later. Every run lost there exactly twice, then switched and won. Since both books have plenty of long-range dependencies, the difference comes down to whether the failure points back at its cause. In *The Water Spider*, required clues come from investigation leads you pick from lists (“Check up on the acid”, “Look into the cargo at Vigo”). Most of the final check’s markers are already required by earlier checks, where falling short doesn’t end the game. The book tells you early, in terms you can act on. In *The Money Spider*, marker 60 is first required at the very end and comes from choices that don’t look like investigation at all. You can see the same thing in individual choices. I looked at 19 choices between a cautious and a bold option. The models’ first instinct was whatever a sensible professional would do: careful around physical danger, proactive when investigating. What decided whether they dropped an instinct was how soon it got punished, because a quick penalty is what makes a failure point back at its cause. Accepting a drink gets you drugged soon after, so models stopped accepting within an attempt or two. Landing the balloon looks fine until the final check. Across all attempts, capable models landed 259 times out of 306. The second book surprised me in one more way. My search-based solver, which sees the whole story graph, found a winning route through *The Money Spider* in 0.1 seconds. On *The Water Spider*, it gave up after 10 million states without finding one. Astra won it six times out of six. My guess is that brute force struggles with that book’s cap on how many clues you may hold, while a model that reads “you’ve been following too many trails” learns the rule in an attempt or two. ## So what? A gamebook is a toy version of a kind of work developers do all the time: you learn a system by trying things and remembering what led where. When a failure sits right next to its cause, an agent with a few retries maps the system quickly. That’s *The Water Spider*. You go into the whaling station alone, you get captured a few steps later, and after a loss or two, you stop doing it. But a lot of the work we’d like to hand to agents is *The Money Spider*: the deploy from the start of this post, or a pen test that stalls because the credential you needed was on a host you dismissed as uninteresting an hour ago. The failure tells you *what* is missing but never *where* you dropped it. And the decision that caused it is usually the sensible one, so nobody thinks to question it. Worse, the explanation that seems obvious is often wrong. People fall into the same trap. A postmortem that blames the wrong root cause becomes the runbook, and then everyone follows it faithfully. Terra’s notes said “land the balloon”, your wiki says “restart the service”. If you build or use agents, here’s what to take from this: - Measure learning, not just first tries. No model won on the first attempt, yet the best one won every time with 15. - Budget attempts before you budget a bigger model. In my runs, five extra attempts halved the cost per solve, while Opus cost 15 times as much per decision and learned no faster. Find out [how much model is enough](https://blog.mastykarz.nl/whats-your-minimal-viable-model/) before paying for more. - Keep the raw history. Summaries keep the conclusions and throw away the evidence, including the evidence that the conclusion was wrong. None of the runs that kept notes instead of transcripts won. - Show the agent its state. With the book’s bookkeeping hidden, only one run ever won. - Shorten the feedback loop. *The Water Spider* asked for its clues early, while there was still time to go and get them. In a codebase, that means preflight checks, typed config, tests that fail at the step that’s wrong rather than at the end. Some problems won’t let you do that, because their feedback only arrives at the end. When you hand one of those to an agent, budget for more attempts and expect it to blame the wrong decision. - Treat a confident explanation as a [hypothesis worth testing](https://blog.mastykarz.nl/agent-start-hypothesis-not-plan/) . When an agent keeps failing at the same place with the same explanation, the explanation is the bug. What I keep coming back to is the balloon. Almost every model chose to land, and given what it knew, landing was the right call. Winning, however, took something else: working out, from an ending, which of its good decisions was wrong. The best model got there, but only after 8 to 13 attempts with its whole history in front of it. Working from their own notes, no model got there at all.