{"slug": "the-llm-was-better-at-building-a-solver-than-playing-the-game", "title": "The LLM was better at building a solver than playing the game", "summary": "A developer built a deterministic solver for a Dota 2 drafting game after finding that an LLM underperformed. The solver, tested with common random numbers and strict train/validation/test splits, achieved a higher title rate than the LLM. The project highlights the importance of rigorous benchmarking in AI applications.", "body_md": "I started this project because an LLM annoyed me.\n\nI gave a very strong model [322](https://322-0.app/), a small Dota 2 drafting game. The choices looked like the kind of work a computer should enjoy: repeated packs of players and heroes, visible ratings, familiarity scores, chemistry, rerolls and a simulated tournament at the end.\n\nI was disappointed by how well the LLM did. I am not a Dota expert, and I had only started watching it occasionally again during the previous six months or year. I still seemed to be doing better.\n\nThe interesting engineering question was not how to write a longer prompt. It was how to replace the card-by-card language-model judgement with a deterministic policy, then test that policy without confusing improvement with luck.\n\nThe browser history gave us a useful irritation and almost no reliable comparison.\n\nMy earlier manual record contained 50 runs with a 14% title rate. The LLM won once in nine attempts. Putting 14% beside 11% looks temptingly quantitative, but the random offers, rejected packs and opponent fields were not preserved. The samples were small, unpaired and produced under different choices.\n\nThat is not a model benchmark. It is a reason to build one.\n\nThe offline solver generated every random choice from indexed tapes. Policy A and policy B received the same player offers, hero samples, field candidates and tournament randomness for a given episode. We could then compare the paired result: did the new policy win this exact episode where the old policy lost it?\n\nThis is the common-random-numbers idea in a practical form. Sharing the luck removes a large amount of noise that has nothing to do with the policy change.\n\nBefore evaluating a strategy, we reproduced the game.\n\nThe public client and seven data files were frozen with SHA-256 hashes. Draft legality, automatic hero allocation, chemistry, scoring and the tournament were ported into a deterministic Python engine. Automatic allocation evaluates all 120 player-to-hero permutations, so reproducing that detail mattered.\n\nThe engine also kept an important semantic boundary. A finished roster has an exact score under the copied rules. An unfinished draft does not. A policy can estimate the future value of a player, hero or reroll, but that partial-state estimate is not made exact by giving it several decimal places.\n\nKeeping the simulator authoritative meant we could change policies without changing what success meant.\n\nOpponent fields introduced a slightly odd control problem. If a solver can redraw for free forever, “find an easier field” eventually dominates every other decision.\n\nWe declared three finite budgets instead:\n\n`K=1`\n\naccepts the first opponent field.`K=4`\n\nselects from four fields.`K=16`\n\nselects from sixteen fields.The budgets were nested. The first field in `K=1`\n\nwas also the first field in `K=4`\n\nand `K=16`\n\n; the first four were shared too. Policies could be compared at each budget without quietly receiving different field draws.\n\nThis was a small design choice with a large effect on the claim. A title rate without the field budget would not describe a reproducible policy.\n\nThe solver could be improved and improved and improved. That was satisfying, but it created the usual risk: every time a result influences the next change, that result has become part of training.\n\nWe separated train, validation and test tapes. Candidate policies first went through successive halving on shared training episodes. Weak candidates stopped early; survivors received more budget. The selected candidate then needed to pass a fresh validation gate before the test partition could be opened.\n\nThe gate required positive title-rate results, no meaningful top-four or top-eight regression, zero failures, deterministic replay and acceptable runtime. Once the final test started, policy code, features, model bytes, parameters, seeds and episode counts were frozen.\n\nThe controls themselves still needed scrutiny. An independent audit found that an early replay check compared duplicate in-memory runs rather than the public replay path, and that the first runtime gate reused warm policy caches. Both were repaired before the production claim. The audit separately recomputed the final statistics from the raw logs.\n\nThe inconvenience was deliberate. It stopped “one more tweak” from leaking the answer back into the candidate.\n\nThe final test compared four generations of policy on 10,000 paired episodes at each field budget. Across all policy and budget combinations, that was 120,000 outcomes with zero failures.\n\nAt `K=16`\n\nthe progression was:\n\n| Policy | Title rate |\n|---|---|\n| Pick the highest player rating | 2.30% |\n| Hand-written heuristic | 14.77% |\n| Assignment and chemistry components | 22.33% |\n| Frozen 37-feature value model | 27.45% |\n\nThe final value model was ridge-linear, deterministic and trained offline on 8,511 action rows. It used features for role-feasible future strength, partial hero allocation, current and potential chemistry, familiarity, draft stage and remaining rerolls. It did not call an LLM while playing.\n\nAgainst the previous component policy, its `K=16`\n\ntitle improvement was +5.12 percentage points with a paired 95% interval from +4.24 to +6.00. Top-four and top-eight results improved at every budget too.\n\nThe complete draft loop took a median 47.1 milliseconds with a cold model build. That was roughly 1.8 times the component policy's latency, and the runtime policy made no large-model request for any offer.\n\nWe try to solve everything with LLMs now, and we do not necessarily need to.\n\nFor 322, the runtime decision-maker became ordinary deterministic code. The LLM was much more useful around it: inspecting the browser game, reconstructing mechanics, implementing the engine, proposing policies, running controlled experiments, auditing weak controls and producing visualisations that made the improvement legible.\n\nThe result was quite a cool little statistical project. The useful boundary was not “LLMs are bad at games.” It was more specific: repeated numeric decisions with a reproducible state and measurable outcome did not need an LLM in the hot path.\n\nIf I continued the project, I would use the LLM to choose and build the next experiment. I would leave the next card choice to the solver.\n\nWant to talk about something I’ve written or built? [Get in touch](https://billiem.uk/contact/).\n\nThis article was adapted with AI assistance from [an original article on billiem.uk](https://billiem.uk/posts/llm-better-building-322-solver/). The original article was reviewed before publication.", "url": "https://wpnews.pro/news/the-llm-was-better-at-building-a-solver-than-playing-the-game", "canonical_source": "https://dev.to/billiem/the-llm-was-better-at-building-a-solver-than-playing-the-game-5ck0", "published_at": "2026-08-04 09:47:21+00:00", "updated_at": "2026-08-04 10:13:08.699982+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "developer-tools"], "entities": ["Dota 2", "LLM", "322"], "alternates": {"html": "https://wpnews.pro/news/the-llm-was-better-at-building-a-solver-than-playing-the-game", "markdown": "https://wpnews.pro/news/the-llm-was-better-at-building-a-solver-than-playing-the-game.md", "text": "https://wpnews.pro/news/the-llm-was-better-at-building-a-solver-than-playing-the-game.txt", "jsonld": "https://wpnews.pro/news/the-llm-was-better-at-building-a-solver-than-playing-the-game.jsonld"}}