Part 1: Strapping Claude in the driver’s seat #
1. Claude plays MTG
I was recently playing Magic The Gathering Arena and playtesting a deck against the MTG Arena built in AI bot, Sparky. It became obvious to me that Sparky was not very smart, it seemed to play random cards and even had hardcoded lines of play. So I had an idea. If models are anywhere near as capable as benchmarks claim them to be then this should be a piece of cake? So I started looking for ways to let LLMs play the game and what I settled on was Forge, an unofficial open source MTG rules engine built on Java that I could run locally. After several hours of fiddling around I had a working prototype where agents could take the place of a player and play the game against a human, the built in Forge AI, or another agent. So the next logical step was to set up an 8 player agent vs agent tournament and see which model would win. This is where the issues with the first pass of the harness started popping up.
2. It was the harness, not the model
At first it was exciting to watch the tournament unfold but that excitement soon wore off as I noticed the agents were making severe blunders. Take for example this example where Kimi’s reasoning showed that it should leave Karakas untapped so it could bounce Emrakul to Fable’s hand as it had done the last turn and keep itself alive. This was an extreme blunder that cost it the game but goes against its own plan. The final example has Gemini’s plan confusing a permanent in play for a spell it still has in its hand so it ends up activating its ability and sacrificing its only creature. Here the model completely misunderstood the board state.The models have reasoning traces that makes it clear they understand what to do, so I decided to investigate if it was actually the drivers seat I strapped it into that was the problem. After all, it doesn’t matter how good you are at driving a car if someone cuts the brakes and makes you drive blindfolded.
The first I found was with how decisions were being made. Many of the choices from the model to the Forge engine were being made by naively parsing the model’s replies for strings that matched actions or game objects. In this following example the model is deciding to keep its first hand or mulligan and draw a new hand. Opus said they would keep the hand but the mere mention of the word mulligan caused it to happen.
Model said: "No reason to mulligan, I'll keep this hand."
Parser saw: MULLIGAN
Game did: mulligan
The issue was not even on the model here. It lay with how the harness was handling its instructions. Another issue was that the harness was not giving the model the full view of the game state, which explains how in the reanimate example Grok failed to realize that there were no creatures to reanimate. The issue with Kimi and not leaving its Karakas untapped was a combination of the harness providing a very limited history of events and the biggest issue was hidden fallbacks to Forge AI and its automatic choices for tapping lands choosing attackers, and other choices that at first glance appeared to be made by the model but were oversights in the design of the harness.
It became clear to me that a weak harness could make a strong model look weak. My new goal was to elevate the harness to the level where it would show all the same information to the models as a human player gets from looking at the game. The harness should provide as little resistance to playing as possible.
3. Fixing the seat
So the question then becomes how do I get my harness to not hold anything back and let the models use over 30000 different cards? From watching the previous games and looking at where the models failed to take actions I identified and fixed the following issues.
- Answers as tool calls
- Full game state
- Unique ids for every object and a game log
- No hidden fallback Forge AI
- Rules lookup and simulation
Answers as tool calls
Instead of searching the models replies for text I now use tool calls that answer each decision by filling in a form with fixed fields so nothing can be interpreted from the models own thinking.
Full game state
Every decision will now show the model everything that a human player could see from both battlefields to graveyards and even number of cards in hand or in library. The old harness would also cut off cards with large amounts of text or leave placeholder text. The most important change here is handing the model a list of every legal action that the engine allows.
Unique ids for every object and a game log
Each object in play gets a unique id, like @67 that stays the same until it changes zones. Those ids also make the new game log readable so the models can read the log and see which objects with the same name are still in play. The whole log is not stored in context because models get worse the longer a conversation goes. So instead I kept the most recent and relevant events in context and gave the models tools to search and page through the full log if needed.
No hidden fallback Forge AI
In my old harness the Forge AI silently made decisions for the models and the logs incorrectly parroted that it was the models choice. Now the fallbacks are removed and if the model does for some reason fail to make a choice a default option is chosen and it is clearly marked in the logs.
Rules lookup and simulation
I also gave the models the abilities to search the official MTG rules the same way a player may check them. But the more interesting tool is the ability to simulate what an action would do. This means it can see a preview of what playing a specific card would do. Now this is not cheating as the model cant see the opponents cards or further in its own library but it useful for preventing blunders. For example it has helped when models want to cast a spell that would destroy a creature but that creature has indestructible, meaning it cannot be destroyed. The model can see that the simulation shows the creature would not be destroyed so they wont waste a spell.
I had made the harness so polished that the models had very little difficulty playing not only against other models but also against several of my human friends as well.
Part 2: The eval #
4. What an eval has to do
Now that the models could play friction free the most logical question is which model plays the best? A lot of my inspiration for this project comes from other people building evals and RL environments. Mechanize with their GBA emulator eval ( which is the main reason I started this project ), Prime Intellect with their verifiers library, and Olam Labs, who let agents loose in Minecraft. They all keep coming back to three things a good eval needs
- repeatability - run it again and get the same scores
- model differentiation - better models higher scores, worse models get lower scores
- trustability - you can check why a model got a specific score and attest it was not due to luck or a bad harness
These ideas all lead to the same idea, having a singular scalar score for your benchmark where it can be trusted and a higher score always correlates with better performance
5. Why winning games fails that test
My original idea was having different models play each other and simply seeing who wins the most but this fails all three of those tests. It was not repeatable. Magic has a lot of luck in it. You can play near perfect and still lose because you had bad luck and only drew lands. It doesn’t give a trustworthy evaluation because in Magic decks have a rock paper scissors dynamic where they are weak against some strategies and strong against others. If I only looked at wins it would depend more on what decks were picked than which models were playing. Kaggle’s Game Arena ran into this issue where the model that topped their chess ranking lost money at poker. So another approach was needed.
6. Puzzles: the room with one door
Chess puzzles give you a board state and a goal, such as ‘mate in three’. You either find the solution or you don’t, there is no randomness involved. This is the same idea behind Magic puzzles. You are given a task, usually to win that turn, and cards in play and in your hand that require careful thinking and deep knowledge of interacting systems to solve. This can be very useful for an eval if we make the acceptance criteria solving the puzzles. There is no half solving these puzzles, you either win or you don’t. Most of the Magic puzzles online only take a few steps to complete and the issue with that is that an algorithm making random choices in the engine can solve them with a somewhat high percentage, so I had all of our puzzles take at least 15 actions as this was the sweet spot for filtering out random play. The puzzles are also set in a way that they have one line that will lead to solving but they allow space for models to implement their own strategy to actually get there. In other words it’s like being stuck in a room with one door, the goal is clear but how they navigate the situation and arrive at the solution is not given out.
One of the puzzles in the eval is called Hoofdrain. In this scenario you have a bunch of elves, another creature that drains your opponent’s health each time one of your creatures dies, and Craterhoof Behemoth, a creature that increases the power of each of your creatures by the number of creatures you have in play. You have to play the creatures, attack, and drain your opponent in the correct order as the way to bring the opponent to 0 life and solve the puzzle. If a model does these in the incorrect order then they will miss damage and be unable to win. For example GPT 6 Luna cast Craterhoof too early and when they were finished their opponent was at 15 life and it failed the puzzle.
An especially good puzzle for testing models includes many additional objects in play that may or may not be related to the solution. Puzzles with few objects in play seemed very solvable for most models but as more objects were added they had to think longer over what objects are affecting each possible action.
7. Results
I chose 5 of the hardest puzzles where test runs showed smarter models started failing but while creating this I had created over 500 total puzzles with over half of them being what I deemed as hard or difficult for the models to complete.
If we look at these results we can see that for our example on Hoofdrain 10/20 models solved it.
8. Wrap up
One thing about the Forge engine is that it only offers legal moves so it can’t test whether a model knows the rules well enough to avoid making and planning illegal moves. Another thing to make note of is that these puzzles only cover single turns and not longer games with long term thinking and choices across turns rather than one singular long turn. There is much to still do in this project. I would like to expand the benchmark to include the full set of puzzles and even more models. I would also like to have puzzles that can exhaustively cover each mechanic and combinations of mechanics so we can start mapping out which parts the models easily work through or struggle with. I would also like to work on multi turn puzzles that can test long term thinking.
Some people might wonder if one were to create this full suite of puzzles and use this benchmark for reinforcement learning then wouldn’t the models’ behavior overfit on the solution of the puzzles? Well if we created these puzzles that represented all the possible interactions and situations from Magic then we would still end up with a model best suited for every possible situation!
Another byproduct of creating this eval is that I have created an AI opponent that is the most competent MTG AI that I have played against. I have let several friends play against models using this harness without telling them and they were unable to tell they were not playing against a human when faster models were used.
While building out this harness and preparing for the eval another thought occurred to me. Mechanize’s GBA eval has models building GBA emulators and judging the performance, but why couldn’t I do the same? Couldn’t I create an eval where agents actually work to create the same harness I had made for MTG Forge and then judge it against the acceptance test I made for each mechanic and action? If one is able to ride a bike that says nothing about their understanding of the rules and mechanics that allow that bike to move, but the same could never be said for one who creates bikes, as they must understand each part of a bike and how it will be used to correctly fabricate it.