# Can decision models play games from screenshots? > Source: > Published: 2026-10-20 14:00:53+00:00 Decision models are built to answer one kind of question quickly. You give the model a prompt and a fixed list of options, and it returns a probability for each option. That makes them a good fit for a step inside an agent: pick a tool, pick a route, decide whether to stop. We [recently benchmarked a number of these models](https://arize.com/blog/decision-model-benchmark/), and several of them now accept images, too. Cloudflare’s Clef Flash, Liquid’s d1 models and OpenAI’s Decisions route all handle images, and [TypeSafe’s Jev 1.14](https://typesafe.ai/blog/jev-1-14) added experimental image support. So we asked a fun question: “If a decision model can read an image and use that to make a decision, can it play a game?” Games are a clean test. The screen gives you the information you need, the rules are fixed, the legal moves are known, and the score can tell you exactly how well the model did. The eval here is reduced to a deterministic check: “did the model win?” or “what score did the model get?” We built a small browser arcade with three games and ran eight models through it. We mixed decision models with LLMs, and compared how well they did with images to how well they did with raw text. Then we put them up against a deterministic game engine as a comparison. How did they do? OK, but not great, depending on the game. I’m a terrible gamer and I was able to score way better than most of the AIs. But let’s dig into the models, the games, and the scores. ### Build better agents with Arize Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today. **Prefer open source?** [Try Arize Phoenix for self-hosted, open source agent observability](https://arize.com/phoenix?utm_source=blog&utm_medium=referral&utm_campaign=ax-inline-cta&utm_content=can-decision-models-play-games-from-screenshots-inline-cta-phoenix). ## **Meet the games** **Tic-tac-toe.** The classic three-in-a-row game on a 3 × 3 grid. The model plays against a perfect opponent, so the best it can do is a draw. Spaces are lettered A-I so the models know which spaces are empty and can respond with the letter of the space to play. **Minesweeper.** An 8 × 8 board hides 10 mines. Every opened tile shows how many mines touch it, and the aim is to open all the safe tiles without hitting a mine. If you open a cell with a mine, you lose. The rows are numbered and the columns are lettered so the model can indicate which tile to open. To avoid losing on the first square, there is always a safe square that the model can play to start. **Flappy Bird.** A bird flies forward on its own through gaps between green pipes. The only choice is when to flap, and gravity does the rest. The bird falls under gravity, unless you flap, which raises the bird up a little before gravity brings it back down. If you hit a pipe, the top of the screen, or the ground, you lose. ## **Meet the contestants** | **Model** | **Cloud vs Local** | |---|---| | [Jev 1.14](https://typesafe.ai/blog/jev-1-14) | Cloud | | [GPT-6 Luna Decisions](https://developers.openai.com/api/docs/guides/decisions) | Cloud | | [Clef Flash](https://blog.cloudflare.com/clef-decision-models/) | Cloud | | [Liquid d1](https://docs.liquid.ai/lfm/models/decision-models) | Cloud | | [Liquid d1-3B](https://huggingface.co/LiquidAI/d1-3B) | Local | | [Liquid d1-omni-600M](https://huggingface.co/LiquidAI/d1-omni-600M) | Local | | [GPT-6 Astra Responses](https://developers.openai.com/api/docs/models/gpt-6-astra) | Cloud | | [GPT-6 Luna Responses](https://developers.openai.com/api/docs/models/gpt-6-luna) | Cloud | This model set contains six decision models, a mix of local and cloud. The remaining two are LLMs, which were added as a comparison. GPT-6 Luna Responses and GPT-6 Luna Decisions are the same underlying model, one version is the normal LLM, the other is OpenAI’s version that is tuned for decisions. For each game the models were scored against a baseline using a deterministic game engine, a short piece of code written into the arcade itself, built on a standard technique: - **Tic-tac-toe: minimax.**[Minimax](https://en.wikipedia.org/wiki/Minimax) searches every possible continuation of the game and picks the move that’s best assuming the opponent also plays perfectly. Tic-tac-toe is small enough to search completely, and it’s a[solved game](https://en.wikipedia.org/wiki/Solved_game) . With perfect play on both sides, every game is a draw. The same minimax player is also the models’ opponent. - **Minesweeper: a constraint solver.** It treats every revealed number as a[constraint](https://en.wikipedia.org/wiki/Constraint_satisfaction_problem) on the covered tiles around it. If a clue’s mines are all accounted for, its other covered neighbours are safe. If a clue has exactly as many covered neighbours as missing mines, they’re all mines. It also compares pairs of overlapping clues. It reads only the visible board, as the models do, and if nothing is provably safe it falls back to the lowest-risk guess. - **Flappy Bird: a gap-centering rule.** A one-line autopilot: flap if the bird is falling and more than 18 pixels below the centre of the next gap, otherwise wait. Unlike the models, it reads the exact bird and pipe positions from the game rather than the pixels. The baselines aren’t competitors on equal terms as they have programmatic access to the game state. What they show is what a good score looks like. As a comparison, we also did a text only version, where the models received a prompt with a description of the game state. This was to allow us to compare how the images worked against a pure text input. TypeSafe recently released an image capable version of Jev, and was upfront that images are a work in progress. The [launch post](https://typesafe.ai/blog/jev-1-14) warns that *“all models (Jev included) perform worse on multimodality over pure text.”* These tests allow us to verify this claim. ## **Tic-tac-toe** ### **How we tested it** **Score:** 1 for a win, 0.5 for a draw, 0 for a loss, against the perfect minimax opponent. You can’t beat perfect minimax, so 0.5 per game (a draw) is the best possible result. Each model plays 10 games per input type, five as O and five as X. Each move for the model player is a single request sent to the model. The one exception is the last move. When only one empty square is left there’s nothing to decide, so the arcade plays that square itself without calling the model, as decision models generally reject a single decision prompt. The prompt is very explicit about the logic and describes how to play, how to win, and how to decide on the next move. If you can win, do so. If you need to block the opponent from winning, do so. Otherwise choose a square that helps you win. #### **Image input** When playing using images, the arcade sends a screen shot of the current board, a fixed prompt and the list of empty squares as the decision choices. There is no conversation history, just the image to give the model a view on the current game state. It gets back one choice, and plays it. This is a typical image of a game where the model is O. The minimax game engine has played square F, and the board image is sent to the model. This is the prompt that goes with it. It explains the game rules, what side the model is playing, the instruction, and gives the choices based on the remaining squares. ``` Prompt: Use the image as the source of the current tic-tac-toe board. X and O show occupied squares; letters label empty squares. You are playing O. Choose a legal square that wins immediately if possible, otherwise block an immediate opponent win, otherwise choose a move that preserves a draw against perfect play. Return one offered action. Image: [the 512 × 512 board above] Choices: A: Place O in square A. B: Place O in square B. C: Place O in square C. D: Place O in square D. E: Place O in square E. G: Place O in square G. H: Place O in square H. I: Place O in square I. ``` #### **Text input** The text games used the same positions with no image at all. The board state is sent as a list of rows with the Xs and Os. This is the full request for the same first decision: ``` Prompt: Use the textual board state as the complete tic-tac-toe board. Rows are listed top to bottom; square letters A-I run left to right, then top to bottom. Each square is marked with X, O, or empty. A player wins by placing three of their marks in a horizontal, vertical, or diagonal line; filling the board without a line is a draw. You are playing O. Choose a legal square that wins immediately if possible, otherwise block an immediate opponent win, otherwise choose a move that preserves a draw against perfect play. Return one offered action. State: Row 1: A=empty | B=empty | C=empty Row 2: D=empty | E=empty | F=X Row 3: G=empty | H=empty | I=empty Choices: A: Place O in square A. B: Place O in square B. C: Place O in square C. D: Place O in square D. E: Place O in square E. G: Place O in square G. H: Place O in square H. I: Place O in square I. ``` - **Jev 1.14** played FLAP on every decision in every game. The bird hit the top of the screen in all 10, after five decisions. - **Clef Flash** played FLAP, WAIT, FLAP, WAIT on every course, identical each time, and also hit the top of the screen in all 10. GPT-6 Luna Decisions mostly alternated too, with the odd extra FLAP, and hit the top of the screen in all 10. - **Liquid d1-3B** played WAIT on every decision and fell to the ground in all 10 games, after three decisions. - **Liquid d1-omni-600M** usually played FLAP twice and then waited, and hit the ground in eight of 10. Flapping every other decision is too much lift at this pace and never flapping is too little, so none of these patterns is a decision about the bird. From text the decision models were even more uniform, and most of them alternated FLAP and WAIT straight into the top of the screen. The two LLMs reacted to the course, but only GPT-6 Astra Responses changed its sequence enough to steer. ## **So, can decision models play games?** Yes, but not well. The two LLMs were far better suited to these games than the six decision models. ### **The LLMs played well** GPT-6 Astra Responses drew every Tic-tac-toe game against perfect play and cleared every Minesweeper board, from screenshots and from text alike. GPT-6 Luna Responses was close behind, with 19 draws in 20 and the only other cleared board. Astra was also the only model that really flew in Flappy Bird. Both were slow, though: 6 to 8 seconds per Minesweeper decision. ### **The decision models struggled** Across the six of them: 26 draws in 120 Tic-tac-toe games, against 39 in 40 for the LLMs. No Minesweeper board cleared, and a best gain of 5.5 tiles against Astra’s 21.2. In Flappy Bird, five of the six never passed a pipe. They missed blocks that were plainly visible, clicked tiles the board proved were mines, defaulted to the first option in text mode, and repeated one fixed FLAP and WAIT pattern whatever the course looked like. The near-flat probabilities they returned on lost positions suggest they often weren’t really choosing at all. Jev 1.14 is the one exception: nine of 10 Tic-tac-toe draws from images, the only decision model near the ceiling in any game. But even Jev cleared no Minesweeper boards and passed no pipes. That isn’t a verdict on decision models for the job they’re built for. They answered in well under a second, and in our [earlier benchmark](https://arize.com/blog/decision-model-benchmark/) that speed made them a good fit for routing steps inside an agent. Games are a different kind of decision. ### **Is it reasoning?** GPT-6 Luna is the cleanest comparison in this benchmark, because it’s the same model on two routes. Through the Responses route it drew 19 of 20 Tic-tac-toe games and gained 11.6 Minesweeper tiles from images. Through the Decisions route it drew four of 20 and gained 1.8. The obvious difference between the routes is that Responses reasons before it answers and Decisions doesn’t. So we tested that directly. #### **The test** Every LLM row in this post ran with the API’s reasoning effort set to low, the server default. For this test we ran GPT-6 Luna Responses again with the effort set to none. Every call came back with zero reasoning tokens, so reasoning really was off. Everything else stayed the same: the same 10 Tic-tac-toe games and the same 10 Minesweeper boards, from images and from text, against the same minimax and after the same opening. #### **The scores** | | **Luna Responses, reasoning on** | **Luna Responses, reasoning off** | **Luna Decisions** | |---|---|---|---| | Tic-tac-toe draws, image | 9 of 10 | 7 of 10 | 1 of 10 | | Tic-tac-toe draws, text | 10 of 10 | 4 of 10 | 3 of 10 | | Minesweeper tiles gained, image | +11.6 | +5.5 | +1.8 | | Minesweeper tiles gained, text | +8.7 | +2.3 | +1.5 | | Minesweeper boards cleared | 1 | 0 | 0 | | Seconds per decision | 3 to 8 | 1.2 to 1.7 | 0.2 to 0.5 | With reasoning off, Luna fell from 19 Tic-tac-toe draws to 11, and its Minesweeper games got much shorter: about two moves per game before it hit a mine, against 5.6 with reasoning on. #### **What it shows** **Reasoning is the biggest single factor.** Turning it off cost Luna eight of its 19 Tic-tac-toe draws and at least half of its Minesweeper gain from either input. But the input split shows reasoning isn’t the only reason: - From text, reasoning-off Luna played about as badly as Luna Decisions: 4 draws vs 3, and +2.3 tiles vs +1.5. - From images, it kept a clear lead: 7 draws vs 1, and +5.5 tiles vs +1.8. Same model, no reasoning on either route, yet Responses still did much better from a screenshot, so how each route handles the image matters too. One possible cause: Responses returns a single choice, while Decisions returns a probability for every option and the arcade plays the highest. This test can’t separate the two. **Three cautions:** - 10 games per input, so none of these gaps is statistically significant. - Reasoning off isn’t a single fast pass. Luna still took 1.2–1.7s per decision, vs well under a second through Decisions. - Flappy Bird is excluded; its reasoning-off games ran at an older frame setting. ### **Real time is hard for everyone** Even with the game paused for every decision, GPT-6 Astra Responses averaged 8.4 pipes from images and 3.9 from text, against the gap-centering rule’s 19.3. Astra also needed about 2.6 seconds per decision, so this only worked because the game waited. Real-time control is still a job for code in the inner loop, with a model setting the plan around it. ### **Images held up against text** For both kinds of model, put the three games side by side and there’s no consistent text advantage. For the LLMs, the input barely mattered in the turn-based games. GPT-6 Astra Responses was at the ceiling with both, and GPT-6 Luna Responses drew nine against 10 and gained 11.6 tiles against 8.7. In Flappy Bird, Astra did better from images (8.4 pipes) than from text (3.9). For the decision models, images were if anything slightly better. The six of them drew 15 Tic-tac-toe games from images and 11 from text, and five of the six gained more Minesweeper tiles from images. Jev drew nine against four and gained 5.5 tiles against 1.8, and Clef Flash gained 5.5 against 0.3. GPT-6 Luna Decisions went the other way in Tic-tac-toe, one draw against three. In Flappy Bird they scored zero or close to it from either input. Two cautions. First, with 10 games per cell the confidence intervals overlap almost everywhere, so none of these differences are statistically significant and we’re not claiming they are. Second, the text inputs aren’t a clean “same information, no pixels” control. They spell out coordinates and symbols the model would otherwise have to read off pixels, so they’re more explicit than the images. Our benchmark protocol says so: *“The model image and text conditions compare visual input with an explicit state serialization; they do not isolate an image encoder independently of text detail.”* So the fair reading is narrow. On these three games, reading a screenshot was not the bottleneck. A model that played well did so from either input, and a model that didn’t, didn’t. That’s a little at odds with TypeSafe’s warning that models do worse with images than with text. We don’t think it contradicts it. Our boards are simple, high-contrast and labelled, and the games are small. A busier image, or a task where reading the picture really is the hard part, could easily flip the result. ### **The baselines draw the lines** Minimax and the constraint solver both hit the ceiling, which shows these games are fully solvable from the visible board. The free Minesweeper opening shows that a raw score of 33 to 38 is mostly free tiles. And the Flappy Bird baseline shows what’s possible: 19.3 pipes for a one-line rule that reads the game state on every frame. If you’re choosing between a decision model and an LLM for a step in your agent, the principle that carries over is to measure each one against the floor as well as the ceiling. A score means very little until you know what you’d get for free and what perfect play gets. Those two numbers told us more about these models than the scores did, and they made the gap between the LLMs and the decision models impossible to miss. ## **Test your own decision points** A game is a toy version of a question every agent builder has to answer: which model should make this decision, and how do you know it’s making it well? The same approach works on your own agent. Collect the real decision points from your traces into a dataset, run each candidate model over it as an [experiment in Arize AX](https://arize.com/docs/ax/develop/datasets-and-experiments), and score the results with a deterministic check wherever the right answer is known, the way minimax and the solver did here. Put the floor and the ceiling next to every model, and the scores start to mean something. You can [get started free with Arize AX](https://app.arize.com/auth/join) and find out how your own decision models play.