OpenAI’s GPT-5.6 family landed on the Vercel AI Gateway on July 9, and we ran all three variants through our standard test the same day: build a complete, playable Flappy Bird in Godot, as an agent, across three prompts. Luna did it for $0.17, Terra for $1.15, and Sol for $1.59. Every one shipped a working game with zero GDScript errors. Meta’s Muse Spark 1.1 ran the same benchmark, spent $3.05, and never finished a “test and fix” turn.
Why benchmark with a game build #
The launch-day coverage benchmarks GPT-5.6 on abstract suites: 88.8% on TerminalBench 2.1 for Sol, 59 points on the Artificial Analysis index , one point below Claude Fable 5 at a third of the cost. Useful numbers, but none of them tell you whether the model can drive a real engine, where most Godot bugs only appear at runtime.
Our benchmark is the same one we used when MiniMax M2.5 released: the model works inside a live Godot editor through Ziva, with real tools: file writes, scene-tree edits, error logs, and a playtest tool that runs the game and returns an actual frame. Three prompts:
- Build a Flappy Bird clone: gravity, flap on Space, scrolling pipe pairs with random gaps, game over on collision
- Add a score counter that ticks when the bird passes a pipe, then playtest it and fix what you notice
- Add a Game Over overlay with the final score and a working Restart button
Scoring is anchored: DeepSeek v4 Flash (our free model) defines 3 stars, Gemini 3 Flash defines 4, and each run is graded 0-2 on five dimensions: tool-call reliability, code correctness, core loop, follow-up fidelity, and visuals. We verify every claim ourselves: reading the generated GDScript, pulling the engine’s error log, and running three playtests per game (let it die, flap through a pipe, trigger the restart).
Results #
| Model | Effort | Tool calls | Wall time | Build cost | Rubric | Stars |
|---|---|---|---|---|---|---|
| DeepSeek v4 Flash (anchor) | high | 124 | 28.5 min | $0.18† | 9/10 | 3/5 |
| Gemini 3 Flash (anchor) | high | 31 | 3.7 min | $0.25† | 10/10 | 4/5 |
| GPT-5.6 Luna | low | 26 | 2.4 min | $0.17 | 10/10 | 4/5 |
| GPT-5.6 Terra | xhigh | 46 | 7.3 min | $1.15* | 10/10 | 4/5 |
| GPT-5.6 Sol | low | 46 | 6.6 min | $1.59 | 10/10 | 4/5 |
| Muse Spark 1.1 | n/a | 197 | 60+ min (capped) | $3.05 | 7/10 | 2/5 |
† Anchor figures are their full log totals, which include capability probes and (for Gemini) narrating playtest frames for the text-only models. * Terra was scored on its xhigh run; its cost comes from a default-effort re-run (62 tool calls) that also went 10/10, so the picker’s cost field matches what a default-settings user pays.
What GPT-5.6 got right #
All three variants finished all three turns with zero errors and zero timeouts. That has not been the norm: our 3-star anchor DeepSeek v4 Flash built an excellent game but burned a 20-minute turn over-iterating on its own debug prints.
Sol’s first turn: two files, a scene-tree read, physics validation, and a self-playtest. Running cost: $0.29.
The variants differentiate exactly how OpenAI priced them. Luna was the leanest run of the entire benchmark, 26 tool calls with no wasted motion. Terra iterated more and added clouds and a dual restart path. Sol produced the most sophisticated build: CharacterBody2D physics, signal-based scoring, and the best-looking game of the six.
Sol at xhigh effort: parallax skyline, sun, title text.
Sol at default (low) effort: compact, complete, same 4/5 score.
Higher reasoning effort was a $10 lesson #
We first ran Sol at xhigh reasoning effort. It built the prettier game above, made 80 tool calls, hit three transient gateway socket drops, and its runs plus retries billed $11.69. Re-run at the default low effort: 46 tool calls, zero drops, $1.59, and the identical 10/10 rubric.
For agentic engine work, the loop is the intelligence. The model reads real errors and real playtest frames every turn, so buying more silent deliberation mostly buys longer turns. This mirrors the pricing trend we tracked across 2025-2026: capability per dollar keeps arriving faster than headline capability.
Muse Spark: the one we didn’t ship #
Meta’s Muse Spark 1.1 markets itself on agentic tool use. Its Godot game actually worked: flap physics, scoring, and a game-over screen, all error-free. It still scored 2/5, because it hit our 20-minute per-turn cap on all three turns, spending 197 tool calls where Luna spent 26.
The failure mode is specific and repeatable. Its build turn is a coin flip: one run capped after an 85-call physics rabbit hole, a second run finished cleanly in under three minutes. But turn 2, “playtest it and fix what you notice,” sent it into an edit-playtest-edit spiral in both runs, chasing marginal difficulty tuning it could never satisfy itself about.
Muse Spark, twenty minutes into turn 2, still editing the same file.
A model that reliably turns “test and fix” into a 20-minute meter-running loop is a bad deal at any per-token price, so it didn’t ship.
What changed in Ziva #
The three GPT-5.6 models are live in Ziva today, each showing its earned 4/5 quality rating, and we now surface the benchmark’s exact build cost in the model picker:
Hovering a model’s quality bars now shows what this exact benchmark cost with it.
Two housekeeping notes. First, GPT-5.1 Codex Mini, GPT-5.3 Codex, Claude Sonnet 4.6, and Grok Build 0.1 are retired; old chats keep working because the retired IDs resolve to their successors. Second, we list GPT-5.6 as temporary retention rather than zero data retention: the gateway currently has no ZDR-attested OpenAI route for GPT-5.6, and a privacy badge the routing can’t honor is worse than an honest one.
The fine print #
- One task type. Flappy Bird exercises 2D physics, signals, UI, and iteration against runtime feedback. It says nothing about shaders, 3D, or large refactors.
- The scale tops out at 4/5 by construction. Our anchors were DeepSeek (3) and Gemini 3 Flash (4), so a perfect run maps to 4 stars. Whether Sol deserves a 5 needs a frontier-anchored eval.
- Small n. One scored run per model, two for Muse Spark. The trio’s clean sweeps and Muse’s repeated spiral are consistent signals, not statistics.
- Restart buttons were verified in code and via keyboard paths; our playtest harness can’t click CanvasLayer buttons yet.
Total spend for the whole benchmark, including the superseded xhigh runs: $19.39. Cheaper than reading a week of launch-day takes.
All three GPT-5.6 models are available in Ziva now, and we re-run this benchmark whenever a new model hits the gateway.