# OpenAI Says GPT 5.6’s Score On ARC-AGI 3 Tripled After Turning On Two API Settings > Source: > Published: 2026-07-30 07:09:27+00:00 AI labs are still finding out novel ways to improve on the ARC-AGI benchmarks, which are designed to be particularly hard for AIs to solve. OpenAI has published a new analysis showing that GPT-5.6 Sol’s score on [ARC-AGI-3](https://officechai.com/ai/arc-agi-3/) nearly tripled once the company swapped out the benchmark’s default harness for two settings it already uses inside ChatGPT and Codex. With the official ARC-AGI-3 harness, GPT-5.6 Sol scored 13.3% on the public task set. Using OpenAI’s Responses API with retained reasoning and compaction turned on, the same model scored 38.3%, while using roughly six times fewer output tokens per game. The gap is the kind of thing that tends to get buried in a leaderboard. ARC-AGI-3 is a benchmark of 2D puzzle games where an agent has to explore an unfamiliar environment and work out the rules on its own, without instructions. On the public leaderboard for the game OpenAI tested with, no frontier model gets past the first level. With the modified harness, GPT-5.6 Sol cleared all six. ## What was actually going wrong OpenAI’s researchers, Ilan Bigio and Ted Sanders, said they went looking for an explanation after GPT-5.6 Sol scored just 7.8% on ARC-AGI-3, while GPT-5.5 barely registered at 0.4%. Both numbers looked odd next to what the models had done elsewhere. GPT-5.6 Sol has cleared Pokémon FireRed on a vision-only setup, played through Slay the Spire 2 using Codex’s computer use tools, and made progress on the early stages of Baba Is You. Puzzle games clearly weren’t the issue by themselves. The actual culprits, according to the post, were two features baked into ARC’s generic test harness. After every action the agent took, its private reasoning was thrown away, so the model had to re-figure out the game from scratch on each turn. It could still see a log of past moves and short notes attached to them, but not the thinking that produced those moves. On top of that, the harness used rolling truncation, discarding the oldest parts of the conversation once it grew past 175,000 characters, so the model was also losing track of earlier observations as a run went on. OpenAI says this combination is why the model looked, in its own words, not particularly bright while playing. It kept dwelling on individual actions and failed to build on what it had already learned about a given game. ## The fix, and why it matters for benchmark design Since GPT-5.6 Sol was built through OpenAI’s Responses API, reasoning persists across turns by default if a developer passes along the previous response ID, rather than being discarded after every call. OpenAI reimplemented the ARC-AGI-3 harness this way and replaced rolling truncation with compaction, a Responses API feature that summarizes older context instead of dropping it outright. With both changes in place, the model spent less time re-deriving the rules of each game before acting and held on to strategies it had already worked out, which OpenAI says explains both the higher score and the drop in token usage. The company’s broader argument is that benchmark results say as much about harness design as they do about the underlying model. It’s a point ARC itself raised in an earlier writeup examining GPT-5.5’s weak performance, which appears to be what nudged OpenAI toward digging into this in the first place. OpenAI’s recommendation to developers building on its API is straightforward: use the Responses API rather than the older Chat Completions API, keep reasoning retained across turns, and turn on compaction for long-running tasks. ## ARC Prize’s response The ARC Prize Foundation, which runs the benchmark, responded to OpenAI’s post by acknowledging the result while defending its own testing approach. The organization said its verified scores intentionally use what it calls a “no harness” setup, where every model receives the same observations, system prompt, and action limits, and conversation state is managed on the client side through the standard completions-style interface rather than any provider’s proprietary session management. The reasoning, according to ARC, is to prevent developers from tuning a harness around a specific model’s quirks and to keep comparisons across labs fair. ARC said it wants progress on its benchmark to reflect genuine gains in general intelligence rather than familiarity with a particular API’s formatting or settings, and that it is now working with OpenAI and other labs to figure out how server-side state management, of the kind OpenAI demonstrated, might eventually be folded into verified testing without giving any one provider an edge. The exchange lands at a point where [ARC-AGI-2 is edging toward saturation](https://officechai.com/ai/gemini-3-deep-think-benchmarks-arc-agi/) among frontier labs, and ARC-AGI-3 has taken over as the harder, more contested test. François Chollet, who co-founded ARC Prize, [has said he expects AGI itself is still years away](https://officechai.com/ai/agi-likely-by-early-2030s-when-arc-agi-6-or-7-will-be-released-arc-prizes-francois-chollet/), with later versions of the benchmark meant to track that gap as it closes. How much of that closing turns out to be model capability versus harness engineering is exactly the question this exchange between OpenAI and ARC Prize has put back on the table.