# OpenAI Proves GPT-5.6 Sol's ARC-AGI-3 Score Triples Just by Fixing the Harness > Source: > Published: 2026-07-30 13:09:21+00:00 # OpenAI Proves GPT-5.6 Sol's ARC-AGI-3 Score Triples Just by Fixing the Harness OpenAI published a blog post showing GPT-5.6 Sol's ARC-AGI-3 score tripled from 7.8% to 38.3% just by turning on two API settings: Retained Reasoning and Compaction. The standard evaluation harness was dropping reasoning between turns and using rolling truncation. François Chollet confirmed the settings are permitted since they're general-purpose API features. OpenAI published a blog post this week that should force everyone to rethink how they read AI [benchmark](/glossary/benchmark) scores. GPT-5.6 Sol's ARC-AGI-3 result jumped from an unremarkable 7.8% to a competitive 38.3%. The model didn't change. The API settings did. Two specific settings accounted for the entire 30.5-percentage-point swing: Retained [Reasoning](/glossary/reasoning) and Compaction. Neither is particularly dramatic in isolation. Together they reveal that the standard [evaluation](/glossary/evaluation) harness for ARC-AGI-3 was quietly breaking the model rather than measuring it. ## What the Standard Harness Was Doing Wrong The ARC-AGI-3 evaluation harness — the software that runs the benchmark — was executing two operations between test turns that effectively sandbagged every model that passed through it. First, it was discarding the model's [chain of thought](/glossary/chain-of-thought) between steps. Every time the harness moved to a new turn in the multi-step reasoning task, it wiped the model's reasoning context. The model had to start each step from scratch, reconstructing its mental model from output tokens alone. That's like asking a mathematician to solve a proof by erasing their scratch paper after every line. Second, the harness used rolling truncation on the [context window](/glossary/context-window). Once the context filled up, it chopped off the earliest content — which in a multi-step reasoning task includes the model's earlier actions and observations. The model literally lost access to what it had done and seen earlier in the problem. ## What Retained Reasoning and Compaction Do Retained Reasoning preserves the model's chain of thought across turns. Instead of wiping internal reasoning state between steps, the setting keeps it available. For multi-step problems — which is what ARC-AGI-3 is designed to test — this matters enormously. Each step builds on the previous one, and without memory of why you made earlier decisions, later steps become guesses. Compaction swaps the truncation strategy for summarization. When the context window fills up, instead of chopping off the oldest content, the system generates a compressed summary of prior turns and keeps it in context. The model retains access to its earlier actions — not perfectly, but functionally. Together, these two settings turned GPT-5.6 Sol from a benchmark laggard into a benchmark leader. ## The Numbers in Context At 7.8%, GPT-5.6 Sol under the standard harness looked weaker than several open-weight models and roughly on par with systems nobody would consider frontier. At 38.3% with proper settings, it clears [Claude Opus](/compare/claude-4-opus-vs-gpt-o3) 5's 30.2% and lands within striking distance of the average human tester score of 48%. Francois Chollet, who created the ARC benchmark series, confirmed the settings are permitted under the rules. They're general-purpose API features, not model-specific tuning or custom scaffolding. Any model with equivalent API capabilities could use them. OpenAI noted that even the 38.3% figure likely understates the model's capability because the compaction process inevitably loses some information compared to truly unlimited context. ## What This Actually Measures The result exposes something uncomfortable about the AI evaluation industry. When two API settings account for a 30-point swing, what exactly is the benchmark measuring? It's measuring the interaction between a specific model and a specific harness configuration — not raw intelligence, not reasoning capability, not scientific understanding. That doesn't make the numbers meaningless. It makes them contextual in a way most benchmark reporting ignores. The difference between 7.8% and 38.3% is the difference between dismissing a model and taking it seriously as a reasoning engine. Most decisions about which models to use, which to build on, and which to regulate are made on exactly these numbers. ARC-AGI-3 is supposed to be the gold standard for measuring AI reasoning progress. Chollet designed it specifically to resist memorization and test genuine abstraction capability. If the harness can swing results this dramatically on the gold standard, what does that say about every other benchmark out there? The answer is that most benchmark scores should come with a footnote: results depend on how you configure the evaluation, not just how good the model is. Some labs publish with optimal settings already. Others use defaults that may be actively harming their scores. Comparing numbers across labs without accounting for harness configuration is comparing apples to oranges while pretending they're both fruit. #### Q: What exactly is ARC-AGI-3? #### Q: What are Retained Reasoning and Compaction? #### Q: Why did the standard harness drop the reasoning context? #### Q: Does this mean GPT-5.6 Sol is smarter than Claude Opus 5? #### Q: How reliable are AI benchmarks in general? Get AI news in your inbox Daily digest of what matters in AI. ## Key Terms Explained [AGI](/glossary/agi) Artificial General Intelligence. [Benchmark](/glossary/benchmark) A standardized test used to measure and compare AI model performance. [Chain of Thought](/glossary/chain-of-thought) A prompting technique where you ask an AI model to show its reasoning step by step before giving a final answer. [Claude](/glossary/claude) Anthropic's family of AI assistants, including Claude Haiku, Sonnet, and Opus.