OpenAI says that two Responses API settings, retained reasoning and compaction, raised GPT-5.6 Sol's public-set ARC-AGI-3 score from 13.3% to 38.3%. The result matters because it shifts part of the explanation for a weak benchmark outcome away from the model alone and toward the software harness used to run it. OpenAI also reports that the configuration reduced output tokens by about six times.
The company had previously presented GPT-5.6 Sol as capable of advanced mathematical work, including attempts at open mathematical problems. Its ARC-AGI-3 investigation instead found that the evaluation setup was not allowing the model to retain what it had learned while working through the benchmark's 2D puzzle games. In OpenAI's detailed ARC-AGI-3 write-up, the company argues that API configuration can materially affect both agent performance and token efficiency.
This is an important deployment lesson, not simply a higher benchmark number. For developers building agents that must reason over multiple steps, the surrounding system determines whether useful intermediate work persists, how context is managed, and how much output is consumed to reach a result.
ARC-AGI-3 is designed around interactive 2D puzzle games, a setting that can require an agent to try actions, observe outcomes, and adapt its approach. OpenAI's analysis says the initial harness created a memory-related limitation: GPT-5.6 Sol could not effectively carry forward relevant reasoning from earlier work.
The company identified two Responses API settings as the remedy:
The distinction is consequential for benchmark interpretation. A score generated through an agent harness reflects the combined behavior of the model, the API settings, the task loop, and the evaluation procedure. It is not solely a measure of the underlying model weights.
| Reported measure | Initial harness | Harness with retained reasoning and compaction |
|---|---|---|
| GPT-5.6 Sol public-set ARC-AGI-3 score | 13.3% | 38.3% |
| Output tokens | Baseline reported by OpenAI | About 6x fewer output tokens | | Reasoning persistence | The harness did not let the model retain what it learned | Retained reasoning enabled |
OpenAI's reported improvement connects reliability and cost. If an agent repeatedly reconstructs reasoning that could have been retained, it can spend more tokens while still performing less consistently. Conversely, preserving relevant work and managing context can allow a model to continue a task more effectively while using fewer generated tokens.
That does not mean the settings automatically improve every workload by the same amount. The 13.3% to 38.3% change is a reported result on the ARC-AGI-3 public set under OpenAI's evaluation configuration. It should not be treated as a universal multiplier for every application, model, or benchmark.
It also should not be conflated with every ARC-AGI-3 result. The supplied ARC Prize leaderboard context cites a 7.78% public ARC-AGI-3 demo score for Sol Max. Different reported scores can reflect different models, modes, harnesses, or evaluation conditions. Developers comparing results should therefore identify the exact model configuration, API workflow, task set, and scoring environment before drawing conclusions.
The practical takeaway is to test the application stack, not just the model selection. A model that appears to underperform in an agent loop may be constrained by how state and context are handled. OpenAI's recommended configuration is specifically to use the Responses API with retained reasoning and compaction enabled.
For teams evaluating reasoning models, a disciplined process should include: Organizations assessing reasoning agents for production can work with Scalevise on ** AI architecture, workflow automation, and API integration** that evaluates model behavior alongside the context, state, and cost controls of the surrounding system.
OpenAI's broader GPT-5.6 Sol materials provide useful context for why the ARC-AGI-3 result attracted attention. The company has documented math-focused model outputs and published material related to a discrete-geometry result involving the unit-distance problem in the plane. Those claims concern a different class of reasoning task, but they reinforce the central lesson of the ARC-AGI-3 investigation: strong capability in one environment does not guarantee that an agent harness will expose that capability effectively in another.
What API settings did OpenAI say improved GPT-5.6 Sol's ARC-AGI-3 score?
OpenAI identified retained reasoning and compaction in the Responses API. It says enabling both settings increased GPT-5.6 Sol's public-set ARC-AGI-3 score from 13.3% to 38.3%.
How much did the new ARC-AGI-3 configuration reduce token use?
OpenAI reports that the configuration reduced output tokens by about six times while improving the reported public-set score.
Does the 38.3% result apply to every GPT-5.6 Sol use case?
No. The figure is OpenAI's reported result for the ARC-AGI-3 public set with a specific harness configuration. It does not establish the same improvement for other tasks or deployments.
Why can an API harness affect an AI benchmark score?
An agent harness controls how a model receives context, retains work, takes actions, and continues a task. OpenAI found that its earlier setup did not let GPT-5.6 Sol retain what it had learned during ARC-AGI-3 tasks.
OpenAI's ARC-AGI-3 analysis shows that agent performance can depend heavily on the execution environment around a model. For GPT-5.6 Sol, retained reasoning and compaction were associated with a substantially higher reported public-set score and lower output-token use. The result is a strong reminder that meaningful model evaluation requires clear reporting of the harness and API configuration, not just the model name.