OpenAI is urging researchers, evaluators, and AI buyers to treat benchmark results as measurements of a model in a particular setup, not as permanent measures of a model's standalone capability. Its new guidance on third-party evaluations says the harness, resource budget, available tools, and approach to memory and context management can materially change observed performance, especially on long-running, multi-step tasks.
In OpenAI's guidance on trustworthy third-party evaluations, the company calls for clearer reporting about the conditions behind a score. The central message is straightforward: when a model's results improve with more compute, better tool access, or a more capable agent harness, the finding should be described as performance under that harness and budget. It should not be presented as a fixed capability ceiling.
That distinction matters as frontier AI systems are increasingly evaluated as agents rather than as single-turn chat systems. An agent may need to retain useful information, choose and use tools, recover from failed steps, and manage its context across an extended task. In those cases, evaluation design is not a minor implementation detail. It is part of what determines the result.
A harness is the operational setup used to run an evaluation. In practice, it can determine how a model receives a task, what tools it can use, how many steps or attempts it receives, whether it can preserve state, and how its output is assessed. OpenAI's guidance places the harness alongside budgets and tools as core variables that evaluators should disclose.
This does not make benchmark results unhelpful. It makes them more interpretable. A score can offer strong evidence about performance in a specified environment. But comparisons become less reliable when two models are tested with materially different prompting, tools, time or compute allowances, memory behavior, or safety constraints.
OpenAI frames this reporting problem through a taxonomy of evaluation claims. It distinguishes among strong elicitation, controlled comparison, and safeguarding robustness. Each claim requires different evidence. An evaluation designed to elicit the strongest possible performance from one system is not automatically a controlled comparison between systems. Likewise, a result about whether safeguards hold up under adversarial conditions answers a different question from a capability benchmark.
| Evaluation element | What an isolated score can obscure | What OpenAI says should be reported |
|---|---|---|
| Harness | How the model was run and guided through the task | The harness configuration used for the evaluation |
| Budget | Whether added compute or task resources improved performance | The budget under which the result was obtained |
| Tools and memory | Whether the system could use tools or preserve task-relevant state | The available tools and context or memory approach |
| Safeguards and evidence | Whether constraints or incomplete evidence affected the outcome | Safeguards and evidence supporting the evaluation claim |
The practical consequence is that benchmark reporting should move beyond a leaderboard-style number. Readers need enough information to understand what was tested, how it was elicited, and what conclusion the measurement can legitimately support.
OpenAI points to GPT-5.5 cyber-range evaluations as a concrete illustration of the issue. The company says performance improved when the evaluation harness used compaction to preserve task-relevant context across turns. For a long-horizon task, that capability can help an agent build on information it has already gathered instead of repeatedly losing important state as the interaction progresses.
The example highlights why memory strategy is a meaningful evaluation variable for persistent AI systems. A model operating over many steps may have to track prior findings, tool outputs, decisions, and unresolved work. If its setup does not retain useful context effectively, an observed failure may reflect the surrounding system as much as the underlying model.
This is particularly relevant to cyber-range and other multi-step evaluations, where success may depend on sequencing actions over time rather than producing one correct answer. It also affects deployment planning. Organizations assessing an agent for extended workflows should test the full operating design, including context handling and tool use, rather than extrapolating solely from a published model score.
For teams building persistent AI workflows, the evaluation question is therefore twofold: which model is being used, and what system enables that model to perform the work over time? Organizations evaluating those choices can work with Scalevise on AI architecture, workflow automation, and implementation designed around their actual operating constraints. OpenAI's guidance is also a call for greater standardization in third-party evaluations. The company identifies several hazards that can distort an apparent result, including reward hacking, contamination, and refusals. These issues can make a high score misleading, or make a lower score difficult to interpret, unless the evaluation design and evidence are transparent.
A useful evaluation report should make clear:
This level of disclosure helps different audiences ask better questions. Researchers can judge whether a result supports a controlled comparison. Developers can identify the operational conditions that may be needed to reproduce performance. Enterprise buyers can distinguish a model's potential from the cost and engineering work required to achieve it in production.
It also provides a more disciplined way to discuss costs. If an agent performs better with a larger budget or a more sophisticated harness, that improvement may be valuable, but it is not free or automatic. Deployment decisions should account for the resources, tools, and context-management design used in the relevant evaluation, rather than treating the headline score as a direct forecast of production performance.
Why does OpenAI say AI benchmark scores depend on the harness?
The harness controls how a model is run during an evaluation, including elements such as task setup, tools, budgets, and context management. Those choices can materially affect observed performance.
What does performance under a harness and budget mean?
It means a result describes how a model performed with a specific evaluation setup and resource allocation. It should not be treated as a fixed measure of the model's capability in every environment.
How did compaction affect GPT-5.5 cyber-range evaluations?
OpenAI says GPT-5.5 performed better when the evaluation harness used compaction to preserve task-relevant context across turns.
What should third-party AI evaluations disclose?
OpenAI recommends reporting the harness, budget, tools, safeguards, and evidence behind the evaluation so readers can interpret and compare the result appropriately.
OpenAI's guidance reframes benchmark interpretation around the full evaluation environment. For long-running agents in particular, memory, tools, budgets, and harness design can shape the result as decisively as the model itself. More transparent reporting will make third-party evaluations more useful for researchers, developers, and organizations deciding how AI systems may perform in real workflows.