{"slug": "openai-s-agi-number-came-from-a-harness-not-the-model", "title": "OpenAI's AGI number came from a harness, not the model", "summary": "OpenAI's GPT-6 Astra scored 99.9% on ARC-AGI-3 only when run through OpenAI's custom Provider Adapter harness, while the same model scored 62.7% under ARC Prize's standard harness, according to ARC Prize's published table. ARC Prize stated it is \"not claiming that it is AGI,\" and co-founder Mike Knoop wrote \"we lack evidence to call this AGI yet.\" OpenAI's launch post also showed five metrics altered after publication, including Astra's hallucination rate fluctuating between 4.2% and 2%.", "body_md": "OpenAI declared the AGI era on the strength of a 99.9% score. Run the same model through the benchmark’s own software, and it scores 62.7%.\n\nThe gap is not a rounding error or a rival’s complaint. It comes from the organisation that built the test. ARC Prize [published both numbers](https://arcprize.org/blog/astra) on the day GPT-6 Astra launched. It printed a full table of every reasoning level it ran.\n\nThe difference is the harness. A harness is the software around a model. It sets the tools the model can reach, what it remembers between requests, and how its context gets managed. Same weights, different scaffolding, different score.\n\n## What the table actually shows\n\nARC Prize ran Astra two ways. Its standard harness gives every model the same minimal interface and lets the model decide which notes to carry forward. OpenAI’s Provider Adapter preserves the model’s opaque reasoning state between requests and compacts longer conversations.\n\nUnder the standard harness at maximum reasoning, Astra scored 62.7% and cost $26,098. Under the Provider Adapter at high reasoning, it scored 99.9% for $18,817. The better score was also the cheaper run.\n\nOne row makes the point harder than any argument. Set the reasoning effort to none inside OpenAI’s adapter and Astra still scores 96.7%. That beats the same model at maximum reasoning inside the standard harness by 34 points. The scaffolding outperformed the reasoning dial outright.\n\nBoth harnesses solved 167 game-reasoning pairs. On those, ARC Prize clocked the adapter runs at 49% fewer tokens and roughly 3.66 times faster.\n\n## The number that travelled\n\nThe figure that spread was 99.9% against GPT-5.6 Sol’s 7.8%. Those are not the same test. Astra’s 99.9% came from the Provider Adapter; Sol’s 7.8% came from the standard harness.\n\nThat comparison ran widely, and it ran here. TNW’s own launch coverage carried [the 99.9% figure](https://thenextweb.com/news/gpt-6-astra-benchmarks-monitorability-cyber) against Sol’s 7.8% without the harness caveat. Our [follow-up on the AGI claim](https://thenextweb.com/news/openai-astra-agi-claim-cybersecurity-containment) described human parity on the same basis.\n\nThe like-for-like comparison is 62.7% against 7.8%. That remains an enormous jump, and it is the one the benchmark supports.\n\nARC Prize itself declined the conclusion OpenAI drew. It stated plainly that it is “not claiming that it is AGI”. Co-founder Mike Knoop wrote that “we lack evidence to call this AGI yet”. Going forward the foundation will publish both harness results side by side.\n\nThere is a small wrinkle inside ARC Prize too. François Chollet gave the standard-harness figure as 66% in a post, against 62.7% in the published table. The blog is the primary record.\n\n## Then the numbers moved\n\nFortune found that the scores kept changing after publication. Emily Forlini [compared archived snapshots](https://fortune.com/2026/09/04/openai-quietly-boosts-some-of-astras-evaluation-metrics-amid-rare-delay-in-publication-of-the-modeblog-post-announcement/) of OpenAI’s launch post and found five metrics altered.\n\nAstra’s hallucination rate read 4.2% in the first snapshot. By 5.20pm it read 2%. It has since gone back to 4.2%. Anthropic’s Fable 5.1 dropped nearly ten points on FrontierMath, from 87.8% to 78%, then settled at 83%.\n\nSol’s ExploitBench score doubled from 5.5% to 11.5%. OpenAI told Fortune it is investigating reverting that one. The 11.5% reflects a reasoning level Sol does not offer commercially.\n\nThe pre-publication draft sent to media under embargo put ARC-AGI-3 at 98.6%. The live post says 99.99%.\n\nNot every edit favoured Astra. Two Anthropic scores on HealthBench Professional went up. Other outlets reached the fine print independently, and The New Stack ran the two harness figures side by side the following day. But the pattern runs mostly one way. OpenAI also pulled the post after publishing it and put it back up, for reasons the company said it could not disclose.\n\nMost evaluations carry noise of a few percentage points, the company told Fortune, depending on checkpoint, scaffold and evaluation run.\n\n## Two Stanford researchers have a word for it\n\nAnka Reuel and Mike Hardy call the practice benchmaxxing, meaning re-running evaluations under different conditions until the number improves. Both work at Stanford, at the Intelligent Systems Laboratory and the Trustworthy AI Lab.\n\nThey also went looking in the system card, which is where the method should be documented. On the internal hallucination benchmark they found “barely any details about the evaluation”, adding that it “doesn’t even include the number of test items”.\n\nNot everyone reads it as gaming. Vincent Sunn Chen of Snorkel AI told Fortune that scores routinely shift in the final hours before a launch. Checkpoint, configuration, harness and grading are all still moving. His suggestion is a norm requiring companies to say what changed when they revise a published figure.\n\nThere is precedent for the harsher reading. Critics accused Meta in 2025 of publishing Llama 4 results from an internal build rather than the public one. Yann LeCun later said the company had fudged them.\n\n## Where Astra actually sits\n\n[Artificial Analysis](https://artificialanalysis.ai/articles/benchmarking-gpt-6-astra) ran its own tests and got a duller picture. On its Coding Agent Index, Astra scores 67 in Codex, level with Claude Opus 5 and Fable 5. Fable 5.1 leads at 70.\n\nOn its Intelligence Index, Astra scores 61, the same as the model it replaces, five points behind Fable 5.1 and behind Meta’s Muse Spark 1.3. It costs $10 per million input tokens and $50 per output, two and a half times Sol’s price, which works out 75% more expensive per task at maximum effort.\n\nThe gains are real but narrow. Hallucination on its knowledge benchmark fell from 92% to 51%, and long-horizon knowledge work rose around 80 Elo. An economically-weighted task benchmark fell by roughly the same amount, with further regressions on banking support, scientific Python and long-context reasoning.\n\nArtificial Analysis has since rebuilt the index. Version 4.2 arrived the following day with harder tasks and more private test sets, which the firm says exist to prevent gaming.\n\n## The harness is the product now\n\nWe have run into this word twice already. A cheap harness collapsed Booz Allen’s own [league table of AI threat](https://thenextweb.com/news/booz-allen-cyber-weapon-index-attack-harness). CrowdStrike chose the same word for what it [wraps around OpenAI](https://thenextweb.com/news/crowdstrike-openai-gpt-5-6-cyber-anthropic-claude-marketplace)’s cyber model.\n\nAnthropic, Google and Microsoft all sell harnesses as products with their own pricing. Nvidia built one that took Claude Opus 5 from 30.2% on ARC-AGI-3 to clearing every level.\n\nOpenAI is not hiding anything here. The adapter uses documented API features any developer can call. But the thing that scored 99.9% is an assembled system, and the thing on sale is a model.\n\nThat distinction is the whole story, and it also runs through [the monitorability argument](https://thenextweb.com/news/astra-chain-of-thought-monitorability-debate-greenblatt-pachocki-2025-position-paper-gpai-code-model-report-ai-office). What a model can do and what anyone can independently verify it doing are drifting apart, and the benchmark is where you can see the gap.\n\n## Get the TNW newsletter\n\nGet the most important tech news in your inbox each week.", "url": "https://wpnews.pro/news/openai-s-agi-number-came-from-a-harness-not-the-model", "canonical_source": "https://thenextweb.com/news/openai-astra-arc-agi-3-harness-62-7-vs-99-9-benchmark-revisions", "published_at": "2026-09-09 03:17:24+00:00", "updated_at": "2026-09-09 03:50:48.371575+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-research", "ai-ethics"], "entities": ["OpenAI", "ARC Prize", "GPT-6 Astra", "Mike Knoop", "François Chollet", "Fortune", "Anthropic", "Fable 5.1"], "alternates": {"html": "https://wpnews.pro/news/openai-s-agi-number-came-from-a-harness-not-the-model", "markdown": "https://wpnews.pro/news/openai-s-agi-number-came-from-a-harness-not-the-model.md", "text": "https://wpnews.pro/news/openai-s-agi-number-came-from-a-harness-not-the-model.txt", "jsonld": "https://wpnews.pro/news/openai-s-agi-number-came-from-a-harness-not-the-model.jsonld"}}