# ARC Prize rejects OpenAI's claim that Astra scored 99.9 percent on AGI test > Source: > Published: 2026-10-24 21:08:48+00:00 *ARC Prize built a benchmark specifically to resist gaming. Then OpenAI's GPT-6 Astra scored 99.9 percent on it, and the people who run the test said that number doesn't count.* A benchmark designed to be gaming-proof just got a 99.9 percent score anyway. OpenAI touted that figure for GPT-6 Astra when it released the model on September 3, and Nvidia CEO Jensen Huang cited it three days later when he declared on X that "AGI has arrived," naming the OpenAI team in his praise. The post went up late on September 6 and had cleared 10 million views by Monday morning, per TheNextWeb. But when the ARC Prize Foundation ran Astra through the same neutral harness it uses on every other model, ARC-AGI-3, the score came back at 62.7 percent, still the highest result on the benchmark, but nowhere near what OpenAI and Huang put in front of the world. The 99.9 percent score came from somewhere else entirely. ARC Prize found that OpenAI ran Astra through its own custom "tools-enabled" harness, which lets the model preserve reasoning state across actions in a way the standard test setup does not, and that configuration cost nearly $19,000 to run, according to TheNextWeb's reporting on the benchmark revisions. ARC Prize's verdict was blunt: that score is *not valid*, because it wasn't produced under the conditions every other lab's model has to clear. Run Astra through the same harness as GPT-5.6, Gemini 4 Argon, or Claude Opus 5, and the number drops by more than 37 points. This is the real ARC-AGI-3 story right now, and it's a sharper one than the benchmark's launch-day headline from back in March. When ARC Prize first released ARC-AGI-3 on March 25, 2026, at an event in San Francisco, frontier models were a joke next to humans: Gemini 3.1 Pro managed 0.37 percent, GPT-5.4 scored 0.26 percent, Claude Opus 4.6 came in at 0.25 percent, and Grok 4.2 scored zero. Humans cleared the same environments 100 percent of the time. That gap is what made ARC-AGI-3 famous. It's also seven months old, and it is not what's happening on the leaderboard today. What actually moved the needle wasn't a smarter model. It was better scaffolding around the same one. By early October, an open-sourced harness called Prime Agent had pushed public scores from roughly 30 percent, where Claude Opus 5 had been sitting alone, up past 95 percent on the same weight class of model, according to reporting on the benchmark's October revisions. Separately, a team at Nvidia took that same insight further: five Nvidia engineers reportedly got Claude Opus 5, the identical model that scores 30 percent by itself, to hit 100 percent just by rebuilding the software layer that pilots it through the game. The model didn't get smarter. The code wrapped around it got better at remembering what it had already tried. [Sam Altman and Dario Amodei Can't Agree on What AGI Even Means](https://startupfortune.com/sam-altman-and-dario-amodei-cant-agree-on-what-agi-even-means/) Bloomberg reports that Sam Altman now calls AGI a sloppy term while Dario Amodei has all but stopped using it, calling it powerful AI instead. Their timelines range from Anthropic's late 2026 to Altman's few thousand days, or roughly 2032, and neither shares a testable benchmark with the other. - [what does artificial general intelligence actually mean](https://startupfortune.com/sam-altman-and-dario-amodei-cant-agree-on-what-agi-even-means/) - [why AI leaders disagree on AGI definition](https://startupfortune.com/sam-altman-and-dario-amodei-cant-agree-on-what-agi-even-means/) That's the uncomfortable finding sitting underneath Huang's AGI claim. On ARC-AGI-3's own public leaderboard as of late October, GPT-6 Astra at its highest configuration leads with 62.7 percent, with lower-compute variants trailing at 59.3 and 54.8 percent. Those are ARC Prize's own verified numbers, not vendor marketing. They're a genuine leap from March's near-zero scores. But the gap between 62.7 and 99.9 is the whole argument: on a benchmark built explicitly to resist memorization and brute-force tricks, the harness around the model can swing the score by nearly 40 points without changing a single weight. ARC Prize revised five scoring metrics but still declined to call Astra's progress AGI. None of this makes ARC-AGI-3 irrelevant to the spending story. CoreWeave and Nvidia's AI infrastructure build-out is being justified, in part, on the idea that general intelligence is close enough to warrant hundred-billion-dollar capital commitments. A benchmark where the top verified, harness-neutral score is still under two-thirds and the 99.9 percent headline got rejected by the people who wrote the test is a specific, checkable reason to be skeptical of that timeline. Frankly, if the clearest path to a better score is a cleverer piece of orchestration code rather than a better model, that says something about what's actually scarce right now, and it isn't raw model intelligence. ARC Prize has said it expects scores to keep climbing through the rest of 2026 as more labs adopt harness techniques like Prime Agent's. Whether that counts as progress toward AGI, or just better bookkeeping around the same underlying model, is exactly the question Astra's two scores leave unresolved. **Also read:** [How to Calculate the Right AI Agent Error Budget Before You Ship](https://startupfortune.com/how-to-calculate-the-right-ai-agent-error-budget-before-you-ship/) • [The AI Boom Is Now Big Enough to Need Its Own Brokers](https://startupfortune.com/the-ai-boom-is-now-big-enough-to-need-its-own-brokers/) • [TypeSafe AI admits a leak exposed every customer's Jev model output](https://startupfortune.com/typesafe-ai-admits-a-leak-exposed-every-customers-jev-model-output/) *This article is posted in [AI News](https://startupfortune.com/category/ai/), check it out for more related stories.* ## Join the discussion [Open in the community →](https://startupfortune.com/community/) Almost there. Sign in and your reply posts straight away.