Google seems to have made a big comeback in the AI race with Gemini 4 Carbon.
Artificial Analysis has refreshed AA-Omniscience, its benchmark for measuring how reliable AI models are with knowledge and how often they hallucinate, to version 1.1, and Google’s recently released Gemini 4 Carbon has come out on top by a wide margin.
The update was rolled out as part of the broader Artificial Analysis Intelligence Index v5 release. AA-Omniscience’s core idea remains the same: it tests whether models give correct answers, penalizes them for confidently wrong ones, and — crucially — doesn’t punish a model for simply admitting it doesn’t know. That last part is measured by Net Score, the benchmark’s primary metric, which ranges from -100 to 100. A score of zero means a model gets as many answers right as wrong, so moving into positive territory signals genuinely reliable knowledge rather than lucky guessing.
Gemini 4 Carbon’s Lead #
On the updated benchmark, Gemini 4 Carbon (high) leads with a Net Score of 54, a sizeable 6 points clear of second-placed Claude Opus 5.5 (48, max with fallback) and 8 points ahead of Claude Fable 5.1 (46, max with fallback). That gap matters because Claude Opus 5.5 had itself opened up one of the biggest leads in months when it originally took the top spot on Artificial Analysis’s Intelligence Index, with AA-Omniscience listed among the evaluations it led on at the time.
GPT-6 Astra (max) and GPT-6.1 Sol (max) are tied for fourth at 42 each, followed by Grok 4.7 (xhigh) and Meta’s Muse Wonder Full (max) at 31. Among open-weights models, Muse Spark Open (xhigh) leads the pack at 26, putting it 28 points behind Gemini 4 Carbon at the top — a reminder of how much distance still separates proprietary frontier models from openly available ones on knowledge reliability specifically, even as that gap narrows on other benchmarks.
Further down the chart, scores fall off sharply. GLM-5.3 (max) sits at 6, Mistral Large 4 (high) at 5, and a cluster of models — including DeepSeek V4.1 Flash, Nemotron 3.5 Super VL, MiMo-V2.6-Pro, K2 Horizon 375B A23B, Qwen3.8 27B, MiniMax-M3.1-Flash, and GPT-6 Luna (max) — post negative Net Scores, meaning they get more answers wrong than right once hallucinations are factored in.
What Changed In v1.1 #
Version 1.1 isn’t just a re-run of the same test. Artificial Analysis trimmed the dataset from 6,000 questions down to 3,000, stripping out questions that had either become public and saturated (meaning models could have been trained on them) or that contained errors and ambiguities. New questions were added to backfill the set, which still spans the same six domains and 42 topics as before — Science, Engineering and Mathematics; Software Engineering; Finance; Health; Humanities and Social Sciences; and Law — though the distribution across those domains has shifted slightly.
Artificial Analysis also released a newly calibrated 300-question public set on Hugging Face, designed to match the difficulty of the private evaluation set. At launch, the public and private scores correlate at 0.99 on both Pearson and Spearman measures for Net Score, with a median difference of exactly 0.0 points — meaning anyone running the public set should get a very close read on how a model would score on the full private benchmark.
Scores Shift, But Rankings Mostly Hold #
Because v1.1 removed questions models may have already seen, most models score lower than they did on the original version — by 1.9 points on average, with a median drop of 0.6 points. Artificial Analysis is explicit that this reflects the harder, cleaner dataset rather than any actual regression in model capability.
Despite the overall dip, the rank order between v1 and v1.1 is largely preserved, with a 0.98 Spearman rank correlation between the two versions. A handful of models did see much steeper declines, however. MiniMax M3.1 Flash fell 22 points, from 16 down to -6, while MiMo V2.6 Pro dropped 13 points, from 8 to -5 — both swinging from clearly positive to negative Net Scores.
Accuracy, which measures the raw percentage of questions answered correctly, also slipped slightly, down a median of 1.5 percentage points across models. Claude Fable 5.1 (max with fallback) posted the highest accuracy on the updated set at 68%, with Claude Opus 5.5 (max with fallback) just behind at 66%. On the flip side, hallucination rates — how often a model guesses rather than admits uncertainty — actually improved slightly, with the median Hallucination Rate falling 1.4 percentage points. Gemini 4 Argon (high), the first model in Google’s Gemini 4 family released ahead of Carbon, recorded the lowest hallucination rate of any model at just 4%.
Taken together, AA-Omniscience v1.1 suggests that while the leaderboard reshuffling at the very top continues at pace — with a new model claiming the lead roughly every few weeks — the underlying gap in raw knowledge reliability between frontier labs and the rest of the field remains wide, and open-weights models in particular still have real ground to make up.