{"slug": "a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic", "title": "A new way of measuring AI models suggests China's threat to OpenAI and Anthropic may have been overblown", "summary": "Artificial Analysis's overhauled Intelligence Index v5 widened the gap between the best open-weight AI models and the overall leader from 3 points to 12 points, with open-weight leaders scoring 32 against 58 for Anthropic's Claude Opus 5.5. The firm made many of its tests private and changed the tasks and their weighting to better reflect real-world use, a move Snowflake CEO Sridhar Ramaswamy cited on X: \"The day benchmarks went private, the open-model gap went from 3 points to 12.\" The results suggest Chinese open-weight models such as Moonshot's Kimi K3.1 Pro trail US closed models by more than previously measured, potentially strengthening the investment case for Anthropic's planned IPO and a possible OpenAI listing.", "body_md": "# A new way of measuring AI models suggests China's threat to OpenAI and Anthropic may have been overblown\n\n[Business Insider](https://www.businessinsider.com)\n\nChinese AI models fell behind US rivals after Artificial Analysis revamped its tests, raising questions about open-weight AI's performance.\n\n- Artificial Analysis's revamped ranking widened the gap between top closed and open-weight AI models.\n- Many tests are now private, but the firm also changed the tasks and how it weights them.\n- Open-weight AI remains competitive in coding. Usage figures offer only a limited view of demand.\n\n[Chinese AI models](https://www.businessinsider.com/chinese-ai-models-manus-deepseek-alibaba-tencent-2025-3) have fallen sharply behind American rivals in a closely watched ranking, raising fresh questions about whether the threat from cheaper, freely available [AI](https://www.businessinsider.com/biggest-winner-from-cheap-chinese-ai-silicon-valley-2026-7) technology has been exaggerated.\n\nArtificial Analysis, a prominent independent firm that measures AI performance, [recently overhauled its testing system](https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5) to better reflect how people actually use the technology.\n\nOne crucial change: Many of the tests are now kept secret, making it harder for AI companies to train their models specifically to perform well on them.\n\nThe results were striking.\n\nArtificial Analysis said the best so-called open-weight AI models had trailed the overall leader by just three points under the previous system. Under the new system, that gap widened to 12 points.\n\nThe 12-point gap included [Moonshot](https://www.businessinsider.com/china-ai-boom-terrible-business-open-weight-models-2026-7)'s Kimi K3.1 Pro. Among models with publicly released weights, the leaders scored just 32 points, compared with 58 for the then overall leader, [Anthropic](https://www.businessinsider.com/anthropic-claude-internal-ai-use-exceeds-employee-pay-tokens-2026-11)'s [Claude](https://www.machinebrief.com/glossary/claude) Opus 5.5.\n\nThe findings could have major implications for the AI industry and the billions of dollars investors are pouring into it.\n\nChinese companies have been releasing powerful AI models that other businesses can download and adapt, often at lower costs than American alternatives. This fueled concern that companies such as [OpenAI](https://www.businessinsider.com/chatgpt-helped-fix-home-internet-ethernet-cable-zoom) and [Anthropic](https://www.machinebrief.com/glossary/anthropic) could struggle to charge premium prices for their proprietary technology.\n\nThe new rankings suggest that threat may have been overstated, potentially strengthening the investment case for Anthropic's planned IPO and a possible future OpenAI listing.\n\nSridhar Ramaswamy, CEO of Snowflake and a former Google executive, highlighted the significance of the testing change.\n\n\"The day benchmarks went private, the open-model gap went from 3 points to 12,\" [he wrote on X](https://x.com/RamaswmySridhar/status/2115421134929981739). \"Were open models catching up, or was the test leaking?\"\n\n## \"[Benchmark](https://www.machinebrief.com/glossary/benchmark) hacking\"\n\nHis point: When AI companies know exactly how their technology will be tested, they can train it to excel at those particular tasks. That doesn't necessarily mean their models perform equally well in the real world.\n\nMicrosoft CEO Satya Nadella and other AI experts have called this \"benchmark hacking.\" Keeping the tests secret makes these hacks harder.\n\nThe new results suggest American companies have a bigger lead than previously believed, particularly in complex scientific tasks. Even Claude Haiku 5.5, a smaller Anthropic model, outperformed every AI model with publicly released weights.\n\nEven proponents of open-weight models admitted something has likely been awry.\n\n\"I do think the previous index was flattering the open weight models too strongly,\" said [Nathan Lambert](https://www.interconnects.ai/p/artificial-analysis-v5-and-the-benchmarkers), cofounder of Trillium Labs, which researches large language models, especially open-weight models.\n\n## Smaller usage\n\nThere's another reason to question the threat from open-weight AI models: How much are people actually using them?\n\nDax Raad, cofounder of software development company OpenCode, [recently examined](https://x.com/thdxr/status/2115165365630026178) traffic through OpenRouter, a popular service that lets developers access AI models from different providers.\n\nHe estimated that almost all the previous day's DeepSeek traffic flowing through OpenRouter could be handled by a single rack of powerful [Nvidia](https://www.machinebrief.com/glossary/nvidia) computers, costing roughly $6 million to $7 million.\n\nThat's a surprisingly small amount of computing power given the enormous [attention](https://www.machinebrief.com/glossary/attention) Chinese AI models have attracted.\n\nOpenRouter represents only a slice of the market, so its figures don't capture all demand for open-weight models.\n\nBut Raad's broader point is important: Much of the industry's enormous AI infrastructure investment is being driven by major providers such as OpenAI and Anthropic, rather than the smaller developers using open models.\n\nConsider the more than 1 billion people who use ChatGPT every month. These folks are using OpenAI's closed AI models all the time — potentially way more activity than is captured by AI gateway services such as OpenRouter.\n\n## Two questions\n\nTogether, the rankings and usage figures raise two questions about open-weight AI: Is it really as good as the leading American technology, and is demand as substantial as the hype suggests?\n\nThere are important caveats.\n\nTrillium's Lambert said open-weight models remain highly competitive in areas such as software development, and can offer businesses greater control and potentially lower costs.\n\nStill, the new evidence challenges a powerful narrative that has shaped the industry: that Chinese AI companies are rapidly catching America's leaders and threatening their business models.\n\nFor investors weighing the enormous valuations of OpenAI and Anthropic, that distinction matters enormously.\n\n*Sign up for the AI Insider newsletter* [here](https://www.businessinsider.com/subscription/newsletter/tech-memo)*. Reach out to me via email at* [abarr@businessinsider.com](mailto:abarr@businessinsider.com)\n\n[Business Insider](https://www.businessinsider.com/chinese-ai-models-rankings-artificial-analysis-private-tests-anthropic-openai-2026-11)\n\nGet AI news in your inbox\n\nDaily digest of what matters in AI.", "url": "https://wpnews.pro/news/a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic", "canonical_source": "https://www.machinebrief.com/news/a-new-way-of-measuring-ai-models-suggests-chinas-threat-to-o-u9kk", "published_at": "2026-11-03 10:00:02+00:00", "updated_at": "2026-11-03 11:21:54.568486+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-startups"], "entities": ["Artificial Analysis", "Anthropic", "Claude Opus 5.5", "Claude Haiku 5.5", "Moonshot", "Kimi K3.1 Pro", "OpenAI", "Sridhar Ramaswamy"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic", "markdown": "https://wpnews.pro/news/a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic.md", "text": "https://wpnews.pro/news/a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic.txt", "jsonld": "https://wpnews.pro/news/a-new-way-of-measuring-ai-models-suggests-china-s-threat-to-openai-and-anthropic.jsonld"}}