Business Insider Chinese AI models fell behind US rivals after Artificial Analysis revamped its tests, raising questions about open-weight AI's performance.
- Artificial Analysis's revamped ranking widened the gap between top closed and open-weight AI models.
- Many tests are now private, but the firm also changed the tasks and how it weights them.
- Open-weight AI remains competitive in coding. Usage figures offer only a limited view of demand.
Chinese AI models have fallen sharply behind American rivals in a closely watched ranking, raising fresh questions about whether the threat from cheaper, freely available AI technology has been exaggerated.
Artificial Analysis, a prominent independent firm that measures AI performance, recently overhauled its testing system to better reflect how people actually use the technology.
One crucial change: Many of the tests are now kept secret, making it harder for AI companies to train their models specifically to perform well on them.
The results were striking.
Artificial Analysis said the best so-called open-weight AI models had trailed the overall leader by just three points under the previous system. Under the new system, that gap widened to 12 points.
The 12-point gap included Moonshot's Kimi K3.1 Pro. Among models with publicly released weights, the leaders scored just 32 points, compared with 58 for the then overall leader, Anthropic's Claude Opus 5.5.
The findings could have major implications for the AI industry and the billions of dollars investors are pouring into it.
Chinese companies have been releasing powerful AI models that other businesses can download and adapt, often at lower costs than American alternatives. This fueled concern that companies such as OpenAI and Anthropic could struggle to charge premium prices for their proprietary technology.
The new rankings suggest that threat may have been overstated, potentially strengthening the investment case for Anthropic's planned IPO and a possible future OpenAI listing.
Sridhar Ramaswamy, CEO of Snowflake and a former Google executive, highlighted the significance of the testing change.
"The day benchmarks went private, the open-model gap went from 3 points to 12," he wrote on X. "Were open models catching up, or was the test leaking?"
"Benchmark hacking" #
His point: When AI companies know exactly how their technology will be tested, they can train it to excel at those particular tasks. That doesn't necessarily mean their models perform equally well in the real world.
Microsoft CEO Satya Nadella and other AI experts have called this "benchmark hacking." Keeping the tests secret makes these hacks harder.
The new results suggest American companies have a bigger lead than previously believed, particularly in complex scientific tasks. Even Claude Haiku 5.5, a smaller Anthropic model, outperformed every AI model with publicly released weights.
Even proponents of open-weight models admitted something has likely been awry.
"I do think the previous index was flattering the open weight models too strongly," said Nathan Lambert, cofounder of Trillium Labs, which researches large language models, especially open-weight models.
Smaller usage #
There's another reason to question the threat from open-weight AI models: How much are people actually using them?
Dax Raad, cofounder of software development company OpenCode, recently examined traffic through OpenRouter, a popular service that lets developers access AI models from different providers.
He estimated that almost all the previous day's DeepSeek traffic flowing through OpenRouter could be handled by a single rack of powerful Nvidia computers, costing roughly $6 million to $7 million.
That's a surprisingly small amount of computing power given the enormous attention Chinese AI models have attracted.
OpenRouter represents only a slice of the market, so its figures don't capture all demand for open-weight models.
But Raad's broader point is important: Much of the industry's enormous AI infrastructure investment is being driven by major providers such as OpenAI and Anthropic, rather than the smaller developers using open models.
Consider the more than 1 billion people who use ChatGPT every month. These folks are using OpenAI's closed AI models all the time — potentially way more activity than is captured by AI gateway services such as OpenRouter.
Two questions #
Together, the rankings and usage figures raise two questions about open-weight AI: Is it really as good as the leading American technology, and is demand as substantial as the hype suggests?
There are important caveats.
Trillium's Lambert said open-weight models remain highly competitive in areas such as software development, and can offer businesses greater control and potentially lower costs.
Still, the new evidence challenges a powerful narrative that has shaped the industry: that Chinese AI companies are rapidly catching America's leaders and threatening their business models.
For investors weighing the enormous valuations of OpenAI and Anthropic, that distinction matters enormously. Sign up for the AI Insider newsletter here. Reach out to me via email at abarr@businessinsider.com
Business Insider Get AI news in your inbox
Daily digest of what matters in AI.