{"slug": "arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual", "title": "Arena tells LDS that only one AI provider is consistently getting more factual", "summary": "Arena has launched a factuality leaderboard ranking models on correctness of claims alongside human preference, built on more than 2 million labeled claims from real conversations. Arena told Lets Data Science that OpenAI is the only provider consistently improving factuality over an extended period, while most open-source models show declines when factual accuracy is weighted in.", "body_md": "# Arena tells LDS that only one AI provider is consistently getting more factual\n\nArena has launched a factuality leaderboard that ranks models on the correctness of their claims alongside human preference, built on more than **2 million labeled claims** from real conversations. In an email interview with Lets Data Science, Arena said that while human preference scores have climbed across nearly every provider, **OpenAI is the only provider also consistently improving factuality** over an extended period, and that most open-source models trend the other way once factual accuracy is weighted in.\n\nLarge language models have been getting steadily better at what people like. Whether they have been getting more accurate is a different question, and the leaderboards mostly could not answer it.\n\n[Arena](https://arena.ai) has now published a [factuality leaderboard](https://arena.ai/blog/factuality-in-arena/) that ranks models on the correctness of their claims alongside human preference. It rests on more than 2 million labeled claims taken from real conversations, 1.3 million from Text Arena and over 700,000 from Search Arena. Users can switch factuality on and watch the rankings move.\n\nThey do move. Arena answered six questions from Lets Data Science in an email interview, and the sharpest answer concerns what happens when factual accuracy is plotted against release dates.\n\n\"While human preference scores across nearly all providers have climbed over time, plotting pure factuality score against flagship model release date showed OpenAI is the only provider also consistently improving factuality over an extended period,\" Arena told Lets Data Science.\n\nA couple of other providers have improved with their most recent releases. The wider pattern runs the other way. \"Many other preferred models saw declines when including factuality, including a general trend with most open source models,\" Arena said.\n\n### What sits behind the score\n\n| Measure | Figure |\n|---|---|\n| Labeled claims behind the leaderboard | 2 million+ |\n| Claims from Text Arena | 1.3 million+ |\n| Claims from Search Arena | 700,000+ |\n| Default weight on factuality | 25% |\n| Default weight on human preference | 75% |\n\nThe split is deliberate rather than conservative. Weighted this way, Arena said, \"it is difficult for an inaccurate model to top the leaderboard, while the human-preference component prevents a model from succeeding simply by avoiding informative answers.\" A model cannot win by being scrupulously accurate and useless, and it cannot win by being fluent and wrong.\n\n### How a claim gets checked\n\nArena described the pipeline it runs for each factuality audit:\n\n- •Randomly sample real Text Arena and Search Arena battles\n- •Break each model response into individual, or atomic, claims\n- •Filter out subjective, ambiguous, or non-web-verifiable statements\n- •Use search agents to assess each remaining claim and assign a calibrated probability that it is true\n- •Average those claim-level truth probabilities across each response\n- •Convert the difference between the two averages into a factuality outcome for that battle\n- •Combine those outcomes with human preference votes in a composite Bradley-Terry model\n\nThe scope is narrower than the word hallucination usually implies, and Arena was explicit about that. The leaderboard measures \"the accuracy of objective, unambiguous claims about the world that can reasonably be verified using web sources.\" It does not attempt to catch every failure mode, \"such as incorrectly recalling something from an earlier conversation or claiming that a coding action was completed when it was not.\"\n\n### Why measure it now\n\nThe case Arena makes is that trust has outrun verification. \"People are increasingly trusting models to answer questions outside their own areas of knowledge, even though manually checking every claim is slow and labor-intensive,\" the company told Lets Data Science.\n\nThat leaves preference votes doing work they were never built for. Human ratings \"may reward responses that are well written or comprehensive without fully reflecting whether their claims are accurate.\" Arena's argument is that style had become a proxy for accuracy, and that the two needed separating.\n\n### What static benchmarks miss\n\nFactuality and hallucination benchmarks already exist, so the obvious question is what this adds. Arena's answer is that its claims are not written for the test.\n\nThe claims \"arise organically, while most static factuality benchmarks are artificially curated.\" Because the system decomposes complete responses rather than scoring short answers, it also catches errors buried inside long replies that read as authoritative. And because factuality is fused with human preference, a model \"cannot top the composite leaderboard merely by refusing to provide an informative answer or by producing a polished response containing inaccurate claims.\"\n\n### Whether a leaderboard like this can be gamed\n\nAny benchmark that becomes influential becomes a target. Asked how it avoids being overfit to, Arena pointed at its inputs.\n\n\"The Arena leaderboard is inherently resistant to gaming because we rely on fresh, organic user conversations rather than static test sets,\" the company said. \"On the factuality side, a model cannot simply 'overfit' to the leaderboard; it must consistently produce claims on unseen, unpredictable prompts that satisfy multiple independent, search-based verifiers.\"\n\nBecause those verifiers check claims against the live web rather than a fixed answer key, Arena argues it is \"effectively impossible for a model to game the system through memorization or predictable patterns.\"\n\n### Where it goes next\n\nArena expects factuality to move from a public ranking into the tooling enterprises actually buy. The company said it will keep \"integrating it into our enterprise evaluation products and embedding it into agentic workflows, as well as other evaluation methodology behind our public leaderboards.\"\n\nThat is the part worth watching for practitioners. Overall Arena rankings already shape which models teams trial. If factuality scores follow the same path, the gap between a model people enjoy using and a model that is right more often stops being an academic distinction and starts showing up in procurement.\n\nThe full methodology is in [Arena's factuality launch post](https://arena.ai/blog/factuality-in-arena/).\n\n## Key Points\n\n- 1Arena's factuality leaderboard is built on\n**more than 2 million labeled claims** drawn from real conversations, 1.3 million from Text Arena and over 700,000 from Search Arena, with factuality weighted at**25%** against**75%** for human preference. - 2Arena told Lets Data Science that\n**OpenAI is the only provider consistently improving factuality** over an extended period, while many preferred models, and most open-source models, lose ground once factual accuracy is counted. - 3The method breaks each response into\n**atomic claims**, discards anything subjective or not web-verifiable, and has search agents assign a calibrated probability of truth to what remains, which Arena argues makes the leaderboard resistant to memorization and gaming.\n\n## Scoring Rationale\n\nArena's factuality adjustment adds a large-scale accuracy signal to widely followed model rankings and can influence model evaluation and procurement. Its provider trends are useful but remain specific to Arena's traffic, verifier pipeline and weighting choices.\n\n## Sources\n\nOriginal reporting, with the public references used alongside it.\n\nLDS Exclusive\n\nReporting based on written answers given directly to Let's Data Science.\n\nPractice interview problems based on real data\n\n1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.\n\n[Try 250 free problems](/problems)", "url": "https://wpnews.pro/news/arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual", "canonical_source": "https://letsdatascience.com/news/arena-tells-lets-data-science-that-only-one-ai-provider-is-c-6a22338d", "published_at": "2026-07-27 21:43:59+00:00", "updated_at": "2026-07-28 19:27:36.203426+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["Arena", "OpenAI", "Lets Data Science", "Text Arena", "Search Arena"], "alternates": {"html": "https://wpnews.pro/news/arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual", "markdown": "https://wpnews.pro/news/arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual.md", "text": "https://wpnews.pro/news/arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual.txt", "jsonld": "https://wpnews.pro/news/arena-tells-lds-that-only-one-ai-provider-is-consistently-getting-more-factual.jsonld"}}