{"slug": "combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models", "title": "Combining LLMs Rarely Beats the Best Single Model, I tested 67 frontier models", "summary": "A new study testing 67 frontier language models from 21 providers found that combining multiple models rarely outperforms the single best model, with gains capped by a 'co-failure ceiling' where all models are wrong on the same query. The research, submitted to arXiv on June 25, 2026, shows that on open-ended mathematics and code tasks, the observed co-failure rate is 2.5 times higher than predicted by standard correlation metrics, and that improvements come from models failing on different questions rather than adding more models.", "body_md": "# Computer Science > Artificial Intelligence\n\n[Submitted on 25 Jun 2026]\n\n# Title:When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models\n\n[View PDF](/pdf/2606.27288)\n\n[HTML (experimental)](https://arxiv.org/html/2606.27288v1)\n\nAbstract:Multi-model LLM systems such as routing, voting, cascades, fusion, and mixture-of-agents are used to beat single-model accuracy. We show that their gain is capped by a quantity the field rarely reports. For any policy whose output is one member model answer, accuracy cannot exceed one minus beta, where beta is the rate at which every model is wrong on the same query. In contrast, the usual diagnostic, average pairwise error correlation rho, cannot identify beta: error laws with identical marginals and pairwise correlations can have different all-wrong rates. A Clopper-Pearson bound on beta gives a finite-sample certificate on the largest gain any router, vote, or cascade could deliver before training a router.\n\nAcross 67 models from 21 providers, a tetrachoric-calibrated single-factor model still underprices the all-wrong tail: on open-ended mathematics, observed beta is 0.052 versus 0.023 under the full 67-model Gaussian copula, about 2.5 times underpricing, with 90 percent CI 1.7 to 3.4 and k equals 17. The effect recurs on execution-graded code, where beta is 0.079. Re-asking the same GPQA-Diamond questions in free-response rather than multiple-choice form reopens the tail, with beta 0.127 and a five-judge panel with kappa 0.73 to 0.92, locating co-failure in answer format rather than subject. At matched quality, low-rho heterogeneous ensembles beat high-rho Self-MoA, but on checkable tasks in our pool, combining models rarely beats the single best model without a strong query-level routing signal. Gains come from models failing on different questions, not from adding more models.\n\n## Submission history\n\nFrom: Josef Liyanjun Chen [[view email](/show-email/e28d1956/2606.27288)]\n\n**[v1]** Thu, 25 Jun 2026 17:06:06 UTC (230 KB)\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer\n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers\n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps\n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations\n\n*(*[What are Smart Citations?](https://www.scite.ai/))# Code, Data and Media Associated with this Article\n\nalphaXiv\n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers\n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub\n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub\n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face\n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast\n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower\n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender\n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [ Learn more about arXivLabs](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models", "canonical_source": "https://arxiv.org/abs/2606.27288", "published_at": "2026-06-26 16:42:31+00:00", "updated_at": "2026-06-26 17:05:04.964685+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "ai-research"], "entities": ["arXiv", "GPQA-Diamond", "Self-MoA", "Josef Liyanjun Chen"], "alternates": {"html": "https://wpnews.pro/news/combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models", "markdown": "https://wpnews.pro/news/combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models.md", "text": "https://wpnews.pro/news/combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models.txt", "jsonld": "https://wpnews.pro/news/combining-llms-rarely-beats-the-best-single-model-i-tested-67-frontier-models.jsonld"}}