{"slug": "model-equality-testing-which-model-is-this-api-serving", "title": "Model Equality Testing: Which Model Is This API Serving?", "summary": "A statistical test built on Maximum Mean Discrepancy with a simple string kernel detected output-distribution changes in 11 of 31 commercial inference API endpoints serving four Llama models, according to an arXiv paper by Irena Gao and coauthors (v2 revised 8 April 2025). The test, which the authors call Model Equality Testing, achieved a median of 77.4% power against a range of distortions including quantization, watermarking and finetuning, using an average of just 10 samples per prompt. The work matters because API providers can alter the underlying model without notifying users, and the method gives users a way to check whether an endpoint serves the reference weights Meta released.", "body_md": "# Computer Science > Machine Learning\n\n  [Submitted on 26 Oct 2024 (\n\n[v1](https://arxiv.org/abs/2410.20247v1)), last revised 8 Apr 2025 (this version, v2)]\n# Title:Model Equality Testing: Which Model Is This API Serving?\n\n[View PDF](https://arxiv.org/pdf/2410.20247)\n\n[HTML (experimental)](https://arxiv.org/html/2410.20247v2)\n\nAbstract:Users often interact with large language models through black-box inference APIs, both for closed- and open-weight models (e.g., Llama models are popularly accessed via Amazon Bedrock and Azure AI Studio). In order to cut costs or add functionality, API providers may quantize, watermark, or finetune the underlying model, changing the output distribution -- possibly without notifying users. We formalize detecting such distortions as Model Equality Testing, a two-sample testing problem, where the user collects samples from the API and a reference distribution and conducts a statistical test to see if the two distributions are the same. We find that tests based on the Maximum Mean Discrepancy between distributions are powerful for this task: a test built on a simple string kernel achieves a median of 77.4% power against a range of distortions, using an average of just 10 samples per prompt. We then apply this test to commercial inference APIs from Summer 2024 for four Llama models, finding that 11 out of 31 endpoints serve different distributions than reference weights released by Meta.\n    \n\n## Submission history\n\nFrom: Irena Gao [\n[view email](https://arxiv.org/show-email/2476b84d/2410.20247)]\n\n**Sat, 26 Oct 2024 18:34:53 UTC (863 KB)**\n\n[\\[v1\\]](https://arxiv.org/abs/2410.20247v1)\n**[v2]** Tue, 8 Apr 2025 20:19:26 UTC (869 KB)\n\n### Additional Features\n\n### References & Citations\n\nLoading...\n\n# Bibliographic and Citation Tools\n\nBibliographic Explorer \n\n*(*[What is the Explorer?](https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer))\nConnected Papers \n\n*(*[What is Connected Papers?](https://www.connectedpapers.com/about))\nLitmaps \n\n*(*[What is Litmaps?](https://www.litmaps.co/))\nscite Smart Citations \n\n*(*[What are Smart Citations?](https://www.scite.ai/))\n# Code, Data and Media Associated with this Article\n\nalphaXiv \n\n*(*[What is alphaXiv?](https://alphaxiv.org/))\nCatalyzeX Code Finder for Papers \n\n*(*[What is CatalyzeX?](https://www.catalyzex.com))\nDagsHub \n\n*(*[What is DagsHub?](https://dagshub.com/))\nGotit.pub \n\n*(*[What is GotitPub?](http://gotit.pub/faq))\nHugging Face \n\n*(*[What is Huggingface?](https://huggingface.co/huggingface))\nScienceCast \n\n*(*[What is ScienceCast?](https://sciencecast.org/welcome))\n# Demos\n\n# Recommenders and Search Tools\n\nInfluence Flower \n\n*(*[What are Influence Flowers?](https://influencemap.cmlab.dev/))\nCORE Recommender \n\n*(*[What is CORE?](https://core.ac.uk/services/recommender))\nIArxiv Recommender\n\n*(*[What is IArxiv?](https://iarxiv.org/about))\n# arXivLabs: experimental projects with community collaborators\n\narXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.\n\nBoth individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.\n\nHave an idea for a project that will add value for arXiv's community? [**Learn more about arXivLabs**](https://info.arxiv.org/labs/index.html).", "url": "https://wpnews.pro/news/model-equality-testing-which-model-is-this-api-serving", "canonical_source": "https://arxiv.org/abs/2410.20247", "published_at": "2026-10-26 06:14:16+00:00", "updated_at": "2026-10-26 06:44:09.155940+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-research", "ai-infrastructure"], "entities": ["Irena Gao", "Meta", "Llama", "Amazon Bedrock", "Azure AI Studio", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/model-equality-testing-which-model-is-this-api-serving", "markdown": "https://wpnews.pro/news/model-equality-testing-which-model-is-this-api-serving.md", "text": "https://wpnews.pro/news/model-equality-testing-which-model-is-this-api-serving.txt", "jsonld": "https://wpnews.pro/news/model-equality-testing-which-model-is-this-api-serving.jsonld"}}