{"slug": "gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible", "title": "GPT-Rankings: a public directory of 150 GPT API gateways — feedback on reproducible evaluations", "summary": "A contributor to the GPT-Rankings project is seeking feedback on how to make its evaluations of 150 third-party GPT API gateways reproducible, as of November 17, 2026. The directory records advertised price multipliers and project-reported test pass rates, but its README publishes neither a full test suite nor per-run traces, making it hard to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. A commenter, nootxlm, recommended interleaving the same prompt across every gateway within one short window, publishing the synthetic prompts themselves rather than only their hashes, and adding a hash of each raw response so published traces can be checked against scored runs.", "body_md": "[lmdk321](https://discuss.huggingface.co/u/lmdk321)\n1\n \nI’m helping share GPT-Rankings and collect feedback on how to make its evaluations reproducible. The repository indexes 150 third-party GPT API gateways as of November 17, 2026. It records advertised price multipliers and project-reported test pass rates. It is a directory, not an official provider endorsement or a way to authenticate which model served a request.\n\nThe current README describes evaluation dimensions but does not publish a full test suite or per-run traces. That makes it difficult to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. I’d like feedback on a minimal record that another developer could independently repeat:\n\n- **Test setup:** UTC timestamp, task ID and prompt hash, advertised model ID, request settings, number of trials, and whether streaming was used.\n- **Observations:** task result against a published rubric, finish reason, latency, retry count, and reported usage or billing. Keep these separate from any inference about model identity.\n- **Privacy:** use synthetic prompts and redact credentials, personal data, internal endpoints, and request IDs before sharing logs.\n\nWhat would you add to make a single gateway comparison independently auditable? Would you report repeatability across time windows separately from accuracy?\n\nRepository and methodology summary: [https://github.com/lmdk321321/GPT-Rankings](https://github.com/lmdk321321/GPT-Rankings)\n\n \n \n[nootxlm](https://discuss.huggingface.co/u/nootxlm)\n2\n \nYes to reporting repeatability separately — and pairing is what makes either number auditable: run the same prompt on every gateway you are comparing inside one short window, interleaved, rather than one gateway per window, so drift and load land on all of them alike. Since the prompts are synthetic, publish the prompts themselves, not just their hashes — a hash confirms a prompt a reader already has, but nobody can re-run a comparison from hashes alone. Add a hash of each raw response to the record, and a stranger can check that the published traces match the scored runs without seeing any credentials.", "url": "https://wpnews.pro/news/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible", "canonical_source": "https://discuss.huggingface.co/t/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible-evaluations/192020#post_2", "published_at": "2026-11-18 08:18:11+00:00", "updated_at": "2026-11-18 08:43:33.051553+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "ai-products"], "entities": ["GPT-Rankings", "lmdk321", "nootxlm", "Hugging Face", "GitHub"], "also_reported_by": [], "takeaway": "To make gateway comparisons auditable, interleave the same prompt across all gateways in one short window, publish the synthetic prompts themselves rather than only hashes, and record a hash of each raw response alongside latency, finish reason, retry count and reported usage.", "recommendation": {"advice": "Interleave identical prompts across every gateway in one short window and publish the prompts plus response hashes, so drift and load hit all candidates alike and traces can be verified.", "playbook": "evals-testing", "playbook_url": "https://wpnews.pro/playbooks/evals-testing"}, "alternates": {"html": "https://wpnews.pro/news/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible", "markdown": "https://wpnews.pro/news/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible.md", "text": "https://wpnews.pro/news/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible.txt", "jsonld": "https://wpnews.pro/news/gpt-rankings-a-public-directory-of-150-gpt-api-gateways-feedback-on-reproducible.jsonld"}}