cd /news/ai-tools/gpt-rankings-a-public-directory-of-1… · home › topics › ai-tools › article
[ARTICLE · art-171651] src=discuss.huggingface.co ↗ pub= topic=ai-tools verified=true sentiment=· neutral

GPT-Rankings: a public directory of 150 GPT API gateways — feedback on reproducible evaluations

A contributor to the GPT-Rankings project is seeking feedback on how to make its evaluations of 150 third-party GPT API gateways reproducible, as of November 17, 2026. The directory records advertised price multipliers and project-reported test pass rates, but its README publishes neither a full test suite nor per-run traces, making it hard to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. A commenter, nootxlm, recommended interleaving the same prompt across every gateway within one short window, publishing the synthetic prompts themselves rather than only their hashes, and adding a hash of each raw response so published traces can be checked against scored runs.

recommendation

Interleave identical prompts across every gateway in one short window and publish the prompts plus response hashes, so drift and load hit all candidates alike and traces can be verified.

more in playbook: Evals and testing for LLM apps
read2 min views8 publishedNov 18, 2026
GPT-Rankings: a public directory of 150 GPT API gateways — feedback on reproducible evaluations
Image: Discuss (auto-discovered)

lmdk321 1

I’m helping share GPT-Rankings and collect feedback on how to make its evaluations reproducible. The repository indexes 150 third-party GPT API gateways as of November 17, 2026. It records advertised price multipliers and project-reported test pass rates. It is a directory, not an official provider endorsement or a way to authenticate which model served a request.

The current README describes evaluation dimensions but does not publish a full test suite or per-run traces. That makes it difficult to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. I’d like feedback on a minimal record that another developer could independently repeat:

  • Test setup: UTC timestamp, task ID and prompt hash, advertised model ID, request settings, number of trials, and whether streaming was used.
  • Observations: task result against a published rubric, finish reason, latency, retry count, and reported usage or billing. Keep these separate from any inference about model identity.
  • Privacy: use synthetic prompts and redact credentials, personal data, internal endpoints, and request IDs before sharing logs.

What would you add to make a single gateway comparison independently auditable? Would you report repeatability across time windows separately from accuracy?

Repository and methodology summary: [https://github.com/lmdk321321/GPT-Rankings](https://github.com/lmdk321321/GPT-Rankings)

 
 
[nootxlm](https://discuss.huggingface.co/u/nootxlm)

2

Yes to reporting repeatability separately — and pairing is what makes either number auditable: run the same prompt on every gateway you are comparing inside one short window, interleaved, rather than one gateway per window, so drift and load land on all of them alike. Since the prompts are synthetic, publish the prompts themselves, not just their hashes — a hash confirms a prompt a reader already has, but nobody can re-run a comparison from hashes alone. Add a hash of each raw response to the record, and a stranger can check that the published traces match the scored runs without seeing any credentials.

── more in #ai-tools 4 stories · sorted by recency
── more on @gpt-rankings 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/gpt-rankings-a-publi…] indexed:0 read:2min 2026-11-18 · —