lmdk321 1
I’m helping share GPT-Rankings and collect feedback on how to make its evaluations reproducible. The repository indexes 150 third-party GPT API gateways as of November 17, 2026. It records advertised price multipliers and project-reported test pass rates. It is a directory, not an official provider endorsement or a way to authenticate which model served a request.
The current README describes evaluation dimensions but does not publish a full test suite or per-run traces. That makes it difficult to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. I’d like feedback on a minimal record that another developer could independently repeat:
- Test setup: UTC timestamp, task ID and prompt hash, advertised model ID, request settings, number of trials, and whether streaming was used.
- Observations: task result against a published rubric, finish reason, latency, retry count, and reported usage or billing. Keep these separate from any inference about model identity.
- Privacy: use synthetic prompts and redact credentials, personal data, internal endpoints, and request IDs before sharing logs.
What would you add to make a single gateway comparison independently auditable? Would you report repeatability across time windows separately from accuracy?
Repository and methodology summary: [https://github.com/lmdk321321/GPT-Rankings](https://github.com/lmdk321321/GPT-Rankings)
[nootxlm](https://discuss.huggingface.co/u/nootxlm)
2
Yes to reporting repeatability separately — and pairing is what makes either number auditable: run the same prompt on every gateway you are comparing inside one short window, interleaved, rather than one gateway per window, so drift and load land on all of them alike. Since the prompts are synthetic, publish the prompts themselves, not just their hashes — a hash confirms a prompt a reader already has, but nobody can re-run a comparison from hashes alone. Add a hash of each raw response to the record, and a stranger can check that the published traces match the scored runs without seeing any credentials.