GPT-Rankings: a public directory of 150 GPT API gateways — feedback on reproducible evaluations A contributor to the GPT-Rankings project is seeking feedback on how to make its evaluations of 150 third-party GPT API gateways reproducible, as of November 17, 2026. The directory records advertised price multipliers and project-reported test pass rates, but its README publishes neither a full test suite nor per-run traces, making it hard to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. A commenter, nootxlm, recommended interleaving the same prompt across every gateway within one short window, publishing the synthetic prompts themselves rather than only their hashes, and adding a hash of each raw response so published traces can be checked against scored runs. lmdk321 https://discuss.huggingface.co/u/lmdk321 1 I’m helping share GPT-Rankings and collect feedback on how to make its evaluations reproducible. The repository indexes 150 third-party GPT API gateways as of November 17, 2026. It records advertised price multipliers and project-reported test pass rates. It is a directory, not an official provider endorsement or a way to authenticate which model served a request. The current README describes evaluation dimensions but does not publish a full test suite or per-run traces. That makes it difficult to tell whether a score reflects model behavior, gateway settings, truncation, or transient reliability problems. I’d like feedback on a minimal record that another developer could independently repeat: - Test setup: UTC timestamp, task ID and prompt hash, advertised model ID, request settings, number of trials, and whether streaming was used. - Observations: task result against a published rubric, finish reason, latency, retry count, and reported usage or billing. Keep these separate from any inference about model identity. - Privacy: use synthetic prompts and redact credentials, personal data, internal endpoints, and request IDs before sharing logs. What would you add to make a single gateway comparison independently auditable? Would you report repeatability across time windows separately from accuracy? Repository and methodology summary: https://github.com/lmdk321321/GPT-Rankings https://github.com/lmdk321321/GPT-Rankings nootxlm https://discuss.huggingface.co/u/nootxlm 2 Yes to reporting repeatability separately — and pairing is what makes either number auditable: run the same prompt on every gateway you are comparing inside one short window, interleaved, rather than one gateway per window, so drift and load land on all of them alike. Since the prompts are synthetic, publish the prompts themselves, not just their hashes — a hash confirms a prompt a reader already has, but nobody can re-run a comparison from hashes alone. Add a hash of each raw response to the record, and a stranger can check that the published traces match the scored runs without seeing any credentials.