Phase-by-phase latency comparison using rbadillap/ai-gateways-benchmark (Python stdlib only, raw sockets — it splits every request into DNS / TCP / TLS / TTFB / TTFT so connection cost and gateway overhead don't get conflated).
Method: claude-haiku-4.5 on both gateways · same prompt · max_tokens=16 · 75 cold + 75 warm runs, interleaved round-robin so time-of-day drift hits both equally · cold = fresh connection (DNS+TCP+TLS, new TLS context, no session resumption) · warm = second request on an already-open socket · measured 2026-07-22 from a single residential connection · zero errors across all 300 requests.
Results #
| TTFT end-to-end (medians) | TTFB | TTFT (cold) | TTFT (warm) |
|---|---|---|---|
| LLM Gateway | 201ms | 906ms | 814ms |
| OpenRouter (direct) | 1369ms | 1392ms | 1232ms |
TTFT (cold) is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays. TTFT (warm) is the connection-pool case.
Full phase breakdown (medians, ms)
| Gateway | DNS | TCP | TLS | TTFB | TTFT | Cold e2e TTFT | Warm TTFB | Warm TTFT |
|---|---|---|---|---|---|---|---|---|
| LLM Gateway | 3.2 | 20.9 | 40.2 | 200.9 | 829.5 | 906.1 | 175.7 | 813.8 |
| OpenRouter | 3.4 | 7.2 | 12.7 | 1369.4 | 1369.8 | 1392.0 | 1228.7 | 1232.1 |
Spread — median (p10–p90), ms
| LLM Gateway | OpenRouter | |
|---|---|---|
| Cold TTFB | 201 (194–240) | 1369 (979–1643) |
| Cold e2e TTFT | 906 (803–1380) | 1392 (1002–1675) |
| Warm TTFB | 176 (171–193) | 1229 (924–1500) |
| Warm TTFT | 814 (673–1379) | 1232 (924–1501) |
Observations #
- LLM Gateway is ~35% faster to first token cold (906 vs 1392ms) and ~34% faster warm (814 vs 1232ms) at the median. Its p90 cold e2e TTFT (1380ms) is still below OpenRouter's median.
- The TTFB gap is architectural, not just network. LLM Gateway starts the response stream in ~200ms, before the first upstream token, and its warm TTFB is extremely stable (171–193ms p10–p90). OpenRouter's first byte arrives together with its first token (TTFB ≈ TTFT on every run) — it holds the response until the upstream answers. TTFT is the honest headline number; TTFB mostly tells you who streams headers early.
- Different upstreams. OpenRouter picks its upstream per request (a test request was served via Amazon Bedrock); LLM Gateway's runs were pinned to Anthropic's API. Both are each gateway's default behavior for this model.
- Vantage point matters. As the benchmark's README says, results are a property of where you measure from, not a global ranking. OpenRouter's edge terminated in Copenhagen for these runs (
cf-ray: …-CPH); request receipts were captured per run.
Reproduce it #
git clone https://github.com/rbadillap/ai-gateways-benchmark
cd ai-gateways-benchmark
{
"runs_cold": 75,
"runs_warm": 75,
"prompt": "Reply with the single word: pong",
"max_tokens": 16,
"gateways": [
{
"name": "llmgateway",
"host": "api.llmgateway.io",
"path": "/v1/chat/completions",
"model": "anthropic/claude-haiku-4-5",
"auth_value": "Bearer $LLMGATEWAY_API_KEY"
},
{
"name": "openrouter",
"host": "openrouter.ai",
"path": "/api/v1/chat/completions",
"model": "anthropic/claude-haiku-4.5",
"auth_value": "Bearer $OPENROUTER_API_KEY"
}
]
}
LLMGATEWAY_API_KEY=... OPENROUTER_API_KEY=... python3 bench.py config.json