TTFT benchmark: LLM Gateway vs. OpenRouter (Claude-haiku-4.5, 150 runs)

A benchmark comparing LLM Gateway and OpenRouter for Claude Haiku 4.5 found LLM Gateway is ~35% faster to first token cold (906 vs 1392ms) and ~34% faster warm (814 vs 1232ms) at the median. The TTFB gap is architectural: LLM Gateway starts streaming in ~200ms before the first upstream token, while OpenRouter holds the response until the upstream answers. The test used 150 runs per gateway with zero errors across 300 requests.

LLM Gateway vs. OpenRouter — TTFT benchmark Phase-by-phase latency comparison using rbadillap/ai-gateways-benchmark https://github.com/rbadillap/ai-gateways-benchmark Python stdlib only, raw sockets — it splits every request into DNS / TCP / TLS / TTFB / TTFT so connection cost and gateway overhead don't get conflated . Method : claude-haiku-4.5 on both gateways · same prompt · max tokens=16 · 75 cold + 75 warm runs, interleaved round-robin so time-of-day drift hits both equally · cold = fresh connection DNS+TCP+TLS, new TLS context, no session resumption · warm = second request on an already-open socket · measured 2026-07-22 from a single residential connection · zero errors across all 300 requests . Results | TTFT end-to-end medians | TTFB | TTFT cold | TTFT warm | |---|---:|---:|---:| | LLM Gateway | 201ms | 906ms | 814ms | | OpenRouter direct | 1369ms | 1392ms | 1232ms | TTFT cold is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays. TTFT warm is the connection-pool case. Full phase breakdown medians, ms | Gateway | DNS | TCP | TLS | TTFB | TTFT | Cold e2e TTFT | Warm TTFB | Warm TTFT | |---|---:|---:|---:|---:|---:|---:|---:|---:| | LLM Gateway | 3.2 | 20.9 | 40.2 | 200.9 | 829.5 | 906.1 | 175.7 | 813.8 | | OpenRouter | 3.4 | 7.2 | 12.7 | 1369.4 | 1369.8 | 1392.0 | 1228.7 | 1232.1 | Spread — median p10–p90 , ms | | LLM Gateway | OpenRouter | |---|---:|---:| | Cold TTFB | 201 194–240 | 1369 979–1643 | | Cold e2e TTFT | 906 803–1380 | 1392 1002–1675 | | Warm TTFB | 176 171–193 | 1229 924–1500 | | Warm TTFT | 814 673–1379 | 1232 924–1501 | Observations - LLM Gateway is ~35% faster to first token cold 906 vs 1392ms and ~34% faster warm 814 vs 1232ms at the median. Its p90 cold e2e TTFT 1380ms is still below OpenRouter's median. - The TTFB gap is architectural, not just network. LLM Gateway starts the response stream in ~200ms, before the first upstream token, and its warm TTFB is extremely stable 171–193ms p10–p90 . OpenRouter's first byte arrives together with its first token TTFB ≈ TTFT on every run — it holds the response until the upstream answers. TTFT is the honest headline number; TTFB mostly tells you who streams headers early. - Different upstreams. OpenRouter picks its upstream per request a test request was served via Amazon Bedrock ; LLM Gateway's runs were pinned to Anthropic's API. Both are each gateway's default behavior for this model. - Vantage point matters. As the benchmark's README says, results are a property of where you measure from, not a global ranking. OpenRouter's edge terminated in Copenhagen for these runs cf-ray: …-CPH ; request receipts were captured per run. Reproduce it sh git clone https://github.com/rbadillap/ai-gateways-benchmark cd ai-gateways-benchmark config.json: { "runs cold": 75, "runs warm": 75, "prompt": "Reply with the single word: pong", "max tokens": 16, "gateways": { "name": "llmgateway", "host": "api.llmgateway.io", "path": "/v1/chat/completions", "model": "anthropic/claude-haiku-4-5", "auth value": "Bearer $LLMGATEWAY API KEY" }, { "name": "openrouter", "host": "openrouter.ai", "path": "/api/v1/chat/completions", "model": "anthropic/claude-haiku-4.5", "auth value": "Bearer $OPENROUTER API KEY" } } LLMGATEWAY API KEY=... OPENROUTER API KEY=... python3 bench.py config.json