{"slug": "ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs", "title": "TTFT benchmark: LLM Gateway vs. OpenRouter (Claude-haiku-4.5, 150 runs)", "summary": "A benchmark comparing LLM Gateway and OpenRouter for Claude Haiku 4.5 found LLM Gateway is ~35% faster to first token cold (906 vs 1392ms) and ~34% faster warm (814 vs 1232ms) at the median. The TTFB gap is architectural: LLM Gateway starts streaming in ~200ms before the first upstream token, while OpenRouter holds the response until the upstream answers. The test used 150 runs per gateway with zero errors across 300 requests.", "body_md": "# LLM Gateway vs. OpenRouter — TTFT benchmark\n\nPhase-by-phase latency comparison using [rbadillap/ai-gateways-benchmark](https://github.com/rbadillap/ai-gateways-benchmark) (Python stdlib only, raw sockets — it splits every request into DNS / TCP / TLS / TTFB / TTFT so connection cost and gateway overhead don't get conflated).\n\n**Method**: `claude-haiku-4.5` on both gateways · same prompt · `max_tokens=16` · 75 cold + 75 warm runs, interleaved round-robin so time-of-day drift hits both equally · cold = fresh connection (DNS+TCP+TLS, new TLS context, no session resumption) · warm = second request on an already-open socket · measured 2026-07-22 from a single residential connection · **zero errors across all 300 requests**.\n\n## Results\n\n| TTFT end-to-end (medians) | TTFB | TTFT (cold) | TTFT (warm) |\n|---|---:|---:|---:|\n| **LLM Gateway** | 201ms | 906ms | 814ms |\n| **OpenRouter (direct)** | 1369ms | 1392ms | 1232ms |\n\nTTFT (cold) is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays. TTFT (warm) is the connection-pool case.\n\n### Full phase breakdown (medians, ms)\n\n| Gateway | DNS | TCP | TLS | TTFB | TTFT | Cold e2e TTFT | Warm TTFB | Warm TTFT |\n|---|---:|---:|---:|---:|---:|---:|---:|---:|\n| LLM Gateway | 3.2 | 20.9 | 40.2 | 200.9 | 829.5 | 906.1 | 175.7 | 813.8 |\n| OpenRouter | 3.4 | 7.2 | 12.7 | 1369.4 | 1369.8 | 1392.0 | 1228.7 | 1232.1 |\n\n### Spread — median (p10–p90), ms\n\n| | LLM Gateway | OpenRouter |\n|---|---:|---:|\n| Cold TTFB | 201 (194–240) | 1369 (979–1643) |\n| Cold e2e TTFT | 906 (803–1380) | 1392 (1002–1675) |\n| Warm TTFB | 176 (171–193) | 1229 (924–1500) |\n| Warm TTFT | 814 (673–1379) | 1232 (924–1501) |\n\n## Observations\n\n- **LLM Gateway is ~35% faster to first token cold (906 vs 1392ms) and ~34% faster warm (814 vs 1232ms)** at the median. Its p90 cold e2e TTFT (1380ms) is still below OpenRouter's median.\n- **The TTFB gap is architectural, not just network.** LLM Gateway starts the response stream in ~200ms, before the first upstream token, and its warm TTFB is extremely stable (171–193ms p10–p90). OpenRouter's first byte arrives together with its first token (TTFB ≈ TTFT on every run) — it holds the response until the upstream answers. TTFT is the honest headline number; TTFB mostly tells you who streams headers early.\n- **Different upstreams.** OpenRouter picks its upstream per request (a test request was served via Amazon Bedrock); LLM Gateway's runs were pinned to Anthropic's API. Both are each gateway's default behavior for this model.\n- **Vantage point matters.** As the benchmark's README says, results are a property of where you measure from, not a global ranking. OpenRouter's edge terminated in Copenhagen for these runs (`cf-ray: …-CPH`); request receipts were captured per run.\n\n## Reproduce it\n\n```sh\ngit clone https://github.com/rbadillap/ai-gateways-benchmark\ncd ai-gateways-benchmark\n# config.json:\n{\n\t\"runs_cold\": 75,\n\t\"runs_warm\": 75,\n\t\"prompt\": \"Reply with the single word: pong\",\n\t\"max_tokens\": 16,\n\t\"gateways\": [\n\t\t{\n\t\t\t\"name\": \"llmgateway\",\n\t\t\t\"host\": \"api.llmgateway.io\",\n\t\t\t\"path\": \"/v1/chat/completions\",\n\t\t\t\"model\": \"anthropic/claude-haiku-4-5\",\n\t\t\t\"auth_value\": \"Bearer $LLMGATEWAY_API_KEY\"\n\t\t},\n\t\t{\n\t\t\t\"name\": \"openrouter\",\n\t\t\t\"host\": \"openrouter.ai\",\n\t\t\t\"path\": \"/api/v1/chat/completions\",\n\t\t\t\"model\": \"anthropic/claude-haiku-4.5\",\n\t\t\t\"auth_value\": \"Bearer $OPENROUTER_API_KEY\"\n\t\t}\n\t]\n}\n\nLLMGATEWAY_API_KEY=... OPENROUTER_API_KEY=... python3 bench.py config.json\n```\n", "url": "https://wpnews.pro/news/ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs", "canonical_source": "https://gist.github.com/smakosh/0ca360230fb267d375b91dbd50548b67", "published_at": "2026-07-22 16:41:00+00:00", "updated_at": "2026-07-22 16:52:42.136226+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "developer-tools"], "entities": ["LLM Gateway", "OpenRouter", "Anthropic", "Amazon Bedrock", "Claude Haiku 4.5", "rbadillap/ai-gateways-benchmark"], "alternates": {"html": "https://wpnews.pro/news/ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs", "markdown": "https://wpnews.pro/news/ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs.md", "text": "https://wpnews.pro/news/ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs.txt", "jsonld": "https://wpnews.pro/news/ttft-benchmark-llm-gateway-vs-openrouter-claude-haiku-4-5-150-runs.jsonld"}}