{"slug": "openrouter-vs-vercel-vs-llmgateway-performance", "title": "OpenRouter vs Vercel vs LLMGateway Performance", "summary": "A developer benchmarked time-to-first-token (TTFT) across AI gateways, finding LLM Gateway delivered the first content token 35% faster cold (906ms vs 1392ms) and 34% faster warm (814ms vs 1232ms) than OpenRouter, with zero errors across 300 runs. The open-source test used the ai-gateways-benchmark script to separate DNS, TCP, TLS, TTFB, and TTFT phases.", "body_md": "Every AI gateway adds a hop between your app and the model. The question that matters is what that hop costs at the moment your user is staring at a blank chat window: the time to first token. Most gateway latency debates skip the measurement and argue architecture — so we measured it.\n\nWe ran an open-source TTFT benchmark against **LLM Gateway** and OpenRouter, interleaved, from the same machine, on the same model. The medians over 75 runs each: LLM Gateway got the first content token in 906ms on a cold connection and 814ms on a warm one. OpenRouter took 1392ms and 1232ms. That is roughly 35% faster cold and 34% faster warm, with zero errors across the 300 measured runs — 450 HTTP requests in total, counting the throwaway warm-up call that precedes each warm measurement, every one of which returned HTTP 200. The raw per-run data is [published in full](https://gist.github.com/smakosh/0ca360230fb267d375b91dbd50548b67).\n\nWe used [ai-gateways-benchmark](https://github.com/rbadillap/ai-gateways-benchmark), an open-source script by Ronny Badilla that recently made the rounds comparing Vercel AI Gateway, OpenRouter, and Cloudflare AI Gateway. It is Python stdlib only — raw sockets, no HTTP library — and it times every phase of a streaming request separately: DNS, TCP connect, TLS handshake, TTFB (request sent to first response byte), and TTFT (request sent to first content token in the SSE stream).\n\nThat separation is the point. \"Gateway latency\" claims usually conflate connection cost, edge proximity, and actual routing overhead. This tool splits them apart.\n\nOur setup, on July 22, 2026:\n\n`max_tokens: 16`\n\n, identical promptMedians of n=75 per cell. Cold TTFT is end-to-end: DNS + TCP + TLS + time to first content token — what a short-lived process pays.\n\n| Medians | TTFB (cold) | TTFT (cold) | TTFT (warm) |\n|---|---|---|---|\n| LLM Gateway | 201ms | 906ms | 814ms |\n| OpenRouter (direct) | 1369ms | 1392ms | 1232ms |\n\nThe spread, as median with p10–p90 range:\n\n| Metric | LLM Gateway | OpenRouter |\n|---|---|---|\n| Cold TTFB | 201 (194–240) | 1369 (979–1643) |\n| Cold e2e TTFT | 906 (803–1380) | 1392 (1002–1675) |\n| Warm TTFB | 176 (171–193) | 1229 (924–1500) |\n| Warm TTFT | 814 (673–1379) | 1232 (924–1501) |\n\nTwo things stand out beyond the medians. LLM Gateway's p90 cold TTFT (1380ms) came in below OpenRouter's median — the slow tail of one distribution beat the middle of the other. And LLM Gateway's warm TTFB barely moves: 171–193ms across the p10–p90 range, which is the stability you want under a production connection pool.\n\nThe phase breakdown shows the overhead is not in the connection:\n\n| Gateway | DNS | TCP | TLS | TTFB | TTFT (request) |\n|---|---|---|---|---|---|\n| LLM Gateway | 3.2 | 20.9 | 40.2 | 200.9 | 829.5 |\n| OpenRouter | 3.4 | 7.2 | 12.7 | 1369.4 | 1369.8 |\n\nOpenRouter's edge actually wins the handshake — 13ms TLS vs our 40ms. It loses everything after the request is sent.\n\nOne honest caveat on the TTFB column: it flatters LLM Gateway for an architectural reason. Our gateway starts the response stream in about 200ms, before the first upstream token arrives. OpenRouter holds its first byte until the first token is ready, so its TTFB equals its TTFT on every single run. TTFB tells you who streams headers early; TTFT is the number your users feel, and it is the honest headline of this post.\n\nA second caveat: each gateway ran its default routing for this model. OpenRouter picks its upstream per request (a request we inspected was served via Amazon Bedrock), while our runs were pinned to Anthropic's API. Both are what you get out of the box, but they are different upstreams.\n\nWe did not benchmark Vercel ourselves. The benchmark's author published his own run of the same script — from his vantage point, on a different date — comparing Vercel AI Gateway, OpenRouter, and Cloudflare AI Gateway proxying OpenRouter:\n\n| His run (medians of n=5) | TTFB | TTFT (cold) | TTFT (warm) |\n|---|---|---|---|\n| Vercel AI Gateway | 785ms | 1099ms | 822ms |\n| OpenRouter (direct) | 1100ms | 1123ms | 986ms |\n| Cloudflare → OpenRouter | 1300ms | 1420ms | 1279ms |\n\nThese numbers are not directly comparable to ours — different location, different network, different day, n=5 vs n=75. What they are useful for is a sanity check: OpenRouter lands north of a second to first token in his run and in ours, from two different corners of the internet. Against his table, LLM Gateway's 906ms cold and 814ms warm sit at or below the best row — but the only comparison we will state as fact is the one we measured ourselves, interleaved, from one machine.\n\nLocation dependence cuts both ways, and the benchmark's README says so explicitly: results are a property of where you measure from, not a global ranking. Which is why the right move is to run it yourself.\n\nThe whole thing is a config file and two API keys:\n\n```\ngit clone https://github.com/rbadillap/ai-gateways-benchmark\ncd ai-gateways-benchmark\n{\n  \"runs_cold\": 75,\n  \"runs_warm\": 75,\n  \"prompt\": \"Reply with the single word: pong\",\n  \"max_tokens\": 16,\n  \"gateways\": [\n    {\n      \"name\": \"llmgateway\",\n      \"host\": \"api.llmgateway.io\",\n      \"path\": \"/v1/chat/completions\",\n      \"model\": \"anthropic/claude-haiku-4-5\",\n      \"auth_value\": \"Bearer $LLM_GATEWAY_API_KEY\"\n    },\n    {\n      \"name\": \"openrouter\",\n      \"host\": \"openrouter.ai\",\n      \"path\": \"/api/v1/chat/completions\",\n      \"model\": \"anthropic/claude-haiku-4.5\",\n      \"auth_value\": \"Bearer $OPENROUTER_API_KEY\"\n    }\n  ]\n}\nLLM_GATEWAY_API_KEY=... OPENROUTER_API_KEY=... python3 bench.py config.json\n```\n\nIt prints per-run lines while it works, then a medians table, and dumps raw per-run JSON with request-id receipts. If you run it from your region and get different numbers — including ones where we lose — we want to see them.\n\nA gateway earns its hop with failover, unified billing, and one API across [every model it routes](https://llmgateway.io/models). But you pay its latency on every single request, forever. That makes time to first token one of the few gateway properties worth measuring before you commit — and one of the easiest, since the tooling is open source and takes minutes to run.\n\nIf you are on OpenRouter today, LLM Gateway speaks the same OpenAI-compatible API — switching means changing the base URL and swapping in an LLM Gateway API key, covered in the [OpenRouter migration guide](https://docs.llmgateway.io/migrations/openrouter).\n\nTTFT (time to first token) is the delay between sending a request and receiving the first piece of model output in the stream. TTFB (time to first byte) only measures when the server starts responding — a gateway can stream headers immediately while the model is still silent. For anything a user watches in real time, TTFT is the latency they actually experience.\n\nIt depends on the model and your distance from the gateway's edge, so compare gateways against each other rather than an absolute bar. In this AI gateway performance benchmark, first tokens from claude-haiku-4.5 arrived in roughly 800–900ms through LLM Gateway and 1200–1400ms through OpenRouter, measured from the same machine in the same session.\n\nNo — latency benchmarks are a property of the vantage point, and the benchmark's own README is explicit about it. Our numbers come from one residential connection on one day; the author's Vercel numbers come from another. The script is open source and takes minutes to run, so measure from where your servers actually live.\n\nLLM Gateway routes an overlapping catalogue — the full list is on the [models page](https://llmgateway.io/models), spanning the major closed and open-weight providers on the [providers page](https://llmgateway.io/providers). Requests use the OpenAI-compatible format either way, so moving a workload between the two is a base URL and key change.\n\n**Measure it yourself:**", "url": "https://wpnews.pro/news/openrouter-vs-vercel-vs-llmgateway-performance", "canonical_source": "https://dev.to/smakosh/openrouter-vs-vercel-vs-llmgateway-performance-1f6a", "published_at": "2026-07-22 17:43:45+00:00", "updated_at": "2026-07-22 18:00:58.173342+00:00", "lang": "en", "topics": ["artificial-intelligence", "developer-tools", "ai-infrastructure"], "entities": ["LLM Gateway", "OpenRouter", "Vercel AI Gateway", "Cloudflare AI Gateway", "Ronny Badilla", "Anthropic", "Amazon Bedrock"], "alternates": {"html": "https://wpnews.pro/news/openrouter-vs-vercel-vs-llmgateway-performance", "markdown": "https://wpnews.pro/news/openrouter-vs-vercel-vs-llmgateway-performance.md", "text": "https://wpnews.pro/news/openrouter-vs-vercel-vs-llmgateway-performance.txt", "jsonld": "https://wpnews.pro/news/openrouter-vs-vercel-vs-llmgateway-performance.jsonld"}}