{"slug": "go-vs-python-for-an-llm-gateway-what-we-measured", "title": "Go vs. Python for an LLM Gateway: What We Measured", "summary": "A reproducible benchmark on identical Google Cloud c3-standard-4 hardware found that the Go-based KTMS LLM gateway added less latency, used less memory and CPU, and started faster than LiteLLM Proxy v1.106.0 running on Python with 4 Uvicorn workers, according to the KTMS team behind the open-source ktms/bench stand. The benchmark also compared KTMS against eight other self-hosted gateways using a Go mock of the OpenAI Chat Completions API and k6 2.3.0, with the full suite run twice and latency, memory, CPU and maximum throughput agreeing within about 10% between runs, except LiteLLM's p99. The stand surfaced a bug in KTMS's own code, which the team fixed before the measured runs; KTMS's Postgres and Redis dependencies were excluded from its memory figures, while LiteLLM needed neither in its leanest mode.", "body_md": "# Go vs Python for an LLM Gateway: What We Measured\n\nA reproducible benchmark of KTMS (Go) against LiteLLM Proxy (Python) and eight other self-hosted gateways on identical cloud hardware: added latency, time to first token, throughput, memory, CPU and startup time.\n\nPython is the language of machine learning, so most teams reach for a Python gateway first. We built\n[**Keystone. The Model Switchboard. (KTMS)**](https://github.com/ai-process/ktms), our self-hosted\nLLM gateway, in Go instead. A\ngateway doesn’t run models. It moves bytes between your services and model vendors, and that is a\nnetworking problem.\n\n“Go is lighter” is easy to say and hard to check, so we built a reproducible benchmark stand and ran KTMS against LiteLLM Proxy on identical hardware, then against eight more self-hosted gateways. This post walks through what we measured, the bug the stand found in our own code, and where the numbers stop.\n\n## 1. What a Gateway Actually Does\n\nAn LLM gateway sits on the request path of every model call:\n\n- **It is I/O-bound.** It authenticates, routes, enforces limits, converts protocols (gRPC to\nHTTP), forwards the request, and streams the answer back. None of that is compute-heavy.\n- **It holds connections open.** Streaming responses keep a connection busy for as long as the model\ngenerates, so concurrency matters more than raw request rate.\n- **It must stay out of the way.** Model latency is measured in hundreds of milliseconds or more, so\nwhatever the gateway adds should be lost in the noise.\n- **It runs everywhere, all the time.** Staging, preview and production all run a copy, so its idle\nfootprint is paid many times over.\n\nPython gateways such as LiteLLM run on FastAPI and Uvicorn. To use more than one core, they start several worker processes, because of the Global Interpreter Lock. Each worker loads its own copy of the interpreter, the vendor SDKs and the data models. A Go service is one process: the runtime schedules goroutines across all cores and shares one set of connection pools.\n\n## 2. The Benchmark Stand\n\nThe whole stand is open source in [`ktms/bench`](https://github.com/ai-process/ktms/tree/main/bench)\nand reruns with one command. Terraform creates an isolated network and two VMs, the script runs the\nsuite and then destroys everything.\n\n```\n runner VM                     gateway VM (private network)\n ┌──────────┐   HTTP    ┌────────────────────────────────────────┐\n │   k6     │ ────────▶ │ KTMS ──────┐                           │\n │          │           │ LiteLLM ───┼──▶ mock OpenAI upstream    │\n └──────────┘           │ (direct) ──┘                           │\n                        └────────────────────────────────────────┘\n```\n\n- **Hardware:** Google Cloud`c3-standard-4` (4 vCPU, 16 GB, x86_64) for the gateway, and a second,\nidentical VM for the load generator, so k6 never competes with the gateway for CPU.\n- **Versions:** KTMS built from source with the fix described in section 4; LiteLLM`v1.106.0` (released before the\ncompromised 1.107.0, which we never ran) with 4 Uvicorn workers (one per core), master key only\nand no database, its leanest mode; k6 2.3.0.\n- **Upstream:** a small Go mock of the OpenAI Chat Completions API with fixed responses and\nconfigurable delays. Real vendors would add hundreds of milliseconds of noise and make runs\nincomparable. The mock counts the requests it receives, which proves that no answer came from a\ngateway cache.\n- **Baseline:** every scenario also runs with no gateway at all (“direct”), so “added latency” means\ngateway minus direct.\n- **One gateway at a time:** the others are stopped before each run.\n- **Memory:** the cgroup RSS of the gateway container only. KTMS also needs Postgres and Redis, which\nare**not** counted. LiteLLM in this mode needs neither.\n\nWe ran the full suite twice. Latency, memory, CPU and maximum throughput agreed within about 10% between the runs for every gateway; LiteLLM’s p99 was the exception. The charts show the mean of both.\n\n## 3. Results\n\n### Added Latency\n\nNon-streaming requests at a steady 200 RPS against an upstream that answers instantly.\n\nKTMS adds about 0.6 ms at the median and about 1.1 ms at p99. LiteLLM adds about 8.5 ms at the median, and its p99 swung from 27 ms in one run to 229 ms in the other, so the mean in the chart is only a rough guide.\n\n### Time to First Token\n\n50 concurrent streams, each with 20 chunks sent 15 ms apart.\n\nBoth gateways delivered every chunk of every stream. KTMS forwards the first token about 0.7 ms after it arrives from the upstream. LiteLLM takes about 160 ms. It doesn’t merge chunks: the whole stream just starts later, and finishes the same ~160 ms later.\n\n### Throughput\n\nThe offered rate climbs from 250 to 4,000 RPS in steps of 20 seconds, with a 10 ms upstream. A step counts as sustained when at least 95% of the offered rate gets through with fewer than 1% errors.\n\nKTMS kept up with every step, including 4,000 RPS, the highest we tested. Its p99 stayed at 13 ms up to 2,000 RPS (direct: 11 ms) and grew to 26–43 ms at 4,000 RPS. LiteLLM sustained 250 RPS. At 500 RPS a third of its requests already failed, and from 1,000 RPS up it plateaued at about 570 RPS while half of the requests hit the 5-second client timeout.\n\n### Memory\n\nAt idle, KTMS uses about 7 MiB. LiteLLM uses about 2.5 GiB with 4 workers, roughly 0.6 GiB per worker. During a 5-minute soak at 200 RPS, KTMS stayed at about 20 MiB and LiteLLM at about 2.9 GiB, and neither drifted upward. KTMS peaked at 41–46 MiB over the whole run, including 4,000 RPS, well under the 512 MiB limit in our Helm chart.\n\nThe chart uses a linear scale on purpose. The KTMS bars are hard to see, and that is the result.\n\n### CPU\n\nOver the soak, KTMS spent about 0.63 s of CPU time per 1,000 requests. LiteLLM spent about 9.7 s, fifteen times more. This is the number that decides how many cores a given traffic level needs.\n\n### Startup\n\nFrom `docker start` to the first request answered through the gateway, KTMS took about 170 ms and\nLiteLLM about 32 s. That time includes starting the container itself, and the KTMS database was\nalready migrated.\n\nThe images differ just as much. For linux/amd64, the KTMS slim image is about 21 MB to pull and about 67 MB unpacked. The LiteLLM image is about 420 MB to pull and about 1.3 GB unpacked.\n\n## 4. The Benchmark Found a Bug in KTMS\n\nThe first full run did not look like the charts above. KTMS was fine up to 2,000 RPS, then failed\n96% of requests at 3,000 RPS, each in under a millisecond. The log said\n`connect: cannot assign requested address`: the container had run out of outbound ports.\n\nThe cause was one missing line of configuration. The OpenAI, Anthropic and Gemini SDKs fell back to Go’s default HTTP transport, which keeps only two idle connections per host. Every concurrent request beyond two opened a new connection to the upstream, and the closed ones piled up in TIME_WAIT until no ports were left. All vendor clients now share one transport that keeps up to 1,024 idle connections per host. The same run then sustained 4,000 RPS without errors, and p99 at 2,000 RPS dropped from 50–60 ms to 13 ms.\n\nEvery Go service that talks to a single upstream through an SDK can hit this. Check what your HTTP\nclient does with `MaxIdleConnsPerHost`.\n\n## 5. What About Other Gateways?\n\nComparing Go with Python only tells half of the story, so we ran the same suite against eight more self-hosted gateways: Bifrost (Go), Portkey (TypeScript), Kong AI Gateway and Apache APISIX (Nginx/Lua), Higress and Envoy AI Gateway (Envoy), agentgateway (Rust) and New API (Go). Plano (Rust/Envoy) could not hold the 200 RPS scenarios in our setup and is left out.\n\nThe results split into two leagues:\n\n- **Compiled and Nginx/Envoy-based proxies** add 0.25–1.3 ms at p50 and all sustained 4,000 RPS.\nKTMS is in this league.\n- **Python and Node.js gateways** fall behind: Portkey, which ships as a single Node.js process,\nsaturated at 500 RPS, and LiteLLM at 250 RPS. New API (Go) sits in between at 2,000 RPS.\n\nKTMS is not the fastest proxy we measured. agentgateway, APISIX and Kong add 0.25–0.35 ms where KTMS adds 0.64 ms, and spend less CPU per request. They also do less work per request in this setup (a static key check, against KTMS’s key lookup, rule routing and limit checks), but we have not profiled what each step costs, so that is the next thing we will look at. On memory, only agentgateway is as frugal as KTMS; the other compiled proxies need 47–530 MiB.\n\n## 6. Why the Gap With Python Is So Large\n\n**One process instead of four.** LiteLLM needs a worker per core, and each worker carries its own\ninterpreter, imported SDKs, data models and connection pools. That is where most of the 2.5 GiB goes\nbefore the first request arrives. KTMS is a single process. Goroutines start with a small stack that\ngrows on demand, and one PostgreSQL pool and one HTTP transport serve every request.\n\n**No interpreter on the hot path.** Each request in a Python gateway passes through several layers of\ninterpreted middleware, validation and logging. In Go, the same work is compiled code with no\ninterpreter dispatch. It shows up twice: as added latency, and as CPU time per request.\n\n**No import phase at startup.** A Python service must import its dependency tree before it can\nanswer. A Go binary just parses its configuration and connects to its database.\n\nGo is not magic, though: the Rust and Nginx proxies above show that a compiled language is necessary for this class of performance, not sufficient. What matters is what runs per request.\n\n**Predictable memory limits.** Go’s `GOMEMLIMIT` makes the garbage collector work harder as the\nprocess approaches a soft limit, which helps a container ride out traffic spikes without being\nOOM-killed.\n\n## 7. What We Did Not Measure\n\nThese numbers describe this stand, not every deployment:\n\n- **The upstream is a mock.** Real vendors add TLS, network round trips and hundreds of milliseconds\nof generation time, which dwarf either gateway’s overhead at low traffic. The differences matter\nmost for throughput, memory and cost.\n- **We stopped at 4,000 RPS.** The real ceilings of KTMS and the other compiled proxies on 4 vCPU are\nstill unknown.\n- **Every gateway ran with one minimal configuration,** routing to the mock with retries, caching and\nlogs off where possible. LiteLLM ran standalone with no database and no Redis. Other configurations\nwill give different numbers, and we’d welcome pull requests that tune any of them.\n- **Postgres and Redis are excluded from memory.** KTMS and New API need both.\n- **One machine shape, one workload:** short chat completions and streams. We did not test thousands\nof long-lived streams or large payloads.\n- **[light-tokenmeter](https://github.com/ai-process/light-tokenmeter)** was not part of this run.\n\n## 8. Trade-Offs We Accepted\n\nGo isn’t free:\n\n1. **Day-0 provider support.** Vendors ship Python SDKs first. A Python gateway can wrap a new API in\nhours; in Go, we write typed adapters against the raw HTTP contract.\n2. **Extensibility.** Python makes it easy to drop in custom middleware or scripting. Go means\ncompiled code and structured configuration.\n3. **Verbosity.** Explicit error handling and static types mean more lines of code.\n\nFor prototypes, or for teams that need the long tail of niche providers, LiteLLM is a reasonable choice. For a shared production gateway, where latency, density, CPU cost and startup matter, the measurements favour a compiled gateway. Among those, pick by features: KTMS’s are a gRPC API, effort routing, budgets and a pool of local GPU nodes under the Apache 2.0 licence.\n\n## 9. Summary\n\n| Metric (4 vCPU, mock upstream) | KTMS (Go) | LiteLLM (Python, 4 workers) | \n|---|---|---|\n| Added latency, p50 / p99 | ~0.6 / ~1.1 ms | ~8.5 / 27–229 ms | \n| Time to first token, added (p50) | ~0.7 ms | ~160 ms | \n| Max sustained throughput | ≥ 4,000 RPS (highest tested) | ~250 RPS | \n| Memory, idle / 200 RPS | ~7 MiB / ~20 MiB | ~2.5 GiB / ~2.9 GiB | \n| CPU time per 1,000 requests | ~0.63 s | ~9.7 s | \n| Time to readiness | ~170 ms | ~32 s | \n| Image (pull / unpacked, amd64) | ~21 MB / ~67 MB | ~420 MB / ~1.3 GB | \n\nEvery number above comes from two runs of a public stand. Clone\n[`ktms/bench`](https://github.com/ai-process/ktms/tree/main/bench), run it in your own cloud\naccount, and tell us what you get.\n\n## What’s Next?\n\nUpcoming posts:\n\n- Profiling the KTMS request path: closing the gap to agentgateway and APISIX.\n- Configuring multi-vendor fallback chains across OpenAI, Anthropic and Gemini.\n- Usage accounting with `light-tokenmeter` and ClickHouse.\n\nExplore the repositories on GitHub:\n\n- [**KTMS**](https://github.com/ai-process/ktms) : self-hosted LLM gateway.\n- [**light-tokenmeter**](https://github.com/ai-process/light-tokenmeter) : token and spend accounting.\n- [**ktms-admin**](https://github.com/ai-process/ktms-admin) : web console for models, rules and keys.", "url": "https://wpnews.pro/news/go-vs-python-for-an-llm-gateway-what-we-measured", "canonical_source": "https://blog.ai-process.dev/2026/10/26/go-vs-python-llm-gateway/", "published_at": "2026-11-04 18:06:08+00:00", "updated_at": "2026-11-04 18:20:32.144337+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "mlops"], "entities": ["KTMS", "LiteLLM Proxy", "Keystone", "Google Cloud", "k6", "Uvicorn", "FastAPI", "OpenAI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/go-vs-python-for-an-llm-gateway-what-we-measured", "markdown": "https://wpnews.pro/news/go-vs-python-for-an-llm-gateway-what-we-measured.md", "text": "https://wpnews.pro/news/go-vs-python-for-an-llm-gateway-what-we-measured.txt", "jsonld": "https://wpnews.pro/news/go-vs-python-for-an-llm-gateway-what-we-measured.jsonld"}}