cd /news/ai-infrastructure/ai-gateway-performance-comparison-20… · home › topics › ai-infrastructure › article
[ARTICLE · art-174012] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=· neutral

AI Gateway Performance Comparison (2026): Python vs Go vs Rust vs Envoy, and Why the Benchmarks Disagree

A developer published a methodology for benchmarking AI gateways, arguing that widely cited performance charts disagree by up to 40x because they measure different intervals—in-process time versus client-side added latency, with or without production features like logging and spend tracking enabled. The writeup recommends timing eight hot-path steps including JSON parsing, auth, rate limiting and stream relay, and testing with a delayed SSE mock provider plus a k6 load generator rather than trusting vendor-published figures.

recommendation

Benchmark your gateway with auth, rate limiting and logging enabled against a 300 ms SSE mock, and check whether logging blocks the reply before comparing languages.

more in playbook: Inference and self-hosting
by read10 min views2 publishedNov 22, 2026

Published charts that crown a fastest AI gateway usually claim a lead by a factor of forty. They disagree with each other because they do not measure the same interval. One number is time spent inside the process, with JSON and the network left out. Another is added latency at the client, with logging and spend tracking off. A third turns production features on, uses a slower mock, or runs a single pass of a few thousand requests. Those are not competing answers to one question. They are answers to different questions, and each vendor's published run tends to put that vendor first. None of that requires dishonesty. It requires a method, and a decision about what to measure, before any product gets a paragraph.

Claims below were checked against project repos and docs as of November 2026. Every benchmark figure is labeled with who ran it. No new numbers were added for this piece.

Measure proportions before you measure products. A hosted chat completion usually takes hundreds of milliseconds, and often several seconds. A long stream takes longer still. Next to that, a gateway that adds 1 ms and a gateway that adds 5 ms are the same request.

Three cases make the overhead visible.

So the question worth answering is narrower than "which gateway is fastest." How much CPU and memory does this gateway need to hold your p99 at your peak concurrency, with the features you will actually enable?

Ignore the product name and the per-request work is roughly the same. A benchmark that skips a step is a benchmark of a different system. Time these eight:

The language shows up most in decode, encode, and reply: JSON parsing, translation, and stream relay, all of it CPU. Auth, limits, and logging are where deployments actually get slow, because those steps often touch a database or Redis. A synchronous Postgres write on every request will lose to a slower language that logs after the fact. Architecture beats the runtime more often than a chart will admit.

Calls on the hot path. Write down every network hop before the provider: a key lookup in a database, a rate-limit check in Redis, a policy service. Then ask what a slow or dead hop does. Fail open, fail closed, or hang?

Whether logging blocks the reply. Synchronous request logging makes the log store's write time your latency. Confirm that property on the gateway you run before you argue about languages.

Memory per replica. Kubernetes packs pods by memory request. A process that idles in the tens of megabytes and a process that idles in gigabytes do not have the same fleet cost, even at the same latency.

State shared by replicas. Budgets and rate limits have to agree across three replicas. The store for that state (Redis, Postgres, an Envoy rate-limit service) is often the throughput ceiling.

What a rollback costs. A single static binary rolls back in seconds. A stack with schema migrations does not. In an incident, that difference matters more than 2 ms of overhead.

A lab is unnecessary. You need a mock that acts like a provider, a load generator, and the discipline to turn on the features production will actually use.

1. A mock with delay and a stream. Any small server that answers /v1/chat/completions will do, if it can wait a configurable delay (start at 300 ms) and emit SSE chunks. Put it on a different machine from the gateway.

2. A k6 script that streams and varies concurrency.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  scenarios: {
    ramp: {
      executor: 'ramping-vus',
      stages: [
        { duration: '2m', target: 50 },
        { duration: '3m', target: 300 },
        { duration: '3m', target: 1000 },
      ],
    },
  },
  thresholds: { http_req_failed: ['rate<0.01'] },
};

const body = JSON.stringify({
  model: __ENV.MODEL || 'mock-model',
  stream: true,
  messages: [{ role: 'user', content: 'Summarize this paragraph in one line.' }],
});

export default function () {
  const res = http.post(`${__ENV.GATEWAY_URL}/v1/chat/completions`, body, {
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${__ENV.GATEWAY_KEY}`,
    },
  });
  check(res, { 'status is 200': (r) => r.status === 200 });
}

3. Baseline, then gateway. Hit the mock with the same script, then put each gateway in front of it. Publish added latency, gateway minus baseline, at p50, p95, and p99 for every concurrency stage.

4. Repeat with production features on. Per-key auth, budgets, rate limits, request logging, and at least one fallback route. The gap between that run and the features-off run is usually the number that matters.

5. Write down the machines, not just the latency. CPU cores and peak resident memory for each gateway replica, plus the dependencies it drags along (Postgres, Redis, ClickHouse). Divide throughput by total cores if you want an efficiency figure that is fair.

6. Three runs, and publish the config. Numbers you cannot accompany with a config should not be trusted. Do not trust them.

This is every public figure found in the materials checked for this piece, with the context the headline usually leaves out.

Claim Who ran it Setup What to keep in mind
Bifrost adds ~11 µs per request at 5,000 RPS Maxim (Bifrost's maker) AWS t3.xlarge; Bifrost's internal timing Excludes JSON marshalling and the HTTP call; peak memory reported at ~3.3 GB in the same run
LiteLLM Rust p99 added ~0.7 ms vs Portkey 2.3 ms, Bifrost 4.5 ms, LiteLLM Python 257.7 ms LiteLLM (July 2026) Local Rust mock upstream, one host, logging and spend tracking off Single runs, no error bars; the author says to read it as order-of-magnitude
Median added latency at 50 RPS: LiteLLM Rust 0.9 ms, Bifrost 1.2 ms, Portkey 2.1 ms, LiteLLM Python 79.3 ms LiteLLM (Nov 2026) Production features on, LiteLLM Rust from a feature branch Portkey ran without the listed features because it lacks them
Bifrost and LiteLLM's Rust beta tie on throughput (~2,720 RPS at concurrency 64); Python ~180 RPS Maxim (Bifrost's maker) Shared test files, 2,000 requests per cell Bifrost ran with logging to SQLite; LiteLLM with logging off. A second run reversed the p99 order
Kong "over 200%" faster than Portkey and "over 800%" faster than LiteLLM Kong (July 2025) EKS, WireMock mock, proxy-only, no plugins, Kong 3.10 vs Portkey 1.9.19 vs LiteLLM 1.63.7 Old versions; no auth, caching or limits enabled
Kong ~15,900 req/s at 1,000 VUs; Bifrost failed above 300 VUs; LiteLLM capped ~175 req/s; Portkey plateaued ~850 req/s Author of Ferro Labs, a competing Go gateway One 8-vCPU VM, 60 ms mock, default configs, no tuning Author has a stake and says so; Bifrost v1.0.0 was tested, now well behind current releases
Kaeso Nobody yet n/a No published benchmarks. An open benchmark, Gatebench, is in progress with no results

Read the table a second time and the pattern is obvious. The publisher of each row finishes first in that row. Bifrost's run favors Bifrost. LiteLLM's favors the Rust build. Kong's favors Kong. Ferro Labs' favors Ferro Labs. The rows are measuring different things, which is why they can all be careful and still disagree.

The method above is the comparison. The runtimes below are what you are actually deploying when you pick a project. They explain fleet cost and tail behavior. They do not crown a winner the table already refused to crown.

Stock CPython runs bytecode for one thread at a time inside a process, so a Python gateway grows by adding worker processes, not by sharing a core. LiteLLM's sizing guide is the concrete number: about 100 combined RPS per four-worker pod. On that same pod, at 200 RPS, the measured p99 overhead rose to roughly 650 ms. The 3,000 RPS showcase used 33 pods, each with 4 vCPU and 16 GiB. That is the cost of a CPU-heavy proxy in Python, not a verdict on the engineers.

The project is leaving that cost behind in stages. Since v1.110.0-rc.1 the router always runs on a Rust core, and the Python router is gone. A Rust server for /v1/messages is in beta. The roadmap aims to have that endpoint in Rust end to end by December 1, 2026. Other endpoints stay on Python for now. Call it Python with a growing Rust core. Do not call it pure Python.

The Node event loop is one thread, and it is good at concurrent input and output. JSON parsing and stream rewriting occupy that same thread. Once the CPU is busy, every request waits on every other request. Portkey's README advertises "<1ms latency" and a 122 kb footprint. Cloudflare Workers is also a supported target, which hands the scaling problem to the platform.

Goroutines spread work across every core in a single process. The garbage collector is tuned for short s. You get one binary, and vertical scaling is straightforward. The costs are memory from the collector and, in one published run, sensitivity to how the connection pool is sized once concurrency is high. That run is the Ferro Labs row above. The author of that test has a stake, and the Bifrost build under test, v1.0.0, is well behind current releases. Use the pool point as a tuning warning, not as a ranking.

Plugin code in Kong and APISIX runs as LuaJIT inside NGINX's event-driven workers. API gateways have lived in that model for a long time, and the performance shape is well understood. For model traffic the open question is how much of the AI-specific work stays cheap in Lua, and, on Kong, which of those plugins the license lets you run.

Agent Router, the project formerly called Envoy AI Gateway, leaves connection handling on Envoy's C++ data plane and puts the AI work in a Go external processor: provider routing, format translation, token accounting. A separate rate-limit service enforces token-based limits. You inherit Envoy's networking, and you pay an extra call between Envoy and the processor on every request. Higress is a cousin of that design: Envoy, plus Wasm plugins written in Go, Rust, or JS.

No garbage collector, memory use you can predict, and async input and output on Tokio-style runtimes. For a gateway the interesting part is the tail. Without collector s, and with little allocation per request, p99 stays near p50, and each replica stays small in memory. agentgateway, a Linux Foundation project, is Rust. So is API7's AISIX OSS, which ships as one static binary. LiteLLM's new core is Rust too. The language is not a maintenance plan. TensorZero's repository was archived on June 12, 2026. Helicone's standalone AI gateway has had no commit since November 2025.

The Kaeso gateway is one Rust process. The console is TypeScript. Its docs describe the request path as the eight ordered checks above. Logs go to ClickHouse after the reply has already been sent, so the write is off the critical path and does not hold the response. Budgets and limits are evaluated before any provider is called. None of those design choices is a benchmark. No independent numbers have been published, and Gatebench, the open benchmark, has no results to cite.

Language and footprint have to be read together. The default install is Docker Compose: Caddy, the gateway, the console, PostgreSQL, ClickHouse, and Valkey, about a 2 GB download. A Helm chart exists for Kubernetes. A small Rust process beside three stateful services is not a single static binary. The fleet bill includes the stateful services.

As of November 2026 there is no independent answer. Bifrost, LiteLLM's Rust build, Kong, and others each publish a setup in which they win. Run your own traffic, with the features you will leave on.

LiteLLM's published tests and Maxim's published tests put a Rust gateway and a Go gateway close on throughput. The tail-latency order flipped between runs. Rust's clearer edge is memory, and the absence of garbage-collection s. Architecture still outweighs the language.

The proxy grew up in Python. It scales by adding worker processes, and parsing plus translation cost more CPU per request. That is why the hot paths are moving onto a Rust core.

On most single requests, no. The provider is slower by a wide margin. You notice the gateway in a long agent loop, and at peak concurrency, when a CPU-bound process watches its p99 climb.

Not yet. Nothing has been published, and Gatebench is still in progress with no results.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @envoy 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/ai-gateway-performan…] indexed:0 read:10min 2026-11-22 · —