Designing the Langfuse AI Gateway Langfuse published a design post explaining why it built its new AI Gateway in Rust rather than on the TypeScript/Node.js stack that runs the rest of its platform, citing a production median request size of about 290 KB and the need to hold many long-lived streams open without exceeding container memory limits during traffic spikes. The gateway sits in front of every model call, hands out provider-agnostic keys that operators can revoke in one place, keeps provider credentials secret in Langfuse, and records each call as a trace for cost tracking per key, user and team. Langfuse framed the work around coding agents such as Claude Code and Codex, arguing that changing the base URL agents call is a smaller step than installing client-side observability hooks that can break when the agent updates. Designing the Langfuse AI Gateway Users want a gateway that is stable to run and invisible to the agents that call it. Here is what we learned designing it: why the runtime matters for this workload, why we picked Rust, and why we pass provider APIs through without translating them. Langfuse is an AI engineering platform with tools for tracing, improving and monitoring AI applications. To trace an application, you instrument it with a Langfuse SDK or with OpenTelemetry, and it sends traces to Langfuse. Right now, the killer app in AI is coding agents. Their cost is growing fast in most organizations, and there is very little transparency into what that money buys or how well the agents work. Claude Code and Codex offer client-side hooks for observability, but they are clunky and not always reliable. They are installed on every developer's machine and updated separately from the agent, so an agent update can quietly break them. Changing the base URL the agents call is a much smaller step. That's where an AI gateway helps. It is the one place all AI traffic passes through, so platform teams get control back. With the Langfuse AI Gateway https://langfuse.com/docs/gateway/overview , operators hand out gateway keys that work across providers, and manage and revoke them in one place, while the provider credentials stay secret in Langfuse. They can see cost per key, user and team. And because every call lands in Langfuse as a trace, they can use dashboards, evaluations and alerts to monitor and improve how their organization uses coding agents. A gateway that sits in front of every model call has to be stable before anything else. If it falls over, every agent in the company stops. In this post we describe how we designed the gateway for the two groups that depend on it: the operators who run it, and the callers whose agents send traffic through it. Along the way we explain the runtime behavior that drove our decisions, because the lessons apply to any service with a similar workload. What a gateway has to handle what-a-gateway-has-to-handle Before we picked a language, we wrote down what the workload looks like. A gateway is mostly a proxy: it receives a request, checks the key, forwards the request to the provider and streams the response back. Coding-agent traffic makes it demanding in four ways. Large requests. An agent resends the whole conversation on every turn: the system prompt, tool definitions, earlier messages, file contents and tool output. A single request is often a few hundred kilobytes and can grow to several megabytes. In production, the median request through our gateway is about 290 KB. Long-lived streams. Responses stream token by token for seconds to minutes. For most of that time the gateway waits on the provider. So it holds many connections open at once, and each one costs memory for as long as it's open. CPU work on every request. A pure proxy would only copy bytes. We also read the request to resolve the model and capture its parameters for the trace, read every streamed event to pick up token usage, and serialize the telemetry we send to Langfuse, all of it on bodies of hundreds of kilobytes. Bursty traffic. Traffic isn't smooth. A team starts its day and many agents open sessions at once, a batch job fans out over many workers, or a provider recovers from an incident and every client retries together. These spikes are exactly when the gateway is needed most. From that we took three properties to design and measure for: - Throughput: how many requests and open streams one instance can carry. - Overhead: how much time the gateway adds before the first token reaches the agent. - Memory footprint during spikes: a process that exceeds its container's memory limit is killed, and every stream it was serving is cut off mid-response. Picking a runtime picking-a-runtime The Langfuse platform is written in TypeScript and runs on Node.js. Python is the other language much of AI engineering happens in. We know both well, so both were natural candidates. For this workload, both work against us. We already use Rust in the Langfuse ingestion pipeline, so we considered it for the gateway too. Node.js: one thread runs all JavaScript nodejs-one-thread-runs-all-javascript Node.js runs all JavaScript on a single thread, the event loop. Waiting on I/O doesn't block it, which is why Node handles many idle connections well. CPU work does: while JSON.parse works through a 500 KB request body, no other JavaScript runs, so chunks from the provider aren't forwarded and new requests aren't read. Under load, each request waits for the CPU work of every request ahead of it, and latency grows with load even though each request needs the same work. We've hit this first-hand in the Node.js ingestion pipeline of Langfuse itself. A gateway makes it worse: the bodies are larger, there is CPU work on every request and every stream, and a stalled event loop delays tokens for every agent connected to that process. Python: the global interpreter lock python-the-global-interpreter-lock In the standard CPython interpreter, the global interpreter lock GIL lets only one thread run Python code at a time. Threads help while code waits on I/O, but not with CPU work like parsing a request body, so one process runs Python on about one core. Python servers like Uvicorn scale with worker processes instead, each with its own interpreter and memory, so memory grows with every worker. Memory under traffic spikes memory-under-traffic-spikes Node.js and Python both manage memory for you, and neither frees it at the moment a request finishes. V8, Node's JavaScript engine, reclaims memory with a garbage collector that runs when it decides to. During a burst, garbage piles up faster than it's collected, and the process uses much more memory than its live requests need. CPython frees most objects right away through reference counting, but its allocator keeps freed memory for reuse and rarely returns it to the operating system, so a process stays at the size of its largest burst, and the container limit has to cover that peak. A gateway that buffers responses, for logging for example, holds even more. Either way, the memory a process uses isn't a simple function of the requests it is serving right now, which makes it hard to predict. That matters because of how containers fail. When a container exceeds its memory limit, the kernel kills a process. For a gateway, every stream on that process ends mid-response, and every agent on it sees a dropped connection and has to retry a long turn. That happens during a spike, the worst moment to lose streams. Why we picked Rust why-we-picked-rust Rust addresses each of these problems in the language and its runtime: - True multithreading. The gateway runs on Tokio, Rust's async runtime, with one worker thread per core. Requests are spread across all cores, and an idle thread takes queued work from a busy one. CPU work on one request doesn't hold up streams on the other cores. Longer jobs, such as encoding a batch of telemetry, run on a separate thread pool, away from request handling. - Cheap async tasks. Each request is an async task: a small state machine that uses no CPU while it waits and much less memory than an operating system thread. Thousands of open streams that are mostly waiting cost little. - Memory freed deterministically. Rust has no garbage collector. Every value has one owner, and its memory is freed when the owner goes out of scope. When a request finishes, its memory is released right then, not when a collector gets to it. That makes memory predictable enough to budget, which we come back to below. - A single compact binary. The gateway compiles to one native binary. There is no interpreter or runtime to install, so the container image is about 25 MB to download and simple to run. - Vertical and horizontal scaling. One process uses every core it's given, so an instance with more CPU and memory carries more traffic. And because the gateway keeps no state, you can also add instances behind a load balancer. Designing for operators designing-for-operators Langfuse is available on Langfuse Cloud or self-hosted. For both, we wanted a gateway that is simple to run and scales predictably. To check that the design delivers, we ran the same workload through our Rust gateway running on axum, a Node.js gateway running on Hono and a Python gateway running on FastAPI, each on two vCPUs and 4 GiB of memory.