Member-only story
Is Lying to You (sendfile/splice) Your HTTP middleware is probably copying every byte your AI gateway serves. Here’s the throughput you’re leaving on the table — measured, not guessed.
I run an inference gateway in front of our model servers. Last month we had a p99 latency spike during a traffic burst — requests that normally returned fast were stalling, and the CPU on the gateway boxes were pegged while the GPUs sat idle.
My first instinct, like an idiot, was to blame the model. “The 70B is slow under load,” I told my team. We spent two days tuning batch sizes and KV-cache before I finally looked at the gateway itself.
The model wasn’t the problem. The bytes were.
Every response body — multi-megabyte tensors, streamed tokens, the works — was being copied from kernel space into a userspace buffer and back out again, because of a logging middleware wrapper I’d written three months earlier and never thought about again. The gateway was a very expensive, very slow photocopier.
This is the io.Copy
lie. And if you serve any kind of high-throughput traffic — AI inference, file downloads, video — it's costing you real money.