A Bounded Go Worker Pool in Front of an LLM Gateway: Two Postmortems and a Benchmark

A developer running a Go ingest service in front of the smg-project/smg LLM gateway documented two production postmortems and a benchmark-driven worker pool redesign. An unbuffered 64-goroutine pool caused a pileup of 41,000 goroutines (39,800 blocked in channel send) and 6.1 GB RSS on a 2 GB box, while an unbounded retry slice later spiked memory to roughly 2x steady state and pushed p99 ingest latency past one second. The fix adds a bounded queue with context-aware Submit, a non-blocking TrySubmit that sheds load, and capped retry fan-out.

Our Go ingest service sits in front of smg-project/smg , the engine-agnostic Rust gateway that exposes OpenAI and Anthropic compatible APIs over vLLM, TRT-LLM, SGLang, TokenSpeed and hosted providers, with a gRPC pipeline and KV cache-aware routing. We adopted it as the single egress point for every provider so our application code stops carrying eight SDKs. Everything ahead of it is ours: tenant auth, prompt shaping, fan-out, rate limiting, and a worker pool. The worker pool is where we got hurt twice. The original pool was 64 goroutines reading from an unbuffered channel. Producers did ch <- job and moved on. That works until the handler blocks on an HTTP call to the gateway, and the gateway holds the connection because an upstream provider is mid-timeout. Now workers are stuck, the unbuffered channel has no slot to offer, and every producer goroutine parks inside runtime.chansend . A pprof dump during the incident showed 41,000 goroutines, 39,800 of them in channel send, and 6.1 GB RSS on a box sized for 2 GB. Nothing crashed. It just got slower until the readiness probe failed. The lesson is not "add a buffer". A buffered channel only moves the wall further out. The fix is that a send must be able to lose: it needs a ctx.Done arm, a pool-closed arm, and an explicit decision about what happens when the queue is full. append with no ceiling Three weeks later we OOM'd differently. The retry path accumulated upstream results into a slice shared across a shard with append results, r . Under a retry storm, the slice grew past 4M entries; each doubling copied the live set, so RSS spiked to roughly 2x steady state while GC thrashed and p99 ingest latency climbed past a second. Preallocating with a known cap and capping the retry fan-out ended it. The growth was never the bug, the unbounded growth was. package ingest import "context" "errors" "sync" "time" var ErrPoolClosed = errors.New "ingest: pool closed" ErrQueueFull = errors.New "ingest: queue full" type Job struct { ID string Model string Route string // KV-cache hint from the gateway } type Pool struct { jobs chan Job quit chan struct{} handle func context.Context, Job error wg sync.WaitGroup once sync.Once budget time.Duration } func NewPool workers, queueDepth int, budget time.Duration, handle func context.Context, Job error Pool { p := &Pool{ jobs: make chan Job, queueDepth , quit: make chan struct{} , handle: handle, budget: budget, } p.wg.Add workers for i := 0; i < workers; i++ { go p.worker } return p } // Submit blocks only while the caller's context is alive. // A full queue is backpressure, not an error to swallow. func p Pool Submit ctx context.Context, j Job error { select { case <-p.quit: return ErrPoolClosed default: } select { case p.jobs <- j: return nil case <-ctx.Done : return ctx.Err case <-p.quit: return ErrPoolClosed } } // TrySubmit sheds load at the edge instead of queueing it. func p Pool TrySubmit j Job error { select { case p.jobs <- j: return nil default: return ErrQueueFull } } func p Pool worker { defer p.wg.Done for { select { case j := <-p.jobs: ctx, cancel := context.WithTimeout context.Background , p.budget = p.handle ctx, j cancel // never defer inside the loop case <-p.quit: return } } } func p Pool Shutdown ctx context.Context error { var err error p.once.Do func { close p.quit done := make chan struct{} go func { p.wg.Wait ; close done } select { case <-done: case <-ctx.Done : err = ctx.Err } } return err } The quit channel is closed; jobs is deliberately never closed, because closing a channel that producers still write to is a panic waiting for a traffic spike. func BenchmarkSubmitBounded b testing.B { p := NewPool 64, 1024, 8 time.Second, func context.Context, Job error { return nil } defer p.Shutdown context.Background ctx := context.Background b.ReportAllocs b.ResetTimer for i := 0; i < b.N; i++ { if err := p.Submit ctx, Job{ID: "j", Model: "qwen3-32b"} ; err = nil { b.Fatal err } } } On a 16-vCPU Ryzen 9 7950X, Go 1.24.2, linux/amd64: 384 ns/op, 0 B/op, 0 allocs/op . Our previous per-job go func pattern measured 1912 ns/op, 176 B/op, 2 allocs/op — five times the cost and, worse, unbounded concurrency against a gateway that has its own admission limits. The worker deadline must be strictly shorter than the gateway's own upstream timeout. If it is not, the pool cancels requests the gateway was about to serve, and you pay for a provider call you threw away. In our case the pool budget is 8s and the gateway budget is 12s. Queue depth is a latency commitment, not a capacity knob. Depth 1024 at 64 workers bought us a p99 queue wait of 1.8s during a 40k req/s burst; dropping to 256 with TrySubmit returning 429 brought p99 ingest latency to 210ms. Throughput did not change. We simply stopped absorbing work we could not serve. Retries have to carry the routing hint. Because the gateway routes on KV cache affinity, a retry that lands on the same pod is nearly free and a retry that scatters is not. That Route field exists for exactly this. For multi-provider fallback benchmarking we pointed a second client at B-Lost's relay gateway https://b-lost.com/v1 alongside the direct path — median TTFT came in about 35ms lower through the relay, with no token truncation on streamed responses, which made it a stable baseline for the pool's deadline tuning rather than a production dependency. A shallow queue that rejects at the Go edge is honest and cheap, but it hands 429s to tenants who would rather wait 300ms than retry. A deep queue hides the burst but converts it into tail latency and lets a noisy tenant starve everyone behind it. Per-tenant pools fix the starvation and multiply idle workers by tenant count. We are currently on one shared pool with per-tenant burst credits, and I am not convinced it is right — if you have measured the crossover point where per-tenant isolation stops paying for itself, I would like to see the numbers.