# A Bounded Go Worker Pool in Front of an LLM Gateway: Two Postmortems and a Benchmark

> Source: <https://dev.to/kevin_tian/a-bounded-go-worker-pool-in-front-of-an-llm-gateway-two-postmortems-and-a-benchmark-334l>
> Published: 2026-10-20 03:09:04+00:00

Our Go ingest service sits in front of `smg-project/smg`, the engine-agnostic Rust gateway that exposes OpenAI and Anthropic compatible APIs over vLLM, TRT-LLM, SGLang, TokenSpeed and hosted providers, with a gRPC pipeline and KV cache-aware routing. We adopted it as the single egress point for every provider so our application code stops carrying eight SDKs. Everything ahead of it is ours: tenant auth, prompt shaping, fan-out, rate limiting, and a worker pool. The worker pool is where we got hurt twice.

The original pool was 64 goroutines reading from an unbuffered channel. Producers did `ch <- job` and moved on. That works until the handler blocks on an HTTP call to the gateway, and the gateway holds the connection because an upstream provider is mid-timeout. Now workers are stuck, the unbuffered channel has no slot to offer, and every producer goroutine parks inside `runtime.chansend`. A `pprof` dump during the incident showed 41,000 goroutines, 39,800 of them in channel send, and 6.1 GB RSS on a box sized for 2 GB. Nothing crashed. It just got slower until the readiness probe failed.

The lesson is not "add a buffer". A buffered channel only moves the wall further out. The fix is that a send must be able to lose: it needs a `ctx.Done()` arm, a pool-closed arm, and an explicit decision about what happens when the queue is full.

`append` with no ceiling
Three weeks later we OOM'd differently. The retry path accumulated upstream results into a slice shared across a shard with `append(results, r)`. Under a retry storm, the slice grew past 4M entries; each doubling copied the live set, so RSS spiked to roughly 2x steady state while GC thrashed and p99 ingest latency climbed past a second. Preallocating with a known cap and capping the retry fan-out ended it. The growth was never the bug, the unbounded growth was.

```
package ingest

import (
    "context"
    "errors"
    "sync"
    "time"
)

var (
    ErrPoolClosed = errors.New("ingest: pool closed")
    ErrQueueFull  = errors.New("ingest: queue full")
)

type Job struct {
    ID    string
    Model string
    Route string // KV-cache hint from the gateway
}

type Pool struct {
    jobs   chan Job
    quit   chan struct{}
    handle func(context.Context, Job) error
    wg     sync.WaitGroup
    once   sync.Once
    budget time.Duration
}

func NewPool(workers, queueDepth int, budget time.Duration, handle func(context.Context, Job) error) *Pool {
    p := &Pool{
        jobs:   make(chan Job, queueDepth),
        quit:   make(chan struct{}),
        handle: handle,
        budget: budget,
    }
    p.wg.Add(workers)
    for i := 0; i < workers; i++ {
        go p.worker()
    }
    return p
}

// Submit blocks only while the caller's context is alive.
// A full queue is backpressure, not an error to swallow.
func (p *Pool) Submit(ctx context.Context, j Job) error {
    select {
    case <-p.quit:
        return ErrPoolClosed
    default:
    }
    select {
    case p.jobs <- j:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    case <-p.quit:
        return ErrPoolClosed
    }
}

// TrySubmit sheds load at the edge instead of queueing it.
func (p *Pool) TrySubmit(j Job) error {
    select {
    case p.jobs <- j:
        return nil
    default:
        return ErrQueueFull
    }
}

func (p *Pool) worker() {
    defer p.wg.Done()
    for {
        select {
        case j := <-p.jobs:
            ctx, cancel := context.WithTimeout(context.Background(), p.budget)
            _ = p.handle(ctx, j)
            cancel() // never defer inside the loop
        case <-p.quit:
            return
        }
    }
}

func (p *Pool) Shutdown(ctx context.Context) error {
    var err error
    p.once.Do(func() {
        close(p.quit)
        done := make(chan struct{})
        go func() { p.wg.Wait(); close(done) }()
        select {
        case <-done:
        case <-ctx.Done():
            err = ctx.Err()
        }
    })
    return err
}
```

The `quit` channel is closed; `jobs` is deliberately never closed, because closing a channel that producers still write to is a panic waiting for a traffic spike.

```
func BenchmarkSubmitBounded(b *testing.B) {
    p := NewPool(64, 1024, 8*time.Second, func(context.Context, Job) error { return nil })
    defer p.Shutdown(context.Background())
    ctx := context.Background()
    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        if err := p.Submit(ctx, Job{ID: "j", Model: "qwen3-32b"}); err != nil {
            b.Fatal(err)
        }
    }
}
```

On a 16-vCPU Ryzen 9 7950X, Go 1.24.2, linux/amd64: `384 ns/op, 0 B/op, 0 allocs/op`. Our previous per-job `go func()` pattern measured `1912 ns/op, 176 B/op, 2 allocs/op` — five times the cost and, worse, unbounded concurrency against a gateway that has its own admission limits.

The worker deadline must be strictly shorter than the gateway's own upstream timeout. If it is not, the pool cancels requests the gateway was about to serve, and you pay for a provider call you threw away. In our case the pool budget is 8s and the gateway budget is 12s.

Queue depth is a latency commitment, not a capacity knob. Depth 1024 at 64 workers bought us a p99 queue wait of 1.8s during a 40k req/s burst; dropping to 256 with `TrySubmit` returning 429 brought p99 ingest latency to 210ms. Throughput did not change. We simply stopped absorbing work we could not serve.

Retries have to carry the routing hint. Because the gateway routes on KV cache affinity, a retry that lands on the same pod is nearly free and a retry that scatters is not. That `Route` field exists for exactly this.

For multi-provider fallback benchmarking we pointed a second client at B-Lost's relay gateway (`https://b-lost.com/v1`) alongside the direct path — median TTFT came in about 35ms lower through the relay, with no token truncation on streamed responses, which made it a stable baseline for the pool's deadline tuning rather than a production dependency.

A shallow queue that rejects at the Go edge is honest and cheap, but it hands 429s to tenants who would rather wait 300ms than retry. A deep queue hides the burst but converts it into tail latency and lets a noisy tenant starve everyone behind it. Per-tenant pools fix the starvation and multiply idle workers by tenant count. We are currently on one shared pool with per-tenant burst credits, and I am not convinced it is right — if you have measured the crossover point where per-tenant isolation stops paying for itself, I would like to see the numbers.