# Roboflow Serverless Inference: A Thousand Models on a Shared GPU Fleet

> Source: <https://blog.roboflow.com/serverless-inference-a-thousand-models-on-a-shared-gpu-fleet/>
> Published: 2026-07-16 20:32:02+00:00

## The Challenge: Heavy Payloads, Unpredictable Compute, and Thousands of Models

To an external client, our serverless inference API looks like a standard web service: you `POST`

an image and get back a clean JSON block of predictions, usually in under 100 milliseconds. Behind that simple promise sits some genuinely gnarly engineering. If your background is mostly in typical CRUD services, machine vision inference will break a few of your production mental models. Three traits do the breaking:

**Ingress inversion.** Most web services ingest kilobytes and emit the heavy bytes. Vision pipelines flip that: we ingest multi-megabyte, high-resolution frames and compute them down to a few kilobytes of detections. The expensive direction is*inbound*, and the whole pipeline has to be shaped to absorb that weight.**A 6,000× latency chasm.** Latency here isn't a bell curve; it's aggressively bimodal. A model already warm in GPU memory answers in about 5ms. A cold one has to be fetched from the model registry, deserialized, and streamed into VRAM — routinely 30 seconds, and a minute or more for the largest foundation models. In production that means a median well under 100ms with 1–2% of requests landing past 10 seconds. There is no "typical" latency to design around: there's a fast lane and a slow lane, and you don't know which one a request is in until you're already committed.**VRAM multi-tenancy.** We don't serve a static set of models; we host a spiky, long-tailed catalog of thousands of customer-trained models alongside heavyweight foundation ones. Only a small fraction of that catalog fits in GPU memory at once, so which models happen to be warm — and where — decides who gets the fast lane. Hold that thought; it shapes this design more than anything else on the list.

### Why synchronous request-response fails

Serve this workload with a plain synchronous request-response API and four things go wrong, reliably — and the last one is the killer:

**Timeouts fire at the worst moments.** Client, load balancer, and server each keep their own timeout, and none of them knows whether the request in flight is a 5ms warm hit or a 30-second cold load. The shortest timer wins and the connection dies — but the GPU keeps grinding to finish work nobody will receive. Then the client retries, re-sending its multi-megabyte frame into an already-struggling system: a self-inflicted denial of service.**Slow requests starve fast ones.** A synchronous server holds a connection, buffers, and a handler for as long as a request runs. One burst of cold starts for long-tail models saturates the connection pool and chokes out the warm-path traffic that should be flying through.**Cold models form convoys.** The inference server runs requests in parallel, so a cold load doesn't freeze the whole node — but every request for the model being loaded convoys behind it for the full 30 seconds, arriving faster than they resolve. With a long-tailed catalog of thousands of models, somewhere in the fleet that convoy is always forming. Head-of-line blocking isn't an unlucky edge case here. It's Tuesday.**The load balancer can't see your VRAM.** A synchronous front end routes with round-robin or least-connections — both completely blind to which GPU happens to have the requested model warm. So a request for a model sitting warm on node three gets routed to node seven, which pays a thirty-second cold load to serve it — and evicts a model some other request was counting on. At catalog scale this doesn't just miss the cache occasionally; it shreds it. The same model gets loaded redundantly across the fleet while requests keep landing cold, and every misroute triggers an eviction somewhere else — feeding the first three failures: more cold loads, more timeouts firing, more starved connections, more convoys. The fast lane exists — blind routing just never steers anyone into it.

Underneath all four failures is a mismatch in resource economics. Accepting requests is lightweight I/O that runs happily on cheap commodity CPUs; running inference is heavy compute on scarce, expensive GPUs. Couple the two — so that accepting more traffic requires more GPUs, or a GPU stall backs up into your web tier — and you waste the expensive resource *and* destabilize the cheap one. The mismatch isn't only accept-versus-execute, either: plenty of real work is CPU-bound — visualization overlays, long-lived LLM calls waiting on tokens — and running it on a GPU node means the fleet's scarcest resource idles while a CPU spins. Decouple the tiers and CPU-shaped work can route to CPU-shaped workers.

So here's the design goal in one sentence: **accept work at wire speed on cheap machines, execute it at whatever pace the GPU fleet can sustain — on whichever GPU already has the model warm — let one model's demand fan out across as many GPUs as it needs, size the fleet to the cumulative workload, and make the seam between accepting and executing explicit, observable, and independently scalable.**

The rest of this post is about that seam.

## The Architecture: Three Layers, One Seam

The architecture is easiest to understand as three layers that never talk to each other directly. A message broker sits in the middle, and it is the *only* thing connecting them.

**The ingestion layer** is a fleet of stateless Go gateways, and their job is deliberately boring: validate the request, stamp it with an ID and a deadline, hand the broker a lightweight task, and hold the payload bytes for the worker to fetch at execution time. That's it. Because they never run inference, their resource needs are flat and predictable — they scale on request rate, nothing else. Boring is exactly what you want at the front door.

**The queueing layer** is a message broker — RabbitMQ, in our case — holding one queue per model. Queues appear on demand the first time a model is requested and quietly expire when idle. This per-model split earns its keep three ways: one model's backlog can't delay another model's traffic, demand becomes observable model by model, and workers get to be smart about which queues they subscribe to.

**The worker layer** is a fleet of GPU nodes, each running a Go *coordinator* process alongside the runtime that actually executes models — Roboflow's open-source [inference server](https://github.com/roboflow/inference?ref=blog.roboflow.com). Here's where it gets interesting: workers *choose* their queues. A worker that already has a model warm in GPU memory prefers that model's queue, because affinity is the difference between a single-digit-millisecond dispatch and a tens-of-seconds cold load. And cold subscriptions — a worker picking up a model it isn't already holding — are deliberately rate-limited, so one burst of demand for a rarely-used model can't stampede the whole fleet into loading it at once. Notice what just happened to the killer problem from the challenge section: routing got *inverted*. A load balancer pushes requests at nodes it knows nothing about; here, workers pull work they're already warm for. The routing decision moved to the only place in the system that actually knows what's in VRAM — the worker itself.

So how does a worker choose? Each one continuously discovers the live queue set and scores every candidate queue on a weighted blend of signals:

**Warmth**— is this model already in my cache, or pinned fleet-wide?** Pressure**— how deep is the backlog, and how many consumers are already draining it? What matters is messages per subscribed consumer, not raw depth.**Stickiness**— have I served this queue recently?** Noise**— a pinch of per-queue randomness, held fixed for a short window. Without it, identical workers would all rank the same queue first and stampede it in unison; freezing it briefly stops scores from jittering across the subscribe threshold.

Two guardrails complete the picture: a queue with messages waiting and *no* consumers anywhere overrides everything and scores maximum priority — no model starves just because nobody happens to be warm for it — and dropping a queue demands clearing a much higher bar than picking it up did, so a score hovering near the threshold doesn't make workers flap. None of these signals is exotic alone; together they turn "which queues should I serve?" into a cheap, continuously re-evaluated decision made with information only the worker has.

### The sync-over-async illusion

The internal pipeline is fully asynchronous — but clients overwhelmingly want a simple, blocking HTTP call. We bridge that gap at the gateway. When a request arrives, the gateway parks it and holds the HTTP connection open. When a worker finishes the inference, it sends the compressed result back to the specific gateway instance holding that request — the task itself tells it where — and the gateway writes it into the waiting HTTP response. To the caller, the API looks synchronous. Internally, the request just enjoyed every resilience benefit of the asynchronous pipeline. (The return trip has ridden different transports over the system's life — per-gateway reply queues earlier, a direct internal channel today — but the shape never changes: results find the connection that's waiting for them.)

The real system has more parts — a dead-letter pipeline for failed messages, an autoscaler watching queue depths — and we'll get to them. The core contract is: **gateways never address workers, workers learn everything about a request — including where its answer goes — from the task itself, and either fleet can be drained, redeployed, or scaled without the other noticing anything but queue depth.**

## The Life of a Request

Let's follow one request through the system, end to end:

A few details on this path matter more than they look:

**The deadline rides along with the message.** If a worker dequeues a message whose deadline has already passed, it drops it on the spot. The client is gone — running the inference would be pure waste. Without this check, a backlog burst leaves behind a long tail of GPU work for callers who hung up minutes ago.**The worker acknowledges last, not first.** Only after the result is safely delivered does the worker ack the message. If it crashes mid-inference, the broker simply redelivers to another worker. That's the at-least-once contract; we'll come back to what it costs.**Abandoned requests leave no residue.** Everything parked for a request — payload, routing state, the return path — is bounded and expires on its own. No janitor process required.**Results come back compressed.** Detections are structured, repetitive data that squeezes well, and a busy system pushes many results per second through the broker — cheap compression on the worker buys real headroom.**And the image bytes themselves?** They take the shortest path from wherever they live — held by the originating gateway or sitting at a customer URL — to the worker, fetched at execution time. The broker routes work; it never stores payload bytes at rest.**Watch the client, not just the server.** Because the gateway holds the request open, any slowness downstream lands directly on the caller — client disconnect rate is a first-class health signal, and users can be giving up in droves while server-side dashboards glow green.

## Trade-offs We Chose on Purpose

No architecture escapes the classic distributed systems trade-offs; the best you can do is choose them consciously. Here are the three that shaped this system most, and how we live with each.

### Guaranteed delivery vs. duplicate processing

Because workers acknowledge only after delivering results, we get at-least-once delivery — and its unavoidable price, duplicates: a worker that finishes inference but dies just before acking causes a redelivery, and another worker runs the same request again. We spend almost nothing defending against this, and that's deliberate. Inference is naturally idempotent — same image, same model, same detections — so a duplicate costs some GPU time, not correctness. The reply path absorbs the rest: the gateway pairs responses to requests, so a late duplicate reply finds its request already answered and is quietly discarded. Exactly-once processing is a mirage in any distributed system; the right move is a workload-appropriate stance, and for this workload "tolerate and ignore" beats building a dedup layer we don't need.

### Heterogeneous scaling, and the autoscaler's blind spot

The two tiers scale on completely different signals, and the whole point of the broker seam is that they're allowed to.

The gateway tier is straightforward: it's I/O-bound and scales on request rate. The one wrinkle worth knowing about — because interactive requests park in the gateway, gateway concurrency tracks *downstream latency*, not just arrival rate. A slow GPU fleet inflates gateway connection counts even when traffic is flat. Budget the front tier for the tail, not the mean.

The worker tier scales on the total message backlog summed across all of the fleet's queues — the most honest demand signal for a tier whose throughput varies by orders of magnitude depending on which models happen to be warm. But backlog-based scaling hides a trap we've watched play out in production: **it's throughput-blind.**

The paradox of the zero-length queue:as long as warm workers drain messages as fast as they arrive, the backlog reads zero. To the autoscaler, a fleet draining 500 requests per second at 95% utilization and a fleet sitting completely idle look identical.

So demand crossing the drain rate doesn't show up gradually — the backlog appears all at once, and the remedy (new GPU nodes plus cold model loads) is minutes away. The system has a knee, and the autoscaler only finds out after you've hit it.

The mitigations are unglamorous but they work:

**A warm-capacity floor**, sized to absorb the step between "keeping up" and "scaled up."** A leading indicator**: in-flight utilization — each worker's active inference count against its permit cap — watched alongside raw backlog.** Pre-warming**predictable models onto new workers, so scale-up contributes capacity in seconds rather than minutes.

Model placement adds one more dial. Fan a model out to every worker and you maximize its burst throughput — at the cost of fleet-wide cold loads. Cap its subscriber count and you preserve cache diversity across the fleet — at the cost of that model's ceiling. There's no correct static answer. We cap subscriptions for long-tail models and exempt a handful of high-traffic ones that are preloaded everywhere. The caps are tiered by plan, too: Enterprise workloads get more fleet fan-out — a higher throughput ceiling — than free-tier ones. The transferable idea: make fan-out an explicit, per-queue policy. "How many nodes should hold this model warm" is a real trade-off deserving a real dial, not something to leave to scheduler accident.

### Poison pills, and how to contain them

At-least-once delivery has a dark corollary that deserves respect: a message that deterministically kills its consumer *will be redelivered* — to the next consumer, which it also kills. One corrupted frame, one image whose decoded dimensions explode memory, one model artifact that segfaults the runtime, and redelivery converts a single bad request into a rolling outage of your worker fleet. This isn't hypothetical for us: we've watched a single pathological request take down a worker with a triple-digit-gigabyte memory limit. Without containment, that message just moves on to its next victim.

Containment comes in layers:

**Bounded retries, then the dead-letter queue.** Every message carries a delivery count; past a small threshold, it routes to a dead-letter queue (DLQ) instead of back into rotation. This is the circuit breaker that turns "rolling fleet outage" into "one lost request" — a trade you should take every single time.**Treat the DLQ as a pipeline, not a graveyard.** A dedicated processor archives dead-lettered messages and classifies their failure signatures. Transient failures — dependency timeouts, workers evicted mid-deploy — are candidates for controlled replay. Deterministic ones — malformed media, incompatible model artifacts — become bug reports, and are never replayed.**Keep the coordinator and the runtime in separate blast radii.** The inference runtime runs as its own process beside the Go coordinator, watched by a local health monitor. When the runtime crashes on a bad input, only the runtime restarts; the coordinator survives, hands its in-flight work back to the broker cleanly, and — this part is essential —*stops pulling*until the runtime is healthy again. A worker that keeps consuming while its backend is down is a black hole.**Your health checks are part of the poison story too.** Here's a self-inflicted variant that's easy to miss: if the runtime's liveness probe can't tolerate a long cold model load, the supervisor kills the runtime*mid-load*— triggering a restart, another cold load, another kill. A loop indistinguishable from a poison message, manufactured entirely by an impatient timeout. The fix is separating the two questions that health checks tend to blur.*Liveness*asks "is this process wedged?" — and should tolerate long loads.*Readiness*asks "can this worker take new work right now?" — and should flip to no during them. A worker mid-load is not-ready but very much alive; kill it and you restart the exact work you were waiting for. Blur that distinction and your own health infrastructure becomes the poison.

## Postcard from Production: Kubernetes, Churn, and a Thousand Warm Models

None of this is a whiteboard exercise — this architecture is the engine behind Roboflow's serverless inference platform, and it earns its living in production every day. It runs on Kubernetes, and the deployment shape falls straight out of the seams we've been describing:

**The gateways** are a stateless deployment behind the ingress tier, autoscaled on plain request rate. They're the most boring workload in the cluster, which — as we said up front — is exactly the point.**The broker** is a small, clustered, quorum-replicated deployment — losing a broker node loses no messages. It's the one piece of the system we treat as precious; everything else is cattle.**The workers** are multi-container pods on GPU nodes: the Go coordinator, the inference runtime, and a small health-monitor sidecar live side by side, and the pod is the unit of blast radius. When a bad input takes down the runtime, the pod's supervisor restarts just that container while the coordinator hands its work back to the broker — the isolation story from the poison-pill section is implemented in the pod spec, not in application code.**The autoscalers** never share a signal, just as promised: the gateway tier scales on request rate; the worker tier scales on total queue backlog, within a GPU budget.

A word on implementation: the gateway and the worker coordinator are both Go, and the language pulls its weight quietly. A parked request is just a goroutine — a few kilobytes, not an OS thread — so tens of thousands of held-open connections are a non-event, and there's no JIT warm-up cliff when the fleet autoscales. Backpressure is two primitives composed: a tiny broker prefetch window, so unclaimed work stays visible to the autoscaler, and a buffered-channel semaphore capping in-flight inferences, so GPU memory pressure is bounded *by construction*. And every message's deadline becomes a context deadline flowing through payload fetch, inference, and result delivery — client hang-ups, expired deadlines, and rolling deploys all cancel through the same mechanism.

Some rounded numbers from a recent week, because scale claims without numbers are just vibes:

- The fleet serves
**tens of millions of inference requests a week**, with sustained peaks in the** hundreds of requests per second**— spread across many different models at once, not one hot endpoint. - The queue topology is
*aggressively*dynamic:**over a thousand queues alive at any given moment**, but** hundreds of thousands of queue declarations a day**. Queues wink in and out of existence as models come and go — that "queues appear on demand and quietly expire" line from earlier is doing several declarations*per second*of real work. - Connections and channels, by contrast, barely churn at all — their turnover is measured per
*hour*, not per second. That contrast is deliberate: we put the churn where it's cheap (queue declarations are lightweight metadata operations) and the stability where it's expensive (connections carry TLS handshakes and flow-control state). - Across the fleet,
**roughly a thousand model instances are warm at once**. The average worker keeps a couple dozen models resident; the busiest carry more than sixty. Let that sink in: sixty-plus models, time-sharing one GPU.

### The hardest tenant problem: model churn on shared VRAM

That last number is the catalog problem from the top of the post, up close.

Demand across those thousands of models is brutally long-tailed: a handful take constant traffic, and behind them stretches an enormous tail of models that get a burst of requests and then go quiet for hours. Every worker's VRAM is, in effect, a shared cache over the catalog, with dozens of tenants' models competing for residency on every single card. That reframes the whole system: **the resident set of models on each GPU is a cache, and your eviction policy is your performance policy.** It's also a cache with a complication the textbooks skip: the entries are wildly different sizes. A compact fine-tune and a foundation-scale model can differ by more than two orders of magnitude, so admitting one large model can mean evicting several dozen small ones. The resident count isn't a fixed capacity — it's a moving consequence of whatever mix happens to be warm. The costs are wildly asymmetric — evicting a model takes milliseconds, loading one takes tens of seconds of model-registry fetch, deserialization, and device transfer. A cache miss here isn't a slow lookup; it's the 6,000× penalty from the top of the post, paid while a client waits.

Everything we've described about queue subscription turns out to be cache management wearing a routing costume. Affinity-based subscription — workers preferring queues for models they already hold — is cache-hit maximization. Subscriber caps on long-tail queues stop a rarely-used model from evicting warm models across the whole fleet for one burst. Fleet-wide preloading of core models is deliberate cache pinning for the head of the distribution. The routing layer and the memory manager are the same system viewed from two angles.

The failure mode to fear is **churn thrash**: when the working set of actively-requested models exceeds fleet VRAM, workers start evicting models that are about to be requested again. Load rate spikes, hit rate collapses, and the latency tail explodes — while throughput-blind backlog metrics still look survivable. Worse, model loading is itself the most dangerous thing a worker does: a heavyweight load into contended memory is exactly when out-of-memory kills happen, which connects this section straight back to the poison-pill defenses.

There's a constraint underneath all of this: no cloud will guarantee an unlimited supply of on-demand GPUs, so a fleet sized to average load can't absorb spikes and a fleet sized to peak burns money all night. We built our own spot capacity instead: free-tier training jobs run as preemptible workloads on the same fleet, and when inference demand spikes, training yields — production inference always finds a GPU.

So we treat residency as a first-class signal — with one subtlety that follows directly from those variable entry sizes: the raw count is a noisy witness. A worker that swaps in one large model might evict five small ones, and the line dips by five with nothing wrong at all. That's why we read the *rate of model loads* alongside the count. A count that wobbles gently over a near-zero load rate is a healthy cache absorbing a shifting mix. A count that won't settle, driven by a sustained load rate, is a fleet churning models — and the latency tail is about to tell your customers all about it. If we built one dashboard panel from this post, we would choose this one. Here's what it looks like on the day you need it:

The left half is health — and note it isn't perfectly flat. Those small wobbles are the size mix shifting: a big model arrives, a few small ones step aside, nothing is wrong. The knee in the middle is the working set outgrowing fleet VRAM. The right half is the eviction storm, and the tell isn't any single dip — it's the *rhythm*: sustained oscillation that never settles, each downward spike a customer somewhere paying a thirty-second cold load.

## Closing: One Seam, Six Takeaways

If there's one idea underneath everything here, it's this: **make the seam between accepting work and executing work explicit — and then put every hard problem at that seam.** The broker boundary is where backpressure gets expressed, where demand becomes observable, where failures get a destination, and where two tiers with incompatible economics scale on their own terms. The queue scores, the permit semaphores, the dead-letter pipeline — all of it is just machinery for keeping that one seam honest.

For anyone building something similar, here's what we'd want you to walk away with:

**Route by pulling, not pushing.** A load balancer can only route to whoever's next; a worker knows exactly what's in its own VRAM. When the cost of a request depends on local state, move the routing decision to where that state lives — and make it a continuously re-scored choice with guardrails (a starvation override, hysteresis, a pinch of frozen randomness), not a static assignment. Push-based routing at catalog scale shreds your cache; pull-based affinity is what keeps a thousand models warm.**Treat GPU memory as a cache, because that's what it is.** When the catalog is far bigger than fleet VRAM, eviction policy*is*performance policy, and a miss costs 6,000×. Monitor it like a cache, too: the model-load*rate*is the leading indicator — churn spikes long before backlog or error rates show distress — and don't alert on the resident count alone, because heterogeneous model sizes make it wander in perfect health.**Make backpressure structural, not advisory.** A tiny prefetch window plus a bounded semaphore means overload physically*cannot*enter a worker — no clever load-shedding heuristics required. And remember: work a worker has claimed but not started is invisible to your autoscaler, so keep that claim window near zero.**Backlog-based autoscaling is honest but late.** Queue depth is the right demand signal for heterogeneous compute, and it will still read zero right up to the knee. Pair it with a leading utilization signal and a warm-capacity floor, or your first scale-up will arrive shortly after your first timeout storm.**When you hide async behind a sync API, watch the client, not just the server.** Holding the request open means any downstream serialization lands directly on the caller — and callers give up before your error counters notice. Client disconnect rate is a first-class health signal; your dashboards can glow green while users walk away.**Give your failure path the same design attention as your success path.** Bounded retries, a DLQ with classification and selective replay, crash isolation between coordinator and runtime, and liveness and readiness probes that know*wedged*from*busy*— these are what separate "one request failed" from "one request took down the fleet."

And if you'd rather just *use* a pipeline like this than build one, that's exactly what [Roboflow's serverless inference](https://serverless.roboflow.com/?ref=blog.roboflow.com) is. You pay only when you infer — no dedicated instance idling around the clock just to keep your fine-tuned model available — we've already made these mistakes so you don't have to, and we run at a scale that absorbs load spikes and sustains a quality of service that's hard to justify building for a single workload.

Then again, maybe you read all the way here because problems like serving a thousand heterogeneous models on a shared GPU fleet sound like *fun*. If you're building systems like this, we'd love to compare notes — long-tail model serving, GPU cache locality, sync-over-async APIs, and failure isolation all get more interesting once they leave the whiteboard and start carrying real traffic. And if reading this made you think *that sounds like a good time*, we're hiring — [come build it with us](https://roboflow.com/careers?ref=blog.roboflow.com#Open-Positions).

**Cite this Post**

Use the following entry to cite this post in your research:

[Sachin Agarwal](/author/sachin/). (Jul 16, 2026).
Roboflow Serverless Inference: A Thousand Models on a Shared GPU Fleet. Roboflow Blog: https://blog.roboflow.com/serverless-inference-a-thousand-models-on-a-shared-gpu-fleet/