{"slug": "roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does", "title": "Roboflow Serverless vs. Self-Hosted Inference: What a 90-Second Traffic Spike Does to One GPU Server", "summary": "A single self-hosted GPU server running Roboflow's open-source inference server version 1.7.4 stopped returning responses after a 90-second traffic burst to 20 requests per second, while Roboflow Serverless handled the identical load without interruption, according to a Roboflow benchmarking exercise. The self-hosted server on an NVIDIA L40S GPU held 5 requests per second at roughly one-second median latency, but during the burst its successful requests flattened to 13 to 15 per second and median latency climbed toward the 20-second client timeout, with no responses returned even after traffic dropped back to 5 requests per second. The test ran a SAHI-based vehicle-counting Workflow on 1280×1280 aerial frames, splitting each frame into 16 overlapping 427×427 tiles for 16 model inferences per request, or 320 model calls per second at peak, driven by Locust from a Google Cloud VM.", "body_md": "When deploying a computer vision workflow in production, you need underlying infrastructure that can guarantee consistent and reliable performance even when the request rate spikes. In this article, we’ll compare how two options, serverless and self-hosted GPUs, perform.\n\nIf you run real-time vision inference on your own GPUs, you size the deployment for the request rate you expect. When conducting a benchmarking exercise below, one GPU server handled 5 requests a second without any trouble. When I sent it 20 per second for 90 seconds, its median latency climbed to the 20-second timeout, and by the end of the three-minute run it wasn't returning any responses, even with traffic back down to 5 requests per second. Roboflow Serverless got the exact same traffic at the exact same moment and kept responding to inference requests during the whole process.\n\nCapacity planning is stricter for GPU inference than for general compute. When CPU load rises, autoscaling can usually add instances in time. But on-demand GPUs are often unavailable, and a new inference server can take many minutes to start, so a real-time GPU workload has to be sized right ahead of time. On the other hand, a serverless system takes that capacity planning off your hands. Below I walk through the test setup, the recorded run, and what's behind the serverless side.\n\nWhen benchmarking vision infrastructure, running a toy model on a small image wouldn't tell you much about load. So I conducted a test using something heavier: counting vehicles in 1280×1280 aerial frames of a parking lot.\n\nRun the vehicle detector once on the whole frame and it finds two. The model takes a 512×512 input, so the frame gets squeezed down and most cars end up a few pixels wide. This results in poor vehicle detection model performance.\n\nThe Workflow gets around that with SAHI (opens in a new tab). It cuts the frame into 16 overlapping 427×427 tiles, runs the detector on every tile, merges the boxes where tiles overlap and counts what's left. On this frame that's 102 vehicles, even exceeding the 100 a person labeled.\n\nThat's a much better answer, and a much heavier request. The client still makes one API call to the Workflow, but behind it sit 16 model inferences plus the stitching. That's the kind of work I wanted to throw at both sides.\n\nSame Workflow, same traffic, one URL apart\n\nOne side was Roboflow’s hosted inference product: serverless.roboflow.com. The other was a single cloud VM with an NVIDIA L40S (the same GPU model serverless runs on) running our open-source inference server (opens in a new tab), version 1.7.4 with default settings. Both used the same Locust (opens in a new tab) file with the same load test configuration, and the only thing I changed between them was the host URL. Locust ran on a Google Cloud VM rather than my laptop, which keeps local Wi-Fi out of the numbers.\n\nEach simulated user acts like a camera. It sends a frame every 4 seconds and doesn't wait for the last answer before sending the next one, because a real camera doesn't slow down when your server does.\n\nThe run starts with a minute at 20 users, which works out to 5 requests a second. Then comes a 90-second burst to 80 users, or 20 requests a second. At 16 model calls per request, that's 320 model calls a second.\n\nAfter the burst, it drops back to 20 users for 30 seconds. Any answer slower than 20 seconds counts as a failure, because by then the client has given up.\n\nWhat the recorded run looked like\n\nFor the first minute the two lines sit right on top of each other. Both sides handle 5 requests a second at a median of about a second. If you'd sized one server for this traffic, you'd feel good about it.\n\nThen the burst hits. The self-hosted server's successful requests flatten out around 13 to 15 a second while 20 a second keep arriving. Its median latency starts climbing and keeps climbing, in a fairly straight line, for the next minute.\n\nTo me that ramp looks like a backlog building up, with every new frame waiting behind the ones already in line. With 20 requests arriving and about 14 finishing each second, the line would grow by about 6 requests every second. I didn't measure the queue directly, so treat that as my reading of the chart. Nothing's wrong with the server, though: it's one GPU with a fixed amount of throughput, and the burst asked for more than that.\n\nAbout a minute into the burst, the self-hosted median hits the 20-second timeout. Shortly after, its successful requests drop to zero. They stay at zero after the burst is over, even with traffic back down to 5 a second.\n\nLocust's own charts fill in the rest. Failures on the self-hosted side start late in the burst and peak after the user count has dropped back to 20, at a rate well above the 5 requests a second still being sent. So most of those failures are frames from the burst, still timing out after it ended.\n\nServerless had its own wobble. Its median jumped to about 3 seconds early in the burst, then settled around 1.3 seconds while it answered roughly 16 to 24 requests a second. That's in line with how serverless scales: it brings more capacity online within seconds of a spike like this. Once the burst ended, it dropped to about half a second.\n\nOver all three minutes, serverless returned 2,273 successful requests at a median latency of 1.4 seconds. The self-hosted server returned 1,095. Its whole-run median was 20 seconds, which is just the timeout, and Locust marked 50% of its requests as failures against 0% (rounded) for serverless.\n\nTwo ways to fix the self-hosted side\n\nThe first is to provision for the peak. Run enough GPUs to absorb the biggest burst you expect, plus some headroom. Most of that sits idle between bursts, and you pay for it anyway.\n\nThe second is to autoscale. When I set up this demo, getting the GPU VM booted, the inference container pulled and the server answering took about five minutes. The burst lasted 90 seconds, so even an autoscaler that reacted to the instant traffic jump would have delivered its new GPU long after the burst was over.\n\nMore cameras aren't the only thing that causes a spike. Maybe a new production line comes online, or someone kicks off a backfill of last night's footage. Or someone changes the Workflow to cut each frame into more tiles, which adds GPU work to every request without changing the request rate at all.\n\nSelf-hosting is a reasonable call when your traffic is steady and predictable. With bursty traffic it gets a lot harder to size.\n\nWhat's behind the serverless side\n\nServerless runs the same open-source inference server you can run yourself. What's different is everything around it.\n\nRequests go into a queue per model, and GPU workers pull from those queues. Workers prefer models they already have loaded in VRAM, because a warm model answers in about 5 ms and a cold load takes around 30 seconds.\n\nThe fleet scales on the total backlog across all models, and it keeps a warm floor of spare capacity so a burst has somewhere to land before any new GPU boots. When inference needs more GPUs, our own training jobs step aside. You pay per image, so that spare capacity isn't something you're paying to keep idle.\n\nIf you run your own GPUs, size them for the day your traffic does something you didn't plan for. If you'd rather not plan for that day, the same Workflow runs on Roboflow Serverless with nothing changed but the URL.", "url": "https://wpnews.pro/news/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does", "canonical_source": "https://roboflow.com/blog/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does-to-one-gpu-server", "published_at": "2026-11-09 18:16:06+00:00", "updated_at": "2026-11-09 18:22:24.072633+00:00", "lang": "en", "topics": ["computer-vision", "ai-infrastructure", "ai-tools"], "entities": ["Roboflow", "Roboflow Serverless", "NVIDIA L40S", "Locust", "SAHI", "Google Cloud"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does", "markdown": "https://wpnews.pro/news/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does.md", "text": "https://wpnews.pro/news/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does.txt", "jsonld": "https://wpnews.pro/news/roboflow-serverless-vs-self-hosted-inference-what-a-90-second-traffic-spike-does.jsonld"}}