Why Your Self-Hosted vLLM Server Never Says No

VLLM 0.33.0, on PyPI since November 2, ships a request cap, max_num_queued_reqs, that defaults to None, so the server queues every request instead of refusing it, according to a November 9 AI Engineer channel talk by Harshul Jain (Audible) and Parv Khatri (SAP). In the speakers' simulated 40-call agent burst on 4 slots, a tight cap refused 33 calls and cut P95 time-to-first-token for the remaining calls from 37 seconds to 2.3 seconds. The talk cites postmortems from OpenAI (November 2023) and Anthropic (January 2026) as evidence that refusing load earlier is the fix.

vLLM 0.33 ships a request cap, — max-num-queued-reqs, that is off by default. In a simulated 40-call agent burst on 4 slots, a tight cap refused 33 calls and cut P95 time-to-first-token for the rest from 37 s to 2.3 s. Your self-hosted vLLM server accepts every request, and a coding agent can send 40 in two seconds. In a simulation of that burst, the slowest 5% of calls waited 37 seconds or more for a first token. Here is a one-command check of your install, and the flag that turns that silent wait into a fast 503. On November 9 the AI Engineer channel posted “Your Self-Hosted LLM Needs to Say No” https://www.youtube.com/watch?v=ntVpbyDTYaQ , by Harshul Jain Audible and Parv Khatri SAP . Their point: an agent fixing one test can make 40 to 50 model calls in seconds. Every layer lets them through. The gateway counts API keys, the router picks a replica, and the engine finds out it is full only after saying yes. The missing piece is admission control : deciding before a request takes GPU memory. Admit, defer HTTP 503, “come back later” , or reject 429 when one caller is over its share, 529 when the pool is full, as Anthropic does . They cite postmortems from OpenAI November 2023 and Anthropic January 2026 . Both fixes were about refusing load earlier. In vllm/v1/core/sched/scheduler.py, add request calls self. enqueue waiting request request . With the default first-come-first-served policy, the waiting queue is a deque subclass FCFSRequestQueue in request queue.py . Nothing in that path refuses. vLLM 0.33.0, on PyPI since November 2, has two fields in vllm/config/scheduler.py: Both default to None: no limit. The docstring says the first "bounds vLLM's otherwise unbounded request queue." Out of the box, vLLM never says no. It says "wait." This script parses vLLM’s source with Python’s ast module: no GPU, server or API key, just Python 3.9+. It prints the queue-cap defaults and checks which load metrics are registered. """queuecap.py - will your installed vLLM ever say no? Reads vLLM's sourcewith ast: queue-cap defaults, and which load metrics are really registered."""import ast, os, sysQUEUE FIELDS = "max num seqs", "max num queued reqs", "max num queued tokens" load signals a policy might read; gpu cache usage perc is the old nameSIGNALS = "vllm:num requests waiting", "vllm:kv cache usage perc", "vllm:gpu cache usage perc", "vllm:admission rejections" def field defaults path : """Return {field: default source text} from class-level annotations.""" tree = ast.parse open path .read consts = {} for node in ast.walk tree : if isinstance node, ast.AnnAssign and isinstance node.target, ast.Name : if node.value is not None: consts node.target.id = node.value out = {} for name in QUEUE FIELDS: v = consts.get name if v is None: out name = " field not found " continue if isinstance v, ast.Call : Field default=... kw = {k.arg: k.value for k in v.keywords} v = kw.get "default", v text = ast.unparse v if isinstance v, ast.Name and v.id in consts: text += " = " + ast.unparse consts v.id out name = text return outdef registered metrics path : """Every name="vllm:..." keyword passed to a call in the logger.""" names = set for node in ast.walk ast.parse open path .read : if isinstance node, ast.Call : for k in node.keywords: if k.arg == "name" and isinstance k.value, ast.Constant : if str k.value.value .startswith "vllm:" : names.add k.value.value return namesdef main argv : root = argv 1 if len argv 1 else "" if os.path.isdir os.path.join root, "vllm" : root = os.path.join root, "vllm" repo checkout given sched = os.path.join root, "config", "scheduler.py" logger = os.path.join root, "v1", "metrics", "loggers.py" if not os.path.isfile sched and os.path.isfile logger : sys.exit "usage: python3 queuecap.py /path/to/site-packages/vllm" print "source:", sched for name, default in field defaults sched .items : flag = "--" + name.replace " ", "-" print f" {flag:<26} default {default}" if name == "max num seqs": print " class default; vllm serve may raise it per GPU " caps = field defaults sched if caps "max num queued reqs" == "None" and \ caps "max num queued tokens" == "None": print "verdict: no queue cap by default - every request is" " accepted and waits" metrics = registered metrics logger print f"signals {len metrics } vllm: metrics registered in loggers.py :" for s in SIGNALS: state = "present" if s in metrics else "MISSING - naive scrapers read 0" print f" {s:<28} {state}" if name == " main ": main sys.argv Point it at your installed package find spec finds it without importing : python VLLM=$ python3 -c 'import importlib.util as u; print u.find spec "vllm" .origin ' python3 queuecap.py "$ dirname "$VLLM" " No vLLM installed? The 47 MB source tarball works too: curl -LO https://files.pythonhosted.org/packages/source/v/vllm/vllm-0.33.0.tar.gztar xzf vllm-0.33.0.tar.gzpython3 queuecap.py vllm-0.33.0/vllm Here is its stdout for 0.33.0: source: vllm-0.33.0/vllm/config/scheduler.py --max-num-seqs default DEFAULT MAX NUM SEQS = 128 class default; vllm serve may raise it per GPU --max-num-queued-reqs default None --max-num-queued-tokens default Noneverdict: no queue cap by default - every request is accepted and waitssignals 42 vllm: metrics registered in loggers.py : vllm:num requests waiting present vllm:kv cache usage perc present vllm:gpu cache usage perc MISSING - naive scrapers read 0 vllm:admission rejections present Three lines matter. Both queue caps read None. And vllm:gpu cache usage perc is gone. More on that in section 07. The 128 is a class default; for the API server, arg utils.py sets max num seqs to 1,024 on H100-class and larger GPUs, 256 on others A100 included . That caps how many run at once, not how many wait. I have no GPU here, so this number is a simulation, and the script says so. It models the talk’s scenario: 4 slots, 40 agent calls in 2 seconds, seeded random prompt times 0.15–0.6 s and outputs median about 130 tokens in this seed, at 25 ms each . Three policies face the same burst: no cap the default , slots+8 in flight 12 here , and slots+2 6 here . Six is 4 running plus 2 waiting, the policy the speakers benchmarked. """burst.py - what a queue cap does to a coding-agent burst.Toy engine with SLOTS slots vLLM's --max-num-seqs . A policy caps in-flightrequests running + waiting like --max-num-queued-reqs; over cap = 503.Basis: SIMULATION with seeded random lengths, not a GPU measurement."""import heapq, random, sysdef make burst calls, secs, seed=7 : rng = random.Random seed jobs = for t in sorted rng.uniform 0, secs for in range calls : prefill = rng.uniform 0.15, 0.6 seconds tokens = min 900, int rng.lognormvariate 5.0, 0.8 jobs.append t, prefill, tokens 0.025 25 ms per token return jobsdef run jobs, slots, cap : free at, done at, ttft, rejected = 0.0 slots, , , 0 for t, prefill, decode in jobs: in flight = sum 1 for d in done at if d t if cap is not None and in flight = cap: rejected += 1 instant 503, holds no memory continue first = max t, heapq.heappop free at + prefill heapq.heappush free at, first + decode done at.append first + decode ttft.append first - t return ttft, rejecteddef p95 xs : xs = sorted xs return xs max 0, round 0.95 len xs - 1 if xs else 0.0def main argv : try: slots, calls = int argv 1 , int argv 2 secs, slo = float argv 3 , float argv 4 except IndexError, ValueError : sys.exit "usage: python3 burst.py SLOTS CALLS BURST SECS SLO SECS" jobs = make burst calls, secs print f"{slots} slots | {calls} calls in {secs:g} s | TTFT SLO {slo:g} s" print f"{'cap':<16}{'admitted': 9}{'p95 TTFT': 10}{'met SLO': 9}" f"{'goodput': 9}" for cap in None, slots + 8, slots + 2 : ttft, rej = run jobs, slots, cap met = sum 1 for x in ttft if x <= slo name = "none default " if cap is None else f"{cap} in flight" print f"{name:<16}{len ttft : 6}/{calls:<2}{p95 ttft : 9.2f}s" f"{met: 9}{met / calls: 9.0%}" print "goodput = calls that met the SLO / ALL calls, rejected included" print "basis: simulation, seed 7" if name == " main ": main sys.argv python3 burst.py 4 40 2 2 4 slots | 40 calls in 2 s | TTFT SLO 2 scap admitted p95 TTFT met SLO goodputnone default 40/40 36.73s 5 12%12 in flight 13/40 11.48s 5 12%6 in flight 7/40 2.27s 6 15%goodput = calls that met the SLO / ALL calls, rejected includedbasis: simulation, seed 7 With no cap, all 40 calls get in, and the P95 time-to-first-token TTFT, the wait for the first word is 36.7 seconds. Only 5 calls meet the 2-second target. With 6 in flight, 33 calls get a 503 immediately. The 7 that get in all see a first token within 2.3 seconds with 7 calls, P95 is simply the slowest . A loose cap of 12 still leaves P95 at 11.5 s, meeting the target no more often; the cap must sit close to the slot count. The honest part: goodput calls that met the target divided by all calls, rejected included barely moves, 12% to 15%. A cap creates no GPU capacity. It buys a fast, honest answer. With 8 slots, python3 burst.py 8 40 2 2, the default's P95 drops to 15.6 s and a 10-in-flight cap gives 1.3 s. This is a toy with no batching speed-ups or retries, so every 503 counts as a lost call. The speakers’ real-GPU run Qwen 2.5 0.5B, A10, 4 sequences is better evidence: two waiting cut P95 TTFT from about 2.5 s to about 1 s. Turn it on at launch flag from 0.33.0’s arg utils.py; I have not run this on a GPU : vllm serve Qwen/Qwen3-8B --max-num-seqs 256 --max-num-queued-reqs 288 The docstring’s sizing rule: data parallel size max num seqs plus the queue depth you tolerate. One replica: 256 running plus 32 waiting is 288. With a latency target, the token cap is sharper: target TTFT prefill throughput. A 2-second target at 10,000 prefill tokens per second gives --max-num-queued-tokens 20000. Then watch the vllm:admission rejections counter scraped as ... total , which counts rejections by reason. The speakers’ favorite result was a silent failure: if the engine stops reporting a policy’s signal, a careless scraper reads zero, and a “reject above 90% KV” rule admits everything. vLLM has exactly this trap. The 0.8.5 source registers vllm:gpu cache usage perc. In 0.33.0, loggers.py registers 42 vllm: names, and the old name appears nowhere in the package. The current name is vllm:kv cache usage perc. A dashboard or autoscaler using the old name gets nothing, maybe read as 0%. On their small model, KV usage peaked at half a percent while the queue hit 24 deep. Watch vllm:num requests waiting, not just KV usage. The talk’s client-side advice, for vLLM: Reading the other engines’ source by hand: +------------------+-------------------+| engine | queue cap default |+==================+===================+| vLLM 0.33 | none || SGLang main | none || llama.cpp server | none || Ollama | 512, only on load |+------------------+-------------------+ SGLang’s --max-queued-requests defaults to None. llama.cpp's server queue is a std::deque with no size check. Ollama's OLLAMA MAX QUEUE defaults to 512 and answers "server busy" with a 503, but sched.go applies it only when the model isn't loaded yet or needs a reload. Calls to a loaded model skip it and wait without limit. Reproduce it: queuecap.py prints section 03's output and burst.py section 04's table. The real-GPU figures and postmortems come from the talk. vLLM, SGLang and llama.cpp accept every request by default, and so does Ollama once its model is loaded. vLLM 0.33 has the fix built in, but you must turn it on. A cap won’t add capacity; it turns silent waits of 37 seconds and more into a fast 503 your agent can act on. What does your agent do today when your own model server answers 503? If this saves you an outage, a clap helps others find it. Follow me for daily LLM deep-dives. Why Your Self-Hosted vLLM Server Never Says No https://pub.towardsai.net/why-your-self-hosted-vllm-server-never-says-no-b0612bfd32e3 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.