vLLM 0.33 ships a request cap, — max-num-queued-reqs, that is off by default. In a simulated 40-call agent burst on 4 slots, a tight cap refused 33 calls and cut P95 time-to-first-token for the rest from 37 s to 2.3 s.
Your self-hosted vLLM server accepts every request, and a coding agent can send 40 in two seconds. In a simulation of that burst, the slowest 5% of calls waited 37 seconds or more for a first token. Here is a one-command check of your install, and the flag that turns that silent wait into a fast 503.
On November 9 the AI Engineer channel posted “Your Self-Hosted LLM Needs to Say No”, by Harshul Jain (Audible) and Parv Khatri (SAP).
Their point: an agent fixing one test can make 40 to 50 model calls in seconds. Every layer lets them through. The gateway counts API keys, the router picks a replica, and the engine finds out it is full only after saying yes.
The missing piece is admission control: deciding before a request takes GPU memory. Admit, defer (HTTP 503, “come back later”), or reject (429 when one caller is over its share, 529 when the pool is full, as Anthropic does).
They cite postmortems from OpenAI (November 2023) and Anthropic (January 2026). Both fixes were about refusing load earlier.
In vllm/v1/core/sched/scheduler.py, add_request() calls self._enqueue_waiting_request(request). With the default first-come-first-served policy, the waiting queue is a deque subclass (FCFSRequestQueue in request_queue.py). Nothing in that path refuses.
vLLM 0.33.0, on PyPI since November 2, has two fields in vllm/config/scheduler.py:
Both default to None: no limit. The docstring says the first "bounds vLLM's otherwise unbounded request queue." Out of the box, vLLM never says no. It says "wait."
This script parses vLLM’s source with Python’s ast module: no GPU, server or API key, just Python 3.9+. It prints the queue-cap defaults and checks which load metrics are registered.
"""queuecap.py - will your installed vLLM ever say no? Reads vLLM's sourcewith ast: queue-cap defaults, and which load metrics are really registered."""import ast, os, sysQUEUE_FIELDS = ("max_num_seqs", "max_num_queued_reqs", "max_num_queued_tokens")# load signals a policy might read; gpu_cache_usage_perc is the old nameSIGNALS = ("vllm:num_requests_waiting", "vllm:kv_cache_usage_perc", "vllm:gpu_cache_usage_perc", "vllm:admission_rejections")def field_defaults(path): """Return {field: default source text} from class-level annotations.""" tree = ast.parse(open(path).read()) consts = {} for node in ast.walk(tree): if isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name): if node.value is not None: consts[node.target.id] = node.value out = {} for name in QUEUE_FIELDS: v = consts.get(name) if v is None: out[name] = "(field not found)" continue if isinstance(v, ast.Call): # Field(default=...) kw = {k.arg: k.value for k in v.keywords} v = kw.get("default", v) text = ast.unparse(v) if isinstance(v, ast.Name) and v.id in consts: text += " = " + ast.unparse(consts[v.id]) out[name] = text return outdef registered_metrics(path): """Every name="vllm:..." keyword passed to a call in the logger.""" names = set() for node in ast.walk(ast.parse(open(path).read())): if isinstance(node, ast.Call): for k in node.keywords: if k.arg == "name" and isinstance(k.value, ast.Constant): if str(k.value.value).startswith("vllm:"): names.add(k.value.value) return namesdef main(argv): root = argv[1] if len(argv) > 1 else "" if os.path.isdir(os.path.join(root, "vllm")): root = os.path.join(root, "vllm") # repo checkout given sched = os.path.join(root, "config", "scheduler.py") logger = os.path.join(root, "v1", "metrics", "loggers.py") if not (os.path.isfile(sched) and os.path.isfile(logger)): sys.exit("usage: python3 queuecap.py /path/to/site-packages/vllm") print("source:", sched) for name, default in field_defaults(sched).items(): flag = "--" + name.replace("_", "-") print(f" {flag:<26} default {default}") if name == "max_num_seqs": print(" (class default; vllm serve may raise it per GPU)") caps = field_defaults(sched) if caps["max_num_queued_reqs"] == "None" and \ caps["max_num_queued_tokens"] == "None": print("verdict: no queue cap by default -> every request is" " accepted and waits") metrics = registered_metrics(logger) print(f"signals ({len(metrics)} vllm:* metrics registered in loggers.py):") for s in SIGNALS: state = "present" if s in metrics else "MISSING - naive scrapers read 0" print(f" {s:<28} {state}")if __name__ == "__main__": main(sys.argv)
Point it at your installed package (find_spec finds it without importing):
VLLM=$(python3 -c 'import importlib.util as u; print(u.find_spec("vllm").origin)')python3 queuecap.py "$(dirname "$VLLM")"
No vLLM installed? The 47 MB source tarball works too:
curl -LO https://files.pythonhosted.org/packages/source/v/vllm/vllm-0.33.0.tar.gztar xzf vllm-0.33.0.tar.gzpython3 queuecap.py vllm-0.33.0/vllm
Here is its stdout for 0.33.0:
source: vllm-0.33.0/vllm/config/scheduler.py --max-num-seqs default DEFAULT_MAX_NUM_SEQS = 128 (class default; vllm serve may raise it per GPU) --max-num-queued-reqs default None --max-num-queued-tokens default Noneverdict: no queue cap by default -> every request is accepted and waitssignals (42 vllm:* metrics registered in loggers.py): vllm:num_requests_waiting present vllm:kv_cache_usage_perc present vllm:gpu_cache_usage_perc MISSING - naive scrapers read 0 vllm:admission_rejections present
Three lines matter. Both queue caps read None. And vllm:gpu_cache_usage_perc is gone. More on that in section 07.
The 128 is a class default; for the API server, arg_utils.py sets max_num_seqs to 1,024 on H100-class and larger GPUs, 256 on others (A100 included). That caps how many run at once, not how many wait.
I have no GPU here, so this number is a simulation, and the script says so. It models the talk’s scenario: 4 slots, 40 agent calls in 2 seconds, seeded random prompt times (0.15–0.6 s) and outputs (median about 130 tokens in this seed, at 25 ms each).
Three policies face the same burst: no cap (the default), slots+8 in flight (12 here), and slots+2 (6 here). Six is 4 running plus 2 waiting, the policy the speakers benchmarked.
"""burst.py - what a queue cap does to a coding-agent burst.Toy engine with SLOTS slots (vLLM's --max-num-seqs). A policy caps in-flightrequests (running + waiting) like --max-num-queued-reqs; over cap = 503.Basis: SIMULATION with seeded random lengths, not a GPU measurement."""import heapq, random, sysdef make_burst(calls, secs, seed=7): rng = random.Random(seed) jobs = [] for t in sorted(rng.uniform(0, secs) for _ in range(calls)): prefill = rng.uniform(0.15, 0.6) # seconds tokens = min(900, int(rng.lognormvariate(5.0, 0.8))) jobs.append((t, prefill, tokens * 0.025)) # 25 ms per token return jobsdef run(jobs, slots, cap): free_at, done_at, ttft, rejected = [0.0] * slots, [], [], 0 for t, prefill, decode in jobs: in_flight = sum(1 for d in done_at if d > t) if cap is not None and in_flight >= cap: rejected += 1 # instant 503, holds no memory continue first = max(t, heapq.heappop(free_at)) + prefill heapq.heappush(free_at, first + decode) done_at.append(first + decode) ttft.append(first - t) return ttft, rejecteddef p95(xs): xs = sorted(xs) return xs[max(0, round(0.95 * len(xs)) - 1)] if xs else 0.0def main(argv): try: slots, calls = int(argv[1]), int(argv[2]) secs, slo = float(argv[3]), float(argv[4]) except (IndexError, ValueError): sys.exit("usage: python3 burst.py SLOTS CALLS BURST_SECS SLO_SECS") jobs = make_burst(calls, secs) print(f"{slots} slots | {calls} calls in {secs:g} s | TTFT SLO {slo:g} s") print(f"{'cap':<16}{'admitted':>9}{'p95 TTFT':>10}{'met SLO':>9}" f"{'goodput':>9}") for cap in (None, slots + 8, slots + 2): ttft, rej = run(jobs, slots, cap) met = sum(1 for x in ttft if x <= slo) name = "none (default)" if cap is None else f"{cap} in flight" print(f"{name:<16}{len(ttft):>6}/{calls:<2}{p95(ttft):>9.2f}s" f"{met:>9}{met / calls:>9.0%}") print("goodput = calls that met the SLO / ALL calls, rejected included") print("basis: simulation, seed 7")if __name__ == "__main__": main(sys.argv)
python3 burst.py 4 40 2 2
4 slots | 40 calls in 2 s | TTFT SLO 2 scap admitted p95 TTFT met SLO goodputnone (default) 40/40 36.73s 5 12%12 in flight 13/40 11.48s 5 12%6 in flight 7/40 2.27s 6 15%goodput = calls that met the SLO / ALL calls, rejected includedbasis: simulation, seed 7
With no cap, all 40 calls get in, and the P95 time-to-first-token (TTFT, the wait for the first word) is 36.7 seconds. Only 5 calls meet the 2-second target.
With 6 in flight, 33 calls get a 503 immediately. The 7 that get in all see a first token within 2.3 seconds (with 7 calls, P95 is simply the slowest). A loose cap of 12 still leaves P95 at 11.5 s, meeting the target no more often; the cap must sit close to the slot count.
The honest part: goodput (calls that met the target divided by all calls, rejected included) barely moves, 12% to 15%. A cap creates no GPU capacity. It buys a fast, honest answer.
With 8 slots, python3 burst.py 8 40 2 2, the default's P95 drops to 15.6 s and a 10-in-flight cap gives 1.3 s.
This is a toy with no batching speed-ups or retries, so every 503 counts as a lost call. The speakers’ real-GPU run (Qwen 2.5 0.5B, A10, 4 sequences) is better evidence: two waiting cut P95 TTFT from about 2.5 s to about 1 s.
Turn it on at launch (flag from 0.33.0’s arg_utils.py; I have not run this on a GPU):
vllm serve Qwen/Qwen3-8B --max-num-seqs 256 --max-num-queued-reqs 288
The docstring’s sizing rule: data_parallel_size * max_num_seqs plus the queue depth you tolerate. One replica: 256 running plus 32 waiting is 288.
With a latency target, the token cap is sharper: target_TTFT * prefill_throughput. A 2-second target at 10,000 prefill tokens per second gives --max-num-queued-tokens 20000.
Then watch the vllm:admission_rejections counter (scraped as ..._total), which counts rejections by reason.
The speakers’ favorite result was a silent failure: if the engine stops reporting a policy’s signal, a careless scraper reads zero, and a “reject above 90% KV” rule admits everything.
vLLM has exactly this trap. The 0.8.5 source registers vllm:gpu_cache_usage_perc. In 0.33.0, loggers.py registers 42 vllm:* names, and the old name appears nowhere in the package. The current name is vllm:kv_cache_usage_perc. A dashboard or autoscaler using the old name gets nothing, maybe read as 0%.
On their small model, KV usage peaked at half a percent while the queue hit 24 deep. Watch vllm:num_requests_waiting, not just KV usage.
The talk’s client-side advice, for vLLM:
Reading the other engines’ source by hand:
+------------------+-------------------+| engine | queue cap default |+==================+===================+| vLLM 0.33 | none || SGLang (main) | none || llama.cpp server | none || Ollama | 512, only on load |+------------------+-------------------+
SGLang’s --max-queued-requests defaults to None. llama.cpp's server queue is a std::deque with no size check. Ollama's OLLAMA_MAX_QUEUE defaults to 512 and answers "server busy" with a 503, but sched.go applies it only when the model isn't loaded yet or needs a reload. Calls to a loaded model skip it and wait without limit.
Reproduce it: queuecap.py prints section 03's output and burst.py section 04's table. The real-GPU figures and postmortems come from the talk.
vLLM, SGLang and llama.cpp accept every request by default, and so does Ollama once its model is loaded. vLLM 0.33 has the fix built in, but you must turn it on. A cap won’t add capacity; it turns silent waits of 37 seconds and more into a fast 503 your agent can act on.
What does your agent do today when your own model server answers 503?
If this saves you an outage, a clap helps others find it.
Follow me for daily LLM deep-dives.
Why Your Self-Hosted vLLM Server Never Says No was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.