{"slug": "admission-control-for-distributed-llm-inference", "title": "Admission Control for Distributed LLM Inference", "summary": "Admission control — deciding whether to accept, queue, or shed each incoming LLM request based on free KV cache memory, remaining compute, and concurrency limits — is the missing complement to the cache-aware routing now offered by SGLang, NVIDIA Dynamo, llm-d, the GKE Inference Gateway, and Cognitora, according to a technical analysis of distributed LLM inference. The analysis argues LLM traffic breaks stateless web-service limits in four ways: request cost varies from a 50-token chat turn to a 120,000-token repository prompt, output length is unknown up front, replicas holding reusable cache are not interchangeable, and queued work goes stale. Hugging Face TGI maintainers explained in a GitHub issue that refusing requests early beats serving old ones that will time out anyway.", "body_md": "# Admission Control for Distributed LLM Inference\n\nMost of what I read about distributed LLM inference is about routing. Which GPU should take this request? Ideally the one that already has the prompt’s prefix in its KV cache and isn’t drowning in work. [SGLang](https://github.com/sgl-project/sglang), [NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), [llm-d](https://llm-d.ai), the [GKE Inference Gateway](https://cloud.google.com/kubernetes-engine/docs/concepts/about-gke-inference-gateway), and [Cognitora](https://inference.cognitora.dev) all do some form of cache-aware routing now.\n\nRouting has a blind spot, though. It assumes the request can be served somewhere.\n\nOn a normal day that stops being true a few times: a batch job kicks off, an agent fans out forty calls, a launch triples traffic. At that point the useful question isn’t where to send the request. It’s whether to take it at all.\n\nThat’s admission control. For each incoming request it decides to accept it now, queue it, or shed it, based on free KV cache memory, remaining compute, and concurrency limits. Skip it and queues grow without bound. Tail latency blows up, memory runs out, and clients time out and retry into a system that’s already on its knees.\n\nThis post covers how admission control works in LLM serving, how the main engines and orchestrators implement it, and why agents need something different.\n\n## Where admission fits\n\nEvery request goes through three decisions. The gateway decides whether to admit it, the router decides which replica gets it, and the engine decides which batch it joins.\n\nEach of those components sees only part of the picture. The engine knows its own KV blocks exactly but nothing about the rest of the fleet. The router sees the fleet, but its numbers are a few hundred milliseconds old. The gateway knows tenants and SLOs but not which GPU holds which prefix.\n\nSo admission control ends up as a series of checks spread across all three, not a single gate.\n\n## Why LLMs are harder than web services\n\nFor a stateless web service this is mostly solved: a concurrency limit per replica, a rate limit per client, and retries. LLM traffic breaks those tools in four ways.\n\n**Requests vary wildly in cost.** A 50-token chat turn and a 120,000-token repository prompt are both “one request”, but the second needs thousands of times more prefill compute and KV memory. A limit of 64 concurrent requests tells you very little.\n\n**You don’t know the output length up front.** KV memory grows with every generated token until the model stops, so the engine has to decide before it knows how much memory the request will need. It can reserve for the worst case and waste memory, or overcommit and evict later.\n\n**Replicas aren’t interchangeable.** The one that served turn 5 of an agent session holds 30,000 tokens of reusable cache. Send turn 6 somewhere else and you recompute all of it. Cache affinity is great for efficiency, and it also piles load onto the same few nodes.\n\n**Queued work goes stale.** A request that has waited 40 seconds has probably been abandoned by its client, so if the GPU runs it anyway, nobody reads the output. When [TGI](https://github.com/huggingface/text-generation-inference) users asked why they kept getting “Model is overloaded”, the maintainers [explained](https://github.com/huggingface/text-generation-inference/issues/439) that refusing early beats serving old requests that are going to time out anyway.\n\nPut together, you get a goodput cliff. Past saturation, tokens per second hold steady or even rise, but the share of requests that meet their latency target collapses, because everyone is waiting behind everyone else. Admission control keeps you on the right side of that cliff.\n\n## Cache-aware routing needs a way to say no\n\nA quick recap. The router splits the prompt into token blocks and chains their hashes, so each block’s hash covers everything before it. Attention is causal, so a replica can reuse block *i* only if it also holds every block before it. What counts is the longest unbroken prefix, not how many blocks match.\n\nEvery router then mixes that prefix score with load. In Cognitora it’s a weighted sum:\n\nllm-d does something similar with weighted scorers, plus an affinity filter that breaks stickiness when the sticky pods get much slower than the rest.\n\nThat handles moderate imbalance. Under saturation, though, the weighted sum still picks a winner, the least-bad node, and the request lands in yet another long queue.\n\nRouting always returns a node. Only admission control can return none.\n\n## What “busy” means on a GPU\n\nAdmission is only as good as its load signal, and request count is the worst one available. The signals that work measure what actually runs out:\n\n- **KV cache utilization** , the closest proxy for the memory limit. llm-d treats 0.8 as saturated by default, and Dynamo deployments often use about 0.85.\n- **KV cache hit rate.** A full cache with a falling hit rate is churning, not working. This one comes back in the agent section.\n- **Queued prefill tokens** , which predict time to first token far better than request count.[vLLM](https://github.com/vllm-project/vllm) ’s`max_num_queued_tokens` and Dynamo’s prefill-token threshold use it.\n- **In-flight tokens** , which llm-d’s token-load scorer uses in place of request counts.\n- **Engine queue depth** : cheap and fast to read, but blind to request size.\n- **Power draw and hardware fit** , so a node near its watt cap, or a 24 GB card, doesn’t get a 70B model’s traffic.\n\n## Five layers of admission\n\nIn practice admission control is a sequence of checks, cheapest first. Products order them a little differently, but the same five keep showing up.\n\n### 1. Validation\n\nReject anything that should never reach a GPU: bad JSON, unknown models, missing credentials, oversized bodies, out-of-range parameters. TGI bounds things like `--max-best-of` and `--max-stop-sequences` and returns 422, Dynamo caps body size, and per-key rate limits belong here too. It costs microseconds.\n\n### 2. Context and budget\n\nCheck that the request fits the model’s context window: *input tokens + max output tokens ≤ context window*. vLLM returns 400 when it doesn’t, TGI 422.\n\nThis layer matters more than it looks, because `max_tokens` is the worst-case KV reservation. A client that asks for 32,000 tokens and stops after 200 makes conservative schedulers hold memory for nothing. SGLang’s [tuning guide](https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning) points at exactly this when KV usage is low but requests are still queueing.\n\nSensible `max_tokens` defaults per route give you capacity for free.\n\n### 3. KV and token budget\n\nThis is the check only LLM serving needs: can the GPU hold this request’s KV cache right now without running out of memory or evicting someone? Engines answer it differently:\n\n- **vLLM** caps`max_num_seqs` and`max_num_batched_tokens` , overcommits memory, and preempts and recomputes a request when it runs out.\n- **SGLang** predicts how much running requests will grow (tunable with`--schedule-conservativeness` ) and retracts requests when the guess turns out wrong.\n- **[TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)** lets you choose.`GUARANTEED_NO_EVICT` , the default, reserves enough for every admitted request to finish.`MAX_UTILIZATION` packs harder and pauses requests when memory runs out.\n\nThat last choice has a real cost. One [benchmark on v1.2.1](https://github.com/NVIDIA/TensorRT-LLM/issues/16827) saw `MAX_UTILIZATION` lose 40% of its throughput, with 9x worse p99 time per output token, under memory pressure. The gap mostly closed in 1.3.0rc22, but the point stands: overcommitting only pays while eviction stays rare.\n\nAt the fleet level, [Dynamo](https://docs.nvidia.com/dynamo/user-guides/fault-tolerance/request-rejection) marks a worker busy when its decode KV usage or active prefill tokens cross a threshold, and llm-d’s saturation detector does the same with KV utilization and queue depth.\n\nOne gotcha from the Dynamo docs: decode-block thresholds only work if output tokens are counted (`--router-track-output-blocks`). The most common failure is a threshold that never fires because load events aren’t reaching the frontend.\n\n### 4. Queue capacity\n\nThe engine’s memory check decides who runs next, not who waits, so something has to cap the waiting room. Here’s how the main systems do it:\n\n- **TGI** has done this since 2023 with`--max-concurrent-requests` (default 128) and a 429.\n- **vLLM** ’s queue was[unbounded for a long time](https://github.com/vllm-project/vllm/issues/18826) . It now has`max_num_queued_reqs` , which returns 503, and`max_num_queued_tokens` , which caps the prompt tokens waiting for prefill.\n- **SGLang** has`--max-queued-requests` .\n- **[Ray Serve](https://docs.ray.io/en/latest/serve/index.html)** has`max_queued_requests` , with a 503 or 429 plus`Retry-After` .\n- **Dynamo** returns 529 when every worker is busy, and can also hold requests in a router queue and route them later with fresher metrics.\n- **Cognitora** counts in-flight requests per model and role (prefill, decode, or both), so a burst of long prompts can’t eat the capacity short chat turns need.\n\nThe vLLM docs suggest sizing the token cap as *target TTFT × prefill throughput*. That turns a queue limit into a latency budget, and it’s my favourite knob in this whole space.\n\n### 5. Priority and flow control\n\nThe last layer decides who gets whatever capacity is left. llm-d and the [Gateway API Inference Extension](https://gateway-api-inference-extension.sigs.k8s.io/) go furthest here.\n\nRequests wait in the gateway, sorted by tenant and priority band. A dispatcher picks by priority, then by fairness within a band, and nothing goes out while the saturation detector says the pool is full. Negative-priority requests are sheddable, protected ones are never displaced, and every queued request has a TTL that clients can shorten with a header.\n\nInside the engines, vLLM and SGLang both support priority scheduling. When SGLang’s queue is full, it drops the lowest-priority waiting request instead of the new one if the new one outranks it. Cognitora can also shed low-priority traffic while grid carbon intensity is high, which pushes deferrable batch work to cleaner hours.\n\nUnder this layer sits a real design question: where should the queue live? Queue in each engine and batching stays efficient, but a request stuck in replica A’s queue can’t move when B frees up first. Queue centrally and you keep control over priority and placement.\n\nThe [llm-d flow control guide](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/flow-control.md) recommends a small “healthy buffer” on each engine to keep batches full, with everything else held in the gateway. I think that’s the right default.\n\n## Admitting only what can meet its SLO\n\nThe five layers ask whether there’s room. A better question is whether this particular request can still meet its latency target. If it can’t, rejecting it right away helps everyone, including the caller, who can retry elsewhere.\n\n[Mooncake](https://github.com/kvcache-ai/Mooncake), the system behind Moonshot’s Kimi, does this in production with separate prefill and decode pools. Two lessons from [the paper](https://arxiv.org/abs/2407.00079) stuck with me.\n\nFirst, check decode capacity before prefill starts, or every request rejected at decode has wasted its prefill.\n\nSecond, use predicted load, not current load. Rejecting on current decode load made the two pools oscillate: prefill accepted a wave, decode filled up a few seconds later, everything got rejected, decode drained, and the cycle repeated. Predicting decode load for the moment the request would actually arrive fixed it.\n\nCognitora has a cruder version: estimate time to first token from the node’s queue depth, and reject if that blows the deadline. Research goes much further:\n\n- [SCORPIO](https://arxiv.org/abs/2505.23022) rejects requests whose deadlines can’t be met and reports up to 14.4x more goodput.\n- [SLOs-Serve](https://arxiv.org/abs/2504.08784) reroutes them or serves them best-effort instead.\n- [JITServe](https://www.usenix.org/conference/nsdi26/presentation/zhang-wei) and[Cascade](https://arxiv.org/abs/2608.06557) give each request just enough capacity to hit its target.\n\nUtilization tells you the GPU is busy. Feasibility tells you whether a request is worth starting.\n\n## Agents change the unit of admission\n\nEverything so far treats each request as independent. Agents don’t work that way.\n\nAn agent generates, calls a tool, waits, and comes back with a longer context. While the tool runs, its KV cache sits idle, and request-level admission has no idea that the idle 40,000-token prefix belongs to someone who’ll be back in two seconds.\n\n[CONCUR](https://arxiv.org/abs/2601.22705) (PMLR 2026) calls the result middle-phase thrashing. In traces of DeepSeek-V3 agents, prefixes are shared early on and hit rates approach 90%. Then contexts grow, KV usage sits near full, and the hit rate collapses, because paused agents lose their cache while they wait and recompute it when they return.\n\nRecomputation took 49.1% of end-to-end latency, and this phase lasted over 90% of the run. The GPU isn’t out of memory in the usual sense. It’s full of state that gets evicted just before it’s needed. A request-level cap didn’t help much either: the hit rate still fell to 32%.\n\nCONCUR admits agents instead of requests. It grows the active set slowly while KV usage is low and halves it when usage is high and the hit rate is low. Agents only pause between steps, so nothing is interrupted mid-generation. It reports up to 4.09x throughput on Qwen3-32B.\n\nHit rate is an admission signal. The right number of concurrent agents is lower than what the KV pool can technically hold.\n\n[TAPER](https://arxiv.org/abs/2605.06914), released as [PACE](https://github.com/Swapnil-Gandhi/pace), applies the same idea inside a single request. When a reasoning method splits its output into parallel branches, running them all at once slows the shared decode step for everyone in the batch. TAPER admits an extra branch on each step only if its predicted latency cost fits the batch’s slack, and reports 1.48x more goodput than running every branch eagerly.\n\nA naming note to avoid confusion: agent papers also use “memory admission control”, as in [A-MAC](https://arxiv.org/abs/2603.04549), and it means something else entirely. It decides what an agent writes to long-term memory, scoring each candidate on utility, confidence, novelty, recency, and content type.\n\nIt has nothing to do with GPU queues, though the two are connected: every memory you store ends up in future prompts, so a noisy memory store means longer contexts and more KV pressure later.\n\n## Side by side\n\nHere’s where each system puts its admission checks, from single engines up to fleet orchestrators and agent-level schedulers.\n\n## What I’d take away\n\n- **Run the cheap checks first** , and admit before you send the first byte of a stream. Once you’ve returned 200 and opened an event stream, you can’t send a 429 anymore.\n- **Cap `max_tokens`.** It’s the cheapest capacity you’ll ever get.\n- **Measure load in tokens and KV blocks** , and watch hit rate as well as utilization.\n- **Treat cache affinity as a source of load.** A sticky node at 95% KV usage should stop attracting traffic, even if it holds the perfect prefix.\n- **Budget prefill and decode separately** in disaggregated setups, and check both before prefill starts.\n- **Expect stale telemetry and oscillation.** Prediction, AIMD-style windows, and hysteresis all help.\n- **Decide on purpose what happens when metrics disappear.** llm-d stops dispatching, while Cognitora’s carbon check keeps serving. Either is fine, as long as it’s a choice.\n- **Make rejections easy to act on.** Tell “overloaded, retry” apart from “invalid, don’t”, send`Retry-After` , and back off with jitter in clients.\n- **Let callers mark what can wait.** Batch and eval jobs should almost always be sheddable.\n\n## Where I’d start\n\nFor a fleet of vLLM replicas behind a cache-aware router:\n\n1. Authenticate, rate-limit per key, and cap request size at the gateway.\n2. Set `max_model_len` deliberately, and use per-route`max_tokens` defaults well below it.\n3. Tune `max_num_seqs` and`max_num_batched_tokens` , watch the preemption counter, and mark replicas busy at around 0.8 to 0.85 KV usage.\n4. Set `max_num_queued_reqs` , size`max_num_queued_tokens` as*target TTFT × prefill throughput* , and return 429 or 529 with`Retry-After` .\n5. Mark interactive traffic as protected and batch work as sheddable, with a queue TTL shorter than the client timeout.\n\nIf agents make up a lot of your traffic, also cap concurrent agents, and lower that cap when the hit rate drops.\n\nThen alert on goodput, meaning the share of requests that hit their latency targets, rather than raw throughput. And load-test past saturation on purpose. How a system behaves at 150% of capacity tells you more than its peak tokens per second.\n\n*Related reading: [LLM Inference: a learning roadmap](https://www.fratepietro.com/2026/llm-inference-learning-roadmap/), [llama.cpp, vLLM, and SGLang: Three Approaches to LLM Inference](https://www.fratepietro.com/2026/llama-cpp-vllm-sglang-inference-tools/), [Cognitora v0.9.1: gossip discovery, live KV prefix, and a native engine](https://www.fratepietro.com/2026/cognitora-inference-update-v0.9/), and [Google Cloud agentic AI architecture: private inference networking](https://www.fratepietro.com/2026/google-cloud-agentic-ai-architecture-private-inference-networking/).*", "url": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference", "canonical_source": "https://www.fratepietro.com/2026/llm-admission-control-distributed-inference/", "published_at": "2026-10-22 22:00:00+00:00", "updated_at": "2026-11-09 01:38:58.069999+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "mlops", "ai-agents"], "entities": ["SGLang", "NVIDIA Dynamo", "llm-d", "GKE Inference Gateway", "Cognitora", "Hugging Face", "TGI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference", "markdown": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference.md", "text": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference.txt", "jsonld": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference.jsonld"}}