{"slug": "admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to", "title": "Admission Control for Distributed LLM Inference: Cache-Aware Routing, GPU-Aware Signals, and When to Say No", "summary": "A technical post on admission control for distributed LLM inference argues that cache-aware routing, now standard in SGLang's router, NVIDIA Dynamo's KV router, llm-d's prefix-cache scorer, the GKE Inference Gateway and Cognitora, is insufficient under bursty traffic and must be paired with an admission layer that accepts, queues or sheds each request. The post attributes the need to four properties of LLM traffic: request cost varying by more than three orders of magnitude between a 50-token chat turn and a 120,000-token repository prompt, unknown output length at admission time, non-interchangeable replicas holding up to 30,000 tokens of reusable KV state per agent session, and queued work going stale. It concludes that for agentic workloads the unit of admission shifts from the request to the agent.", "body_md": "# Admission Control for Distributed LLM Inference: Cache-Aware Routing, GPU-Aware Signals, and When to Say No\n\nSpread an LLM across a fleet of GPUs and most of the engineering attention goes to **routing**: which replica should serve this request so it reuses a warm KV cache and lands on a GPU that is not already saturated. Cache-aware routing is now table stakes. SGLang’s router, NVIDIA Dynamo’s KV router, llm-d’s prefix-cache scorer, the GKE Inference Gateway, and [Cognitora](https://inference.cognitora.dev) all do some version of it.\n\nRouting assumes the cluster *can* serve the request somewhere. Under real traffic that assumption breaks several times a day: a batch job starts, an agent fans out forty parallel calls, a product launch triples the request rate. At that point the right question is not “where” but “whether, and when.”\n\n**Admission control** is the traffic-management and resource-governance layer that answers it. For every incoming generation request it decides one of three things: **accept** it now, **queue** it, or **shed** it. The decision is based on available GPU memory (mostly KV cache), compute capacity, and concurrency limits. Without it, queues grow without bound, and the result is predictable: severe tail-latency spikes, memory exhaustion, and cascading timeouts as clients give up and retry into an already overloaded system.\n\nThis post treats admission control as a general concept, walks through the layers a serving stack needs, and looks at how the current inference engines and orchestrators implement each one. The last part covers agentic workloads, where request-level admission stops working and the unit of admission becomes the agent.\n\n## 1. Three decisions, three components\n\nEvery request in a distributed inference stack passes through three decisions, usually made by different components:\n\n- **Admission** (whether and when) protects the system as a whole: tenants, priority classes, and the latency SLO.\n- **Routing** (where) uses cache locality and per-node load to pick the best replica.\n- **Scheduling** (how, inside the engine) decides which requests join the next forward pass and what happens when KV memory runs out.\n\nEach component only sees part of the picture. The engine knows its own KV blocks but not the rest of the fleet. The router knows the fleet but sees each node’s state a few hundred milliseconds late. The gateway knows tenants and SLOs but not which GPU holds which prefix. That is why admission control in practice is not one check but a stack of checks spread across all three, which section 5 walks through layer by layer.\n\n## 2. Why LLM traffic breaks classical load balancing\n\nFor a stateless web service, admission control is mostly a solved problem: a concurrency limit per replica, a token bucket per client, and a retry policy. LLM inference breaks the assumptions behind those tools in four ways.\n\n**Request cost varies by orders of magnitude.** A 50-token chat turn and a 120,000-token repository prompt are both “one request.” Their prefill cost differs by more than three orders of magnitude, and their KV cache footprint by the same factor. A concurrency limit of 64 requests means nothing when one of those 64 can occupy half the KV pool.\n\n**Output length is unknown at admission time.** The decode phase grows KV memory one token at a time until the model emits a stop token. The engine has to admit a request before knowing how much memory it will eventually need. Engines either reserve for the worst case (safe but wasteful) or overcommit and evict later (efficient but disruptive).\n\n**Replicas are not interchangeable.** With prefix caching, the replica that served turn 5 of an agent session holds 30,000 tokens of reusable KV state. Sending turn 6 anywhere else means recomputing all of it. Cache affinity makes some nodes much better for a given request than others, and that concentrates load on exactly those nodes.\n\n**Queued work goes stale.** A request that waits 40 seconds in a queue will often be cancelled by its client before it starts. If the engine then runs its prefill anyway, the GPU spends compute on output nobody reads. The TGI maintainers [put it plainly](https://github.com/huggingface/text-generation-inference/issues/439) when users asked why the server returned “Model is overloaded”: refusing early is better than serving old requests in order, because those clients will most likely time out and the work is wasted.\n\nThe combined effect is the **goodput cliff**. Push an LLM service past saturation and throughput (tokens per second) stays flat or even rises, while goodput (requests that finish within their SLO) collapses. Every request now waits behind every other request, so nobody meets their latency target. Admission control exists to keep the system on the left side of that cliff.\n\n## 3. Cache-aware routing, and why it needs admission control\n\nA quick recap of cache-aware routing, because it creates the hot spots that admission control has to manage.\n\nThe router hashes the prompt into fixed-size token blocks and **chains** the hashes so that the digest of block *i* covers the entire prefix `[0..i]`. Because attention is causal, a replica can only reuse block *i* if it also holds blocks `0..i-1`. The useful signal is therefore the **longest contiguous prefix** a replica holds, not how many matching blocks it has scattered around. llm-d’s [KV-cache indexer documentation](https://llm-d.ai/docs/architecture/advanced/kv-management/kv-indexer) has a clean example: a pod holding blocks B3 and B4 without B0 to B2 scores zero.\n\nImplementations differ in where that prefix knowledge comes from:\n\nEvery one of them then **blends** prefix overlap with load. Cognitora’s router, for example, scores each candidate as a weighted sum:\n\n```\nscore(n) = 0.55 · kv_overlap + 0.25 · (1 − queue_depth/64)\n         + 0.10 · (1 − power/peer_max_power) + 0.10 · free_kv_blocks/total_kv_blocks\n```\n\nllm-d does the same with a prefix-cache scorer, a KV-utilization scorer, and a queue scorer combined by weight, plus a `prefix-cache-affinity-filter` that has a “TTFT load gate” to break stickiness when the sticky pods have become much slower than the rest.\n\nThe blend handles moderate imbalance. It does not handle saturation. When every replica is busy, the weighted sum still returns a winner: the least-bad node. The request is dispatched, joins a long local queue, and waits. **Routing always picks a node. Admission control is the only layer allowed to pick none.**\n\n## 4. GPU-aware signals: what “busy” actually means\n\nAdmission control is only as good as its definition of load. Request count is the signal every generic load balancer has, and it is the weakest one for LLMs. The systems below use better ones:\n\n- **KV cache utilization** : fraction of KV blocks in use. This is the closest signal to the actual memory limit. llm-d’s default saturation threshold is 0.8. Dynamo’s`--active-decode-blocks-threshold` is typically set around 0.85.\n- **KV cache hit rate** : fraction of prompt tokens served from cache. High utilization with a falling hit rate means the cache is churning, not working. This is the signal agent-level controllers rely on (section 6).\n- **Queued prefill tokens** : total prompt tokens waiting for prefill. This predicts TTFT far better than request count, because prefill time scales with tokens. vLLM’s`max_num_queued_tokens` and Dynamo’s`--active-prefill-tokens-threshold` use it.\n- **In-flight tokens** : llm-d’s`token-load-scorer` uses estimated tokens in flight instead of request count, normalized against a 4Mi-token threshold.\n- **Engine queue depth** : requests waiting inside the engine scheduler. Cheap to read and fast to react, but blind to request size.\n- **Power and thermals** : watts against a per-node cap, from NVML, ROCm SMI, or Redfish. Cognitora uses this as a soft filter that prefers under-cap nodes and as a routing term.\n- **Hardware capability** : VRAM size and GPU vendor, so a 70B model never lands on a 24 GB card in a mixed fleet.\n\nThe pattern across all of them: **measure load in the unit that actually runs out**, which is KV memory and prefill tokens, not HTTP requests.\n\n## 5. The five layers of an admission pipeline\n\nAn effective LLM gateway or serving engine applies admission control as a sequence of checks, ordered from cheapest to most expensive. Each layer rejects a different kind of request, and each one is cheaper to fail at than the next.\n\nThe order is not strict across products (some engines run the KV check after queueing, some gateways run priority before queue caps), but the layers are the same everywhere.\n\n### 5.1 Request and validation layer\n\nThe first layer rejects requests that should never reach a GPU: malformed JSON, unknown models, missing or invalid credentials, oversized bodies, and parameters outside allowed limits. It costs microseconds and protects every layer behind it.\n\nExamples from the systems I looked at:\n\n- **TGI** validates in its Rust router before queueing:`--max-stop-sequences` ,`--max-best-of` ,`--max-top-n-tokens` , and`--max-client-batch-size` all bound what a single request can ask for. Violations return 422`validation` errors.\n- **NVIDIA Dynamo** caps request body size with`DYN_HTTP_BODY_LIMIT_MB` (192 MB by default), which matters once multimodal inputs carry images.\n- **Cognitora** runs auth (API key or OIDC) and a per-subject rate limit before any prefix hashing or routing work.\n- **Every OpenAI-compatible server** rejects unknown model names with 404 before touching the scheduler.\n\nRate limits per API key belong here too. They are not capacity-aware, but they stop a single client from monopolizing the queue before capacity-aware layers ever see the traffic.\n\n### 5.2 Context-window and budget validation\n\nThe second layer checks whether the request can fit in the model at all: `input_tokens + max_output_tokens` against the model’s maximum sequence length. A request that fails here would eventually crash or be truncated, so rejecting it immediately is strictly better.\n\n- **vLLM** rejects prompts whose length plus requested`max_tokens` exceeds`max_model_len` with a 400 error. If`max_tokens` is unset, it fills the remaining context.\n- **TGI** enforces`--max-input-tokens` and`--max-total-tokens` per request and returns 422 on violation.\n- **Cognitora** uses`max_tokens` to derive each request’s implicit deadline (`ttft_slo + max_tokens × tpot` ), so the budget feeds directly into SLO admission later.\n\nThis layer also deserves more attention than it usually gets, because **`max_output_tokens` is the worst-case KV reservation**. A client that sends `max_tokens: 32000` for a reply that ends after 200 tokens makes every conservative scheduler reserve or predict far more memory than it needs. SGLang’s [tuning guide](https://docs.sglang.io/docs/advanced_features/hyperparameter_tuning) calls out exactly this case: low KV usage with a non-empty queue often means users send large `max_new_tokens` and stop early. Tight per-route or per-tenant limits on `max_tokens` increase effective capacity without touching the GPU.\n\n### 5.3 Token-budget and KV cache admission\n\nThe third layer asks the question that is unique to LLM serving: **can the GPU hold this request’s KV cache right now, without triggering an out-of-memory error or evicting someone else?** This happens inside the engine scheduler, on every iteration, and it is where the “unknown output length” problem lives. Engines take different positions on it.\n\n**vLLM** caps concurrency with `max_num_seqs` (sequences per iteration) and `max_num_batched_tokens` (tokens per iteration), and overcommits KV memory. When the pool runs out mid-decode, the scheduler **preempts** a running request and, in V1, recomputes it from scratch later (`PreemptionMode.RECOMPUTE`). The [tuning guide](https://docs.vllm.ai/en/stable/configuration/optimization/) recommends lowering `max_num_seqs` or `max_num_batched_tokens` when preemption warnings show up. A newer `max_num_active_seqs` lets you cap how many requests enter RUNNING without shrinking CUDA graph capture.\n\n**SGLang** admits requests based on a predicted **new-token ratio**: an estimate of how much each running request will still grow. `--schedule-conservativeness` scales that estimate. When the prediction is wrong and the pool fills, SGLang **retracts** running requests (`--retraction-policy length` or `priority`). KV token usage below 0.9 with a non-empty queue means the scheduler is too conservative. Frequent `KV cache pool is full. Retract requests.` warnings mean it is too aggressive.\n\n**TensorRT-LLM** makes the tradeoff an explicit policy. `GUARANTEED_NO_EVICT` (the default) reserves enough KV for each admitted request to finish, so a started request is never paused. `MAX_UTILIZATION` packs requests aggressively and pauses them when memory runs out. A [recent benchmark report](https://github.com/NVIDIA/TensorRT-LLM/issues/16827) measured MAX_UTILIZATION on v1.2.1 under KV oversubscription at a 40% throughput drop and 9x worse p99 TPOT, while the two policies were equivalent when the working set fit in the pool. The gap mostly disappeared on 1.3.0rc22, but the lesson holds: overcommitting memory only pays off while eviction stays rare.\n\nAt the fleet level, the same question gets asked with fleet-wide telemetry. **NVIDIA Dynamo**’s [busy thresholds](https://docs.nvidia.com/dynamo/user-guides/fault-tolerance/request-rejection) mark a worker as busy when its decode KV block utilization (`--active-decode-blocks-threshold`) or active prefill tokens (`--active-prefill-tokens-threshold`) cross a limit, and remove it from the candidate set. **llm-d**’s saturation detector does the same with `kvCacheUtilThreshold` (default 0.8) and engine queue depth. Two details from the Dynamo docs are worth copying:\n\n- Decode-block thresholds only work if output tokens are counted (`--router-track-output-blocks` ). Otherwise a long generation can fill KV without the router ever seeing the threshold crossed.\n- The most common production symptom is a threshold that is accepted at startup but never fires, because worker load events are not reaching the frontend. Check telemetry before tuning.\n\nEngine memory admission controls **who runs next**, not **who waits**. Every request that does not fit stays in a waiting queue, which brings us to layer four.\n\n### 5.4 Queue-capacity backpressure\n\nThe fourth layer enforces a strict maximum on how much work can wait. Past that limit, new requests get a fast rejection (HTTP 429, 503, or 529, ideally with `Retry-After`) instead of joining a queue that only grows. This is the layer that prevents the unbounded-queue failure mode directly.\n\nAt the engine:\n\n- **TGI** :`--max-concurrent-requests` (default 128) is a semaphore in the router. Request 129 gets HTTP 429 with`\"error_type\": \"overloaded\"` . This has been in TGI since 2023 and was the first widely deployed example of “say no early” in an LLM server.\n- **vLLM** : the waiting queue was unbounded for a long time ([RFC #18826](https://github.com/vllm-project/vllm/issues/18826) documents the OOM risk). Recent releases add`max_num_queued_reqs` , which caps waiting plus running requests across all data-parallel ranks and returns HTTP 503 so a load balancer can retry elsewhere. A second knob,`max_num_queued_tokens` , caps the total prompt tokens waiting for prefill. The docs suggest sizing it as`target_TTFT × prefill_throughput` , which turns a queue cap into a**TTFT budget** : if the backlog would take longer than your TTFT target to clear, new requests are rejected.\n- **SGLang** :`--max-queued-requests` aborts incoming requests when the waiting queue is full (the flag is ignored in disaggregation mode).\n\nAt the gateway:\n\n- **Ray Serve** enforces`max_ongoing_requests` per replica (default 5, a value LLM deployments always raise) and`max_queued_requests` per caller. Past the queue limit it raises`BackPressureError` , mapped to HTTP 503 by default.`BackpressureConfig` lets you return 429 instead and attach a`Retry-After` header.\n- **Cognitora** keeps a lock-free atomic counter per**(model, role)** pair, where role is prefill, decode, or both. A request takes a permit before dispatch. If the counter is at`max_queue` the request gets an immediate 429 with no buffering, and the permit is released automatically when the stream ends. Keying on role matters in disaggregated setups: long prompts count against the decode pool’s budget, short colocated ones against the shared pool, so a burst of long prompts cannot exhaust capacity for short chat turns.\n- **NVIDIA Dynamo** returns**HTTP 529 (Overloaded)** when registered workers exist but all of them are over a busy threshold, kept distinct from 503 (no workers at all). It also has`--router-queue-threshold` , which**defers** the routing decision instead of rejecting: the request waits in the router and is routed later with fresher load metrics.\n\nA fixed queue cap is simple and predictable. Its weakness is that the right number depends on the prompt length mix, which changes during the day. vLLM’s token-denominated cap and Dynamo’s utilization thresholds are both attempts to track that mix instead of guessing a constant.\n\n### 5.5 Priority shedding and flow control\n\nThe fifth layer decides **which** traffic gets the capacity that exists. Requests are sorted into tiers. Under heavy load, lower-priority or “sheddable” requests are dropped first so high-priority SLAs hold.\n\n**llm-d and the Kubernetes Gateway API Inference Extension** have the most complete implementation. Behind a feature gate, the Endpoint Picker runs a full **flow control** layer. The [flow control guide](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/site-src/guides/flow-control.md) describes the loop:\n\n1. Requests are sorted into queues by flow (tenant or workload) and priority band.\n2. A dispatcher picks the next request by strict priority, then by a fairness policy within the band.\n3. Before dispatching, it asks the saturation detector whether the pool has room. If not, dispatch halts and the request stays in the gateway queue.\n\nRequests with **negative priority are sheddable**: they can be rejected or evicted when a band is full, while protected requests are never displaced. Every queued request has a TTL (`defaultRequestTTL`), and clients can shorten it with an `x-llm-d-inference-ttl` header. A separate `noEndpointRequestTTL` turns the queue into a waiting room for scale-from-zero, returning 503 instead of 429 when it expires.\n\nThe tuning advice gets at a central design question: **where should the queue live?** Set `queueDepthThreshold` to 1 and nearly all queuing happens centrally in the gateway, which gives full control over priority, fairness, and late routing. Set it higher and the engines keep a small buffer, so continuous batching always has work. The guide calls the target a “healthy buffer”: enough queued work on the model servers to form efficient batches, and no more. Endpoints whose metrics are older than `metricsStalenessThreshold` count as **fully saturated**, so if the gateway loses its view of the fleet, it stops dispatching instead of guessing.\n\nPriority also exists inside the engines:\n\n- **vLLM**`--scheduling-policy priority` orders the waiting queue by request priority.\n- **SGLang**`--enable-priority-scheduling` lets high-priority requests preempt running ones above a configurable priority gap. When`--max-queued-requests` is hit, it aborts the**lowest-priority waiting request** instead of the new one if the new one outranks it.\n- **Dynamo** accepts`nvext.agent_hints.priority` and`strict_priority` to order its router queue.\n\nPolicy can go beyond priority. **Cognitora** polls a grid-intensity provider (static, Electricity Maps, or WattTime) and sheds requests marked `X-CGN-Priority: low` while grid carbon intensity exceeds a threshold. Normal and high-priority traffic is always admitted, so deferrable batch work moves to cleaner hours without touching interactive traffic.\n\n### 5.6 Beyond the five layers: SLO-feasibility admission\n\nThe five layers answer “is there room?” A sharper question is: **can this specific request still meet its SLO?** If not, rejecting it immediately is better for everyone, including the caller, who can retry elsewhere.\n\n**Mooncake** (Moonshot AI’s serving system for Kimi) is the clearest production example. Its prefill and decode pools are disaggregated, and it measures load as SLO satisfaction: predicted TTFT against the TTFT target on prefill nodes, predicted time-between-tokens against the TBT target on decode nodes. [Two findings](https://arxiv.org/abs/2407.00079) from the paper apply to any disaggregated system:\n\n- **Early rejection.** Checking decode capacity only after prefill wastes the prefill compute of every request rejected at decode time. Mooncake evaluates the larger of the two pool loads*before* prefill starts.\n- **Prediction beats measurement.** Early rejection based on current decode load produced anti-phase oscillation: prefill accepts a wave, decode fills up a few seconds later, the conductor rejects everything, decode drains, repeat. Mooncake fixed it by predicting decode load at the time the request would actually arrive there.\n\n**Cognitora** has a simpler version. The router estimates TTFT on the chosen node from its queue depth and per-replica concurrency, and returns 429 if the estimate exceeds the request’s deadline. It is a coarse model, but it rejects requests that are clearly doomed instead of queueing them behind a noisy neighbor.\n\nResearch systems push the idea further:\n\n- [**SCORPIO**](https://arxiv.org/abs/2505.23022) uses a TTFT guard (least-deadline-first ordering plus rejection of unattainable requests) and a TPOT guard (admission based on per-request decode-speed budgets), reporting up to 14.4x goodput over throughput-oriented baselines.\n- [**SLOs-Serve**](https://arxiv.org/abs/2504.08784) uses*soft* admission: requests whose SLO is unattainable on one replica are routed to another or served best-effort instead of dropped.\n- [**JITServe**](https://www.usenix.org/conference/nsdi26/presentation/zhang-wei) (NSDI 2026) schedules with imprecise request information and refines its estimates as generation progresses, giving each request just enough bandwidth to meet its SLO.\n- [**Cascade**](https://arxiv.org/abs/2608.06557) computes a per-request** latency budget** (SLO minus predicted remaining service time) and uses it to drive both scheduling and KV tier decisions.\n\nThe shared idea: **admission based on SLO feasibility, not utilization**. Utilization tells you the GPU is busy. Feasibility tells you whether a specific request is worth starting.\n\n## 6. Agentic workloads: changing the unit of admission\n\nEverything above treats the **request** as the unit of admission. For agents, that assumption breaks.\n\nAn agent is a long-lived, stateful loop: generate, call a tool, wait, generate again with a longer context. Between steps its KV cache sits idle in GPU memory while the tool runs. Request-level admission sees each step as an independent request and has no idea that the idle 40,000-token prefix belongs to an agent that will be back in two seconds.\n\n### 6.1 Agent-level admission: CONCUR\n\n[**CONCUR**](https://arxiv.org/abs/2601.22705) (Chen et al., PMLR 2026) names the resulting failure **middle-phase thrashing**. From trace analysis of DeepSeek-V3 agents in batch inference, it identifies three phases:\n\n1. **Warmup.** Contexts are short and share prompt prefixes. KV hit rates approach 90%, and adding concurrency increases throughput.\n2. **Middle phase.** Agents accumulate history. KV usage sits at 80% to 100% while the hit rate collapses. Paused agents lose their prefixes to LRU eviction while they wait on tools, then pay a full recompute when they resume. Recomputation accounted for 49.1% of end-to-end latency, and this phase covered more than 90% of execution time. Adding agents in this phase*reduces* throughput.\n3. **Cooldown.** Agents finish, memory frees up, and hit rates partially recover.\n\nThe important observation is that the GPU is not out of memory in the classical sense. It is **thrashing**: memory is full of state that gets evicted just before it would have been reused. Request-level controls do not fix it. In the paper’s measurements, SGLang’s hit rate fell to 35% at high concurrency, and a request-level admission cap still dropped to 32%.\n\nCONCUR’s answer is to make the **agent** the unit of admission and to regulate the number of active agents the way TCP regulates a congestion window, with **AIMD** (additive increase, multiplicative decrease). It sits as middleware between the agent framework and an unmodified serving engine, exposes three primitives (admit, pause, resume), and updates the active-agent window `W` from two engine signals, KV usage `U` and cache hit rate `H`:\n\nTwo design choices stand out. Pausing happens only at step boundaries, between a generation and a tool call, so in-flight work is never interrupted. And the multiplicative cut is justified by the cost shape: recomputation scales quadratically with context length, so the controller should leave the thrashing regime quickly rather than gently. The reported result is up to 4.09x batch throughput on Qwen3-32B and 1.9x on DeepSeek-V3.\n\nThe paper targets batch agent inference, but the lesson generalizes to any serving stack running agents: **cache hit rate is an admission signal**, and the right number of concurrent agents is lower than the number of requests the KV pool can technically hold.\n\n### 6.2 Branch-level admission: TAPER (PACE)\n\nA related idea works at a finer granularity: inside a single request. Some reasoning and agent methods split one output into parallel **branches** that decode concurrently (map, process, reduce). Executing every branch eagerly makes each shared decode step slower, which hurts every other request in the batch. A fixed cap on branches gives up the speedup that made branching worth doing.\n\n[**TAPER**](https://arxiv.org/abs/2605.06914) (Gandhi, Hari, Dally, Kozyrakis; NeurIPS 2026), released as the [** PACE**](https://github.com/Swapnil-Gandhi/pace) codebase on top of an SGLang fork, makes branch admission a per-decode-step decision. Every active request is guaranteed one token per step. Extra branches are admitted only when their predicted latency cost, which the authors call the **branch externality**, fits within the batch’s current slack budget, using a calibrated step-latency model `T = a + b·n_tokens + c·L_context`. Because branches share the request’s prefix KV, expanding or shrinking the width needs no memory reclamation. On Qwen3-32B it reports 1.77x goodput over no branching and 1.48x over eager branching, with more than 95% SLO attainment.\n\nCONCUR and TAPER bracket the same principle from two sides: admit **agents** across requests, and admit **branches** within a request, using live feedback instead of fixed caps.\n\n### 6.3 A different “admission control”: agent memory (A-MAC)\n\nThe term also shows up in agent architectures with a completely different meaning, and it is worth separating the two to avoid confusion.\n\n[**A-MAC**](https://arxiv.org/abs/2603.04549) (Adaptive Memory Admission Control, Workday AI, 2026) has nothing to do with GPU queues. It governs **what information gets written into an agent’s long-term memory store**. Without it, agents either store everything, including hallucinated and obsolete facts, or rely on opaque LLM-driven memory policies that are expensive and hard to audit. A-MAC scores each candidate memory on five interpretable factors (future utility, factual confidence, semantic novelty, temporal recency, and a content-type prior) and admits it only if the weighted score clears a learned threshold. On the LoCoMo benchmark it reports an F1 of 0.583 with 31% lower latency than LLM-native memory systems, and the content-type prior turned out to be the most influential factor.\n\nThe connection to inference admission is indirect but real. Every memory an agent admits is retrieved into future prompts, so a noisy memory store becomes longer contexts, more prefill, and a larger KV footprint per step. Admission at the memory layer is upstream of admission at the GPU layer.\n\n## 7. Side by side\n\n## 8. Design lessons\n\nReading these systems together, a few rules come out consistently.\n\n**Order the checks from cheapest to most expensive.** Validation and context checks cost microseconds and should run before any tokenization or routing. Queue and priority checks come before dispatch. The KV check happens in the engine. Mooncake’s early rejection is the same principle applied to disaggregation: a request rejected at decode time has already consumed prefill compute.\n\n**Admit before the first byte of a streaming response.** Once a server has sent `200 OK` and opened a server-sent event stream, it can no longer return a 429. The client gets a broken stream instead of a clean retry signal.\n\n**Bound `max_tokens`.** It is the worst-case KV reservation. Tight defaults per route or tenant increase effective capacity for free.\n\n**Measure load in tokens and KV blocks.** Request counts are easy to read and almost meaningless for LLMs. vLLM’s `max_num_queued_tokens`, Dynamo’s prefill-token threshold, and llm-d’s token-load scorer all moved to token-denominated signals for the same reason.\n\n**Watch hit rate, not only utilization.** Full KV memory with a high hit rate is healthy. Full KV memory with a collapsing hit rate is thrashing, and adding work makes it worse. CONCUR’s control loop is built on exactly this distinction.\n\n**Decide where the queue lives.** A queue in every engine gives the best batching but loses control over priority and placement: once a request sits in replica A’s queue, it cannot move to replica B when B frees up first. A central gateway queue keeps that flexibility. The llm-d “healthy buffer” setting is a good default.\n\n**Treat cache affinity as a load source.** Prefix-aware routing concentrates hot sessions on a few replicas. Admission and routing must share load signals so that a sticky node at 95% KV utilization stops attracting traffic, even when it holds the perfect prefix.\n\n**Account for both pools in disaggregated serving.** Prefill and decode saturate at different times. Admission needs a budget per role (Cognitora’s per-(model, role) counters, Dynamo’s separate prefill and decode thresholds) and ideally a prediction of decode load when the request will get there (Mooncake).\n\n**Use feedback and prediction to avoid oscillation.** Telemetry is always a few hundred milliseconds to a few seconds old. Admission that reacts only to current load accepts a burst, overloads, rejects everything, and repeats. AIMD windows (CONCUR), predictive admission (Mooncake), deferral queues that route with fresh metrics (Dynamo), and hysteresis between thresholds all dampen the cycle.\n\n**Choose fail-open or fail-closed on purpose.** When telemetry disappears, the admission layer has to pick one. llm-d treats stale metrics as saturated and stops dispatching, which protects GPUs but can halt traffic. Cognitora’s carbon admission fails open until the first successful poll, which keeps serving but loses the policy. Both are defensible. Leaving the choice to whatever the code happens to do is not.\n\n**Make rejections actionable.** The status codes differ across systems: 429 (TGI, Cognitora, optionally Ray Serve and llm-d), 503 (vLLM, Ray Serve default, llm-d with no endpoints), 529 (Dynamo). What matters is that clients can tell *overloaded, retry soon* apart from *invalid, do not retry*, that a `Retry-After` header is present where possible, and that client SDKs retry with exponential backoff and jitter.\n\n**Let clients say what they can tolerate.** Priority headers, sheddable classes, request TTLs (`x-llm-d-inference-ttl`), explicit deadlines, and `X-CGN-Priority: low` all tell the system which requests can wait or be dropped. Batch jobs and evaluation runs should almost always be sheddable.\n\n## 9. A practical starting point\n\nIf you run a fleet of vLLM replicas behind a cache-aware router, a reasonable first configuration maps one-to-one onto the five layers:\n\n1. **Validation** : authenticate, rate-limit per API key, and cap body size and sampling parameters at the gateway.\n2. **Context and budget** : set`max_model_len` deliberately and enforce per-route`max_tokens` defaults well below the context window.\n3. **KV and token budget** : set`max_num_seqs` and`max_num_batched_tokens` for your GPU and model, and watch the preemption counter. At the router, mark replicas busy at around 0.8 to 0.85 KV utilization.\n4. **Queue capacity** : bound the engine queue with`max_num_queued_reqs` (roughly`data_parallel_size × max_num_seqs` plus a small buffer) and`max_num_queued_tokens = target_TTFT × measured_prefill_throughput` . Return 429 or 529 with`Retry-After` .\n5. **Priority and flow** : tag interactive traffic as protected and batch or evaluation jobs as sheddable, with a queue TTL shorter than the client timeout.\n\nIf agents make up a large share of traffic, add a sixth control: cap **concurrent agents**, not just concurrent requests, and lower the cap when the KV hit rate drops while utilization stays high.\n\nThen alert on **goodput**, not throughput: the percentage of requests meeting their TTFT and TPOT targets, alongside the rejection rate per layer and reason. And load-test past saturation on purpose. The behavior of a serving system at 150% of capacity says more about its design than its peak tokens-per-second number.\n\n*Related reading on this blog: [LLM Inference: a learning roadmap](https://www.fratepietro.com/2026/llm-inference-learning-roadmap/), [llama.cpp, vLLM, and SGLang: Three Approaches to LLM Inference](https://www.fratepietro.com/2026/llama-cpp-vllm-sglang-inference-tools/), [Cognitora v0.9.1: gossip discovery, live KV prefix, and a native engine](https://www.fratepietro.com/2026/cognitora-inference-update-v0.9/), and [Google Cloud agentic AI architecture: private inference networking](https://www.fratepietro.com/2026/google-cloud-agentic-ai-architecture-private-inference-networking/).*", "url": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to", "canonical_source": "https://www.fratepietro.com/2026/admission-control-distributed-llm-inference/", "published_at": "2026-10-22 22:00:00+00:00", "updated_at": "2026-10-23 14:43:05.767514+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-agents", "mlops"], "entities": ["SGLang", "NVIDIA Dynamo", "llm-d", "GKE Inference Gateway", "Cognitora"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to", "markdown": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to.md", "text": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to.txt", "jsonld": "https://wpnews.pro/news/admission-control-for-distributed-llm-inference-cache-aware-routing-gpu-aware-to.jsonld"}}