The AI Inference Stack in 2026: GPUs, KV Caches, Routing, and Kubernetes

A technical analysis of the AI inference stack in 2026 maps the path of a single request from application to GPU, detailing layers such as API gateway, inference gateway, distributed serving, inference engine, and runtime kernels. The piece emphasizes that infrastructure decisions determine token latency, reliability, and cost, and warns that GPU utilization alone is not a health metric. It advocates for controlled benchmarks that record software variables like CUDA versions and attention backends to ensure credible comparisons.

An application sends one request. A few hundred milliseconds later, the first token appears. The interface makes this look simple: JSON goes in, text comes out. Behind that boundary, a scheduler is admitting work, a serving engine is allocating KV-cache blocks, kernels are running on accelerators, a router may be choosing between replicas, and an observability system is trying to explain why the first token arrived late. This is the useful way to think about AI infrastructure in 2026: The model generates tokens. The infrastructure decides whether those tokens arrive quickly, reliably, and at a cost anyone can tolerate. This piece maps that system. It is deliberately broad. We will not declare a universal “best stack,” and we will not publish benchmark numbers without a controlled workload. Instead, we will follow one request from an application to a GPU and identify the responsibility of each layer along the way. Suppose an internal support assistant sends this request: { "model": "<served-model ", "messages": { "role": "user", "content": "Summarize this incident and propose the next diagnostic step." } , "stream": true} The application does not need to know whether the model is using tensor parallelism, whether another request shares its prompt prefix, or whether the selected worker has a long queue. The infrastructure cannot afford that ignorance. Here is the simplified path: Application / Agent │ ▼API Gateway: identity, quotas, request policy │ ▼Inference Gateway: model routing, endpoint selection │ ▼Distributed Serving: placement, scaling, cache-aware decisions │ ▼Inference Engine: scheduling, batching, KV-cache management │ ▼Runtime + Kernels: attention, matrix multiplication, collectives │ ▼GPU / Accelerator Metrics, traces, logs, and cost signals cross every layer. This is a map, not a law. A small deployment may collapse the first three layers into one process. A larger platform may split them across several control and data planes. Some projects span multiple layers. The important question is not “Which box owns this logo?” It is “Which component makes this decision, and what evidence tells us it made the right one?” GPUs and other accelerators execute the numerical work. Their memory capacity constrains which models and cache sizes fit. Their memory bandwidth, compute throughput, interconnects, and supported numerical formats influence latency and throughput. Yet “Which GPU?” is only the start of a capacity plan. A serving system also cares about: A GPU running at 95 percent utilization is not automatically healthy. If the request queue is growing and p95 time to first token is exploding, the device may be impressively busy while the service is failing its users. That is a recurring theme in this stack: resource utilization is evidence, not the objective. Below the serving engine sit model runtimes and optimized kernels. This layer handles the operations that dominate inference: attention, matrix multiplication, mixture-of-experts routing, sampling, and communication between devices. The details matter because two servers exposing the same HTTP API can execute substantially different paths. Attention backends, graph capture or compilation, quantization kernels, collective libraries, and parallelism strategies can all change performance and compatibility. This is also where a common mistake begins: treating “the GPU” as a single independent variable. A credible comparison must record the software path around the GPU — driver, CUDA or ROCm version, engine version, precision, attention backend, parallelism, and relevant configuration. Otherwise, “same hardware” does not mean “same experiment.” Keeping those variables explicit is what separates a benchmark from an anecdote. An inference engine sits between the request-facing service and model execution. It decides which requests run together, how memory is allocated, and how generated tokens are returned. In 2026, engines such as vLLM https://docs.vllm.ai/en/latest/ and SGLang https://github.com/sgl-project/sglang expose overlapping — but not identical — sets of capabilities. Current vLLM documentation includes PagedAttention, continuous batching, chunked prefill, prefix caching, multiple quantization formats, several parallelism strategies, an OpenAI-compatible server, and disaggregated execution paths. SGLang’s project documentation describes RadixAttention-based prefix caching, continuous batching, chunked prefill, speculative decoding, quantization, multiple parallelism modes, and prefill/decode disaggregation. The engine has four responsibilities worth understanding on their own. Admit and schedule requests. Interactive inference is not ordinary request/response compute. Requests arrive at different times with different prompt lengths and unknown generation lengths. The scheduler continuously decides which tokens to process next while honoring memory limits. Batch work continuously. A static batch waits for a fixed group to finish together. Continuous batching allows completed sequences to leave and waiting work to join over time. That can improve device utilization, but it also creates a tuning problem: maximizing total throughput and minimizing an individual request’s latency are not the same objective. Manage the KV cache. During inference, attention keys and values from earlier tokens are retained so they do not need to be recomputed for every generated token. This KV cache can consume a large share of accelerator memory, and its allocation determines how many active sequences the engine can serve. vLLM’s PagedAttention design https://docs.vllm.ai/en/latest/design/paged attention/ divides KV-cache data into fixed-size blocks. That extra level of indirection helps allocate cache memory as requests grow instead of reserving one large contiguous region for every possible sequence. Stream results. The engine converts generated token IDs back into a response stream. Streaming improves perceived responsiveness, but it also gives us two different latency questions: One average “request latency” hides both behaviors. The KV cache begins as an engine memory-management concern. At scale, it becomes an infrastructure concern. Consider three requests that share a long system prompt and reference document: Request A: shared 10K-token prefix + Question ARequest B: shared 10K-token prefix + Question BRequest C: shared 10K-token prefix + Question C If each request lands on a worker that already holds the shared prefix, the system may avoid repeating some prompt computation. vLLM documents automatic prefix caching https://docs.vllm.ai/en/v0.13.0/features/automatic prefix caching/ for this kind of reuse. But the benefit is conditional: the requests must share reusable prefixes, the useful blocks must still be present, and the workload must spend enough time in prompt processing for the saved work to matter. Now add multiple replicas. Cache state is local unless the system deliberately coordinates or transfers it. A cache-blind load balancer may send the next request to an empty worker even when another worker has the relevant prefix. The scheduling problem is no longer just: Which worker has the shortest queue? It becomes: Which worker has useful state, enough free capacity, and a queue that will not erase the benefit of reuse? That is why modern distributed-inference projects treat cache locality as a routing signal. The llm-d request scheduler https://github.com/llm-d/llm-d/blob/main/docs/architecture/core/router/epp/scheduling.md , for example, follows a filter-score-pick lifecycle and can reason about endpoint state such as queue depth and KV cache. NVIDIA Dynamo documents a KV-aware routing mode https://docs.nvidia.com/dynamo/cli/kv-aware-routing/overview in which workers publish cache events so the frontend can choose a likely cache hit. A conventional load balancer often sees connections, requests, or coarse utilization. An inference-aware router may also consider: There is no universally correct score. A worker with the strongest cache match may also have the longest queue. The least-busy worker may require a complete prefill. A topology-local worker may reduce transfer time but have less free cache capacity. Kubernetes’ Gateway API Inference Extension https://gateway-api-inference-extension.sigs.k8s.io/ formalizes this idea around an inference gateway and endpoint picker. It extends gateway implementations with model-aware endpoint selection rather than assuming every backend endpoint is interchangeable. This is the boundary where application gateways and inference schedulers are easy to confuse: Some models cannot fit on one device. Some workloads need more replicas. Others benefit from separating the two main phases of autoregressive inference: These phases have different performance characteristics. Long prompts can create heavy prefill work, while long generations and many concurrent sequences create sustained decode pressure. Distributed-serving frameworks are increasingly making that split an infrastructure primitive. The original llm-d proposal https://github.com/llm-d/llm-d/blob/main/docs/proposals/llm-d.md describes an architecture around Kubernetes, an inference scheduler, vLLM, disaggregated serving, prefix-cache hierarchy, and autoscaling. llm-d is therefore not usefully described as “vLLM, but distributed.” vLLM is an inference engine; llm-d coordinates a broader Kubernetes-native serving system that can use model-serving workers. NVIDIA Dynamo https://docs.nvidia.com/dynamo/ occupies related distributed-inference territory and documents support for engines including vLLM, SGLang, and TensorRT-LLM. Its disaggregated-serving flow https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving routes a request through a prefill worker, transfers KV-cache state, and continues generation on a decode worker. Disaggregation is not free acceleration. Moving KV data adds coordination and transfer costs. Current Dynamo guidance explicitly notes that aggregated serving may remain simpler or faster for small models, short prompts, low concurrency, or clusters without a fast transfer fabric. That conditional is more useful than a feature checkbox — the real question isn’t whether disaggregation is possible, but where, for a specific workload, it starts paying for itself. Kubernetes is useful for deployment, isolation, placement, rollout, health management, and reconciliation. It can schedule pods onto nodes that advertise accelerators and replace failed workloads. But Kubernetes does not automatically know: This is why AI-infrastructure projects add inference-aware gateways, schedulers, metrics, and custom control loops around Kubernetes. The orchestrator supplies durable resource management; the inference layer supplies workload meaning. Traditional service metrics still matter: request rate, errors, resource saturation, and end-to-end latency. Inference adds token-shaped detail. At minimum, we need to connect: Then we need internal context: scheduler queue depth, active sequences, prefix-cache hits, KV-cache utilization, GPU memory, device utilization, model-loading state, and routing decisions. The goal is not to collect every metric. It is to explain a user-visible outcome with the smallest trustworthy chain of evidence — built from request, engine, cache, and accelerator signals together, and mindful that OpenTelemetry’s generative-AI semantic conventions https://github.com/open-telemetry/semantic-conventions/blob/main/docs/gen-ai/gen-ai-metrics.md are still young enough that their maturity and version are part of the environment too. A benchmark client is not a final garnish. It is how we prevent this from becoming a sequence of attractive diagrams followed by opinions. Any real performance claim needs the same workload contract recorded alongside it: model and immutable revisionengine and versioncontainer image digestGPU model, count, and memoryCPU, RAM, driver, and CUDA or ROCmprecision, quantization, and parallelismprompt source and input-token distributionoutput-token limitsrequest count and arrival patternconcurrencywarm-up and cache staterandom seedclient placementerrors and cancellations The standard result set should include TTFT, ITL, end-to-end latency, throughput, errors, GPU utilization, GPU memory, KV-cache utilization, and queue depth where the system exposes them. Publish raw JSON or CSV beside every chart. Don’t compare an H100 run with a different model on another engine and call the difference architectural. Don’t quote a vendor benchmark as if you reproduced it. And don’t hide a p99 latency problem behind an average. The map is useful only if it improves decisions. Start with the smallest system that answers the workload’s real constraints. Start with one engine when Add orchestration and replicas when The AI Inference Stack in 2026: GPUs, KV Caches, Routing, and Kubernetes https://pub.towardsai.net/the-ai-inference-stack-in-2026-gpus-kv-caches-routing-and-kubernetes-c245a413dcd9 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.