{"slug": "self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1", "title": "Self-Hosting LLMs on Kubernetes: Securing llm-d and vLLM with Kong AI Gateway (Part 1)", "summary": "A two-part Kong blog series details how to secure self-hosted LLM inference on Kubernetes by pairing the open-source llm-d routing framework with vLLM behind Kong AI Gateway. llm-d routes requests across vLLM replicas using prefix-cache-aware routing, prefill/decode disaggregation and LoRA adapter affinity, while Kong AI Gateway restores API keys, rate limits and usage reporting that managed model APIs from OpenAI, Anthropic or cloud providers supply by default. The series positions self-hosting open-weight models such as Z.ai's GLM, Moonshot AI's Kimi and NVIDIA's Nemotron as a production option, with Part 2 running the optimized baseline and serving a LoRA adapter exposed through Kong as its own model alias.", "body_md": "[\\[llm-d\\](https://llm-d.ai)llm-d](https://llm-d.ai) is an open-source, Kubernetes-native framework for serving large language models at scale. It doesn't run models itself. It sits in front of inference engines such as [\\[vLLM\\](https://vllm.ai/)vLLM](https://vllm.ai/), another open-source, high-throughput serving engine, and decides how requests are spread across them.\n\nTeams taking a self-hosted LLM to production usually start with a managed model API. They call a hosted model from OpenAI, Anthropic or a cloud provider over HTTP and pay per token. When evaluating self-hosted LLM vs managed OpenAI cost trade-offs, the managed route means there are no GPUs to provision and no servers to run. With low or unpredictable traffic, it is usually the cheaper option, because you never pay for idle hardware.\n\nThe alternative is to self-host an open-weight model on your own GPUs, in your data center or your cloud account. Open-weight models are now a real option for production work: Z.ai's GLM, Moonshot AI's Kimi and NVIDIA's Nemotron families publish their weights and are built for coding, reasoning and agentic tasks. Compared with a managed API, this is the trade:\n\n- **What you gain:** control over the whole stack. Prompts and data stay in your environment, there are no provider rate limits, and you decide how inference runs. You can route requests to the GPU that already holds their cache, split prefill and decode into separate pools, or serve many fine-tuned LoRA adapters on one base model. Furthermore, when comparing vLLM vs OpenAI API latency, self-hosting eliminates unpredictable network hops and provider-side queueing, offering consistent performance dictated entirely by your own hardware's KV-cache reuse.\n- **What you give up:** simplicity, and the protections a provider gives you. You operate the GPUs, the serving engine and the scheduling. The GPUs cost the same whether traffic arrives or not, so how busy you keep them decides your cost per token. And the endpoint comes with no API keys, rate limits or usage reporting.\n\nllm-d and Kong AI Gateway each address one part of what you give up. When the GPUs are yours, every request that lands on the wrong one wastes money. llm-d makes self-hosted inference efficient by routing each request based on what every vLLM instance is doing. Kong AI Gateway adds back the API keys, rate limits and usage reporting a managed API would have provided.\n\nllm-d's main use cases are about getting the most out of those GPUs:\n\n- **Prefix-cache-aware routing:** each vLLM replica keeps a KV cache, the keys and values it computed for the prompts it has seen, and reuses it when a new request starts with the same text. A plain Kubernetes Service sends requests to any replica, so a repeated system prompt or conversation history is computed again on each one. llm-d's router estimates which replica already holds a request's prefix and sends it there, falling back to the least loaded replica when that one is busy. Part 2 runs this as llm-d's optimized baseline.\n- **Prefill/decode disaggregation:** reading the prompt (prefill) is limited by compute, while generating tokens one at a time (decode) is limited by memory bandwidth. llm-d can run the two phases in separate pods and move the KV cache from one to the other, so each pool is sized for its own work and long prompts don't stall token generation. Part 2 runs it on the four GPUs of one node.\n- **LoRA adapters:** a LoRA adapter is a small set of extra weights that specializes a base model for one task. vLLM serves adapters on top of a single copy of the base model, and llm-d's router prefers the replicas where the requested adapter is already loaded. Part 2 serves one adapter, exposed through Kong as its own model alias.\n\nllm-d covers more than this series does. For example, wide expert parallelism spreads a large Mixture-of-Experts model, too big for one server, across the GPUs of several nodes.\n\nThis blog post is the first of a two-part series:\n\n- **Part 1 (this post)** describes how llm-d, vLLM and Kong AI Gateway work together. It covers the llm-d architecture and the role each layer plays, then builds the stack on Amazon EKS: vLLM serving Llama 3.2 3B, an llm-d router in front of it, and Kong AI Gateway 2.3 exposing the model with per-application API keys and a per-consumer token rate limit.\n- **Part 2** uses the same deployment to run three llm-d use cases: the optimized baseline, compared with a plain Kubernetes Service; P/D disaggregation, which splits prefill and decode across separate GPUs; and LoRA adapters, served next to the base model behind their own Kong model alias.", "url": "https://wpnews.pro/news/self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1", "canonical_source": "https://konghq.com/blog/educational/securing-llmd-vllm-with-kong-ai-gateway", "published_at": "2026-11-05 17:00:00+00:00", "updated_at": "2026-11-06 13:22:04.392577+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops", "ai-agents"], "entities": ["Kong AI Gateway", "llm-d", "vLLM", "Kubernetes", "OpenAI", "Anthropic", "Z.ai", "Moonshot AI"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1", "markdown": "https://wpnews.pro/news/self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1.md", "text": "https://wpnews.pro/news/self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1.txt", "jsonld": "https://wpnews.pro/news/self-hosting-llms-on-kubernetes-securing-llm-d-and-vllm-with-kong-ai-gateway-1.jsonld"}}