{"slug": "introducing-kinference-an-open-source-inference-control-plane-for-vllm-and", "title": "Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes", "summary": "Buoyant released Kinference, an open source inference control plane for vLLM and Kubernetes that adds token rate limiting, tenant usage and cost tracking, KV-cache-aware load balancing, and inference policies to self-hosted open weight model deployments. In Buoyant's public benchmarks against standard Kubernetes load balancing, Kinference cut median time-to-first-token by 95% and median end-to-end latency by 45% on one workload, and served 7.4× the goodput on a shared-prefix workload and about 3.5× on multiturn. Kinference accepts OpenAI- or Anthropic-compatible traffic at its HTTP endpoint, so teams can point existing inference clients at it without changing their stack.", "body_md": "# Introducing Kinference: An Open Source Inference Control Plane for vLLM and Kubernetes\n\nWe’re excited to announce [Kinference \\[kay-inference\\]](https://kinference.ai/), a new open source project for building scalable inference platforms with vLLM.\n\nWe created Kinference at [Buoyant](https://buoyant.io/) after watching our customers struggle to run open weight models in their environments. We designed Kinference to be the control plane layer for the platform you need to build. It uses vLLM to do the underlying inference, and provides the set of features necessary for building an inference platform in practice. This includes token rate limiting, tenant usage and cost tracking, intelligent load balancing, and inference policies.\n\nKinference can be paired with your existing stack. Using Kinference is simply a matter of pointing inference clients to its HTTP endpoint and sending OpenAI or Anthropic-compatible inference traffic there.\n\nKinference serves a heterogeneous, evolving set of models running on a heterogeneous, evolving pool of GPUs, to a heterogenous, evolving set of clients..\n\n## **Inference load balancing and performance**\n\nIn addition to policies and a routing layer, Kinference provides inference-specific load balancing that takes into account signals like GPU queue sizes, historical performance, and the contents of the KV cache on the GPU. This KV-cache awareness, in particular, allows Kinference to send requests to the vLLM worker that will do the least work for that specific request.\n\nModeling the KV cache content of the GPU can produce dramatic gains in performance for distributed inference. For example, in our [public Kinference benchmarks](https://buoyant.io/blog/distributed-inference-benchmarks-kinference-vs-plain-kubernetes), we showed that, when compared to standard Kubernetes load balancing (labeled “service” below), **Kinference (labeled “kinference”) was able to cut median time-to-first-token (TTFT) by 95% and median end-to-end latency by 45%** of requests on one workload type.\n\nBy incorporating cache awareness into policies, platform owners can create policies that will send simple requests to the cheaper model — but only if that’s less work than sending them to the expensive model in the first place.\n\nIn our *unloaded latency* benchmark above, we sought to measure the effect of Kinference’s KV-cache-aware load balancing in isolation. In our *capacity* benchmark, below, we measured the requests per minute can each endpoint can serve within a latency SLO.\n\nIn the capacity benchmark, Kinference served **7.4× the goodput on shared-prefix and about 3.5× the goodput on multiturn**. It was able to attain our chosen SLO at concurrency 32 (the highest measured) on our shared-prefix workload, and concurrency 8 on the multiturn workload, whereas the kube-proxy approach was not able to attain the SLO at any concurrency level.\n\nRead more details on our benchmarking setup, process, and findings [here](https://buoyant.io/blog/distributed-inference-benchmarks-kinference-vs-plain-kubernetes).\n\n## **A policy-driven approach**\n\nCentral to Kinference is the idea of *inference policies*.\n\nInference policies allow platform owners to balance tradeoffs across usage, performance, and priorities. We observed our customers tackle questions like:\n\n- How do we allow users to request not just specific model checkpoints (“Qwen 3.8 with 27B params and the FP8 quantization, please”) but broader classes of models, such as “any Qwen 3.8 model” or “whatever is good for coding”?\n- How do we redirect “easy” inference requests to cheaper models, even if the user requested an expensive model? And how do we do that without sending the whole prompt history to a cold GPU and incurring a huge latency hit?\n\nRather than build a set of one-off features, we built a policy layer in Kinference that can express them all—and many others as well.\n\nFor example, here’s how you can define a YAML policy for Kinference to semantically classify prompts by complexity level, and route simpler prompts to cheaper models—unless, taking cache usage into account, it will be faster to send it to an expensive model.\n\n```\napiVersion: kinference.buoyant.io/v1alpha1\nkind: InferencePolicy\nmetadata:\n  name: smart-semantic-routing\n  namespace: inference\nspec:\n  rules:\n  - name: low-complexity-consider-qwen\n    matches:\n    - model: glm-5.3\n      tag: complexity-low\n    route:\n      consider:\n      - models: { names: [ qwen-3.6-27b, glm-5.3 ] }\n      select: [ { minimize: PredictedTTFT } ]\n```\n\nHere’s another example of using Kinference to burst the excess requests to cloud provider endpoints (like OpenAI, Anthropic, Databricks etc) when queue depth or cache pressure indicate GPUs are saturated.\n\n```\napiVersion: kinference.buoyant.io/v1alpha1\nkind: InferencePolicy\nmetadata:\n  name: burst-policy\n  namespace: inference\nspec:\n  rules:\n  - name: burst-to-cloud\n    route:\n      exclude:\n      - name: drop-overloaded-workers\n        drop:\n          backend:       { type: Local }\n          queuedPrompts: { greaterThan: 2 }\n\n      select:\n      - { type: Local,\n          minimize: PredictedTTFT }\n      - { type: Provider,\n          order: [bedrock, databricks] }\n```\n\nFinally, here’s an example of a policy that semantically classifies prompts according to whether they contain PII, and bursts requests to the third-party provider if the GPU pool is saturated.\n\n```\napiVersion: kinference.buoyant.io/v1alpha1\nkind: InferencePolicy\nmetadata:\n  name: pii-stays-local\n  namespace: inference\nspec:\n  rules:\n  - name: serve\n    route:\n      exclude:\n      - name: pii-never-leaves-cluster\n        if:   { tag: pii-detected }\n        drop: { backend: { type: Provider } }\n      select:\n      - { type: Provider,\n          order: [ databricks ] }\n      - { type: Local,\n          minimize: PredictedTTFT }\n      onEmpty: Reject   # fail closed\n```\n\nWith inference policies, platform owners can create explicit rules for how inference requests should be routed (or rejected) based on tenant, model requested, state of the underlying GPU pool, semantic classification of the prompts, and many other factors. These policies can go through human or agentic review; they can be committed to git; they can be linted and vetted; and they can be applied or rolled back independently.\n\nWe think this policy engine is critical for providing a true inference platform. See our [Kinference policy docs](http://kinference.ai/docs/)  for more.\n\n## **Inference metrics and SLOs**\n\nFinally, there is a comprehensive suite of inference-related metrics for platform owners to monitor, alert on, and build SLOs around. Kinference reports about what’s happening on the system, including not just its own health but:\n\n- Tokens rates in and out, and histograms of input and output token sizes\n- TTFT, ITL, and E2E latency histograms\n- Uncached vs cached prompt tokens, for cost estimation\n- Model usage, GPU saturation, etc\n\nAlmost every metric is also available per tenant, so you can track tenant usage, build specific SLOs around performance for specific tenants, and everything else you might need to ensure the platform you’re building is living up to its requirements.\n\n## **Try it today**\n\nGetting started is easy:\n\n### 01 Install the Gateway API CRDs\n\n**Note:** This guide installs Kinference into the `default` namespace. To install into a different namespace, override the namespaces in the following helm commands.\n\nThe first step is to install the [Gateway API](https://gateway-api.sigs.k8s.io/docs/introduction/) CRDs, which provide standard mechanisms for handling traffic.\n\n```\nGWAPI_VERSION=1.5.1\nkubectl apply --server-side --force-conflicts -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v$GWAPI_VERSION/standard-install.yaml\n```\n\n### 02 Install the Kinference gateway\n\nKinference has two basic components: a traffic gateway and an inference handler. Each is released independently and ships its CRDs as a separate chart, installed before the component itself.\n\nInstall the gateway CRDs and components:\n\n```\nhelm upgrade -i gateway-crds \\\n  oci://ghcr.io/buoyantio/charts/gateway-crds \\\n  --devel\n\nhelm upgrade -i gateway \\\n  oci://ghcr.io/buoyantio/charts/gateway \\\n  --devel --wait\n```\n\n### 03 Install Kinference\n\nNow, install Kinference and the Kinference CRDs:\n\n```\nhelm upgrade -i kinference-crds \\\n  oci://cr.kinference.ai/kinference-ai/charts/kinference-crds \\\n  --devel\n\nhelm upgrade -i kinference \\\n  oci://cr.kinference.ai/kinference-ai/charts/kinference \\\n  --set extension.enabled=true \\\n  --devel --wait\n```\n\n### 04 Make sure it’s working\n\n```\nkubectl get pods\n```\n\nVerify the processes are running:\n\n```\nNAME                           READY   STATUS    RESTARTS   AGE\ngateway-cp-566cddc46d-jsqlq    1/1     Running   0          8s\nkinference-7bb854cfc5-pp4t7    1/1     Running   0          8s\n```\n\nPlease always refer to the latest documentation at [http://kinference.ai/docs/](http://kinference.ai/docs/) as these steps will evolve over time.\n\nKinference is fully open source under Apache v2. It’s available on [GitHub](https://github.com/Kinference-AI/kinference) today. Feedback, issues and benchmark reports from other deployments are very welcome.", "url": "https://wpnews.pro/news/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and", "canonical_source": "https://vllm-project.github.io/2026/11/04/introducing-kinference.html", "published_at": "2026-11-04 00:00:00+00:00", "updated_at": "2026-11-05 00:51:33.377235+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "mlops", "large-language-models", "developer-tools"], "entities": ["Kinference", "Buoyant", "vLLM", "Kubernetes", "Qwen 3.8"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and", "markdown": "https://wpnews.pro/news/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and.md", "text": "https://wpnews.pro/news/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and.txt", "jsonld": "https://wpnews.pro/news/introducing-kinference-an-open-source-inference-control-plane-for-vllm-and.jsonld"}}