{"slug": "announcing-new-vllm-and-sglang-inference-engines-on-tpu", "title": "Announcing New vLLM and SGLang Inference Engines on TPU", "summary": "Google launched native PyTorch backends for the vLLM and SGLang inference engines on TPU, built on its TorchTPU project that runs PyTorch natively on TPU, with vLLM developed in partnership with Inferact and SGLang with RadixArk. The engines keep each framework's worker model, scheduler, KV cache manager and API server while adding TPU implementations and custom Pallas kernels, and use the open-source C++ TPU Sync library for KV cache transfer in disaggregated serving and tiered KV offload. Google said both engines are already running at scale in production with preview customers.", "body_md": "As part of our [broader announcements](https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus) for PyTorch today, we are excited to launch native PyTorch backends for vLLM (developed in partnership with Inferact) and SGLang (developed in partnership with RadixArk).\n\nFor years, TPUs have powered large-scale AI workloads with high performance and efficiency. With the rise of agentic and multimodal workloads, serving efficiently is more important than ever. vLLM and SGLang already run on TPU through JAX-based backend, but we kept hearing from developers that they wanted to customize and debug their serving stack in PyTorch.\n\nSo we started in earnest on [TorchTPU](https://developers.googleblog.com/torchtpu-running-pytorch-natively-on-tpus-at-google-scale/), which runs PyTorch natively on TPU. We then built new [vLLM](https://github.com/vllm-project/vllm-torchtpu) and [SGLang](https://github.com/sgl-project/sglang-torchtpu) engines on top of it.\n\nOur goal is for the PyTorch community to bring models to TPU and make them fast using the tools developers are already familiar with. The engines come with optimized, production-ready serving features and custom Pallas kernels.\n\nServing at scale requires co-design across every layer of the technology stack, from silicon to compilers to orchestration. Today's launch of new inference engines is at the central part of an integrated, open TPU ecosystem.\n\nTorchTPU makes the TPU a native PyTorch device. Tensors are managed as standard PyTorch tensors on a `tpu` device with no wrapper types, code runs eagerly or compiles via `torch.compile` using XLA, and collectives go through `torch.distributed`.\n\nBecause each device runs its own PyTorch process, both engines keep each framework’s worker model, scheduler, KV cache manager and API server etc, and add TPU implementations underneath.\n\nMost models use the standard upstream PyTorch model definitions in vLLM or SGLang, with TPU-specific optimizations in kernels and layer implementations. Custom kernels written in Pallas are called directly from PyTorch as custom operators inside the compiled graph. This platform-plugin design reuses as much of each framework as possible while letting TPU optimizations move at their own pace\n\nBoth engines are built for the full range of production workloads.\n\nHere’s the list of supported features:\n\nAgentic workloads in particular demand several of these features simultaneously. Because agents repeatedly resend expanding multi-turn conversations, serving them efficiently requires finding the cached prefix instead of recomputing it. Both engines keep reusable prefixes in HBM and, with host offload turned on, automatically move older prefixes to host memory as a second tier.\n\nTwo of these features move the KV cache away from the device that computed it. Disaggregated serving runs prefill and decode on separate TPU hosts, so each request's KV cache has to travel from the prefill host to the decode host.Tiered KV offload moves the part of the KV cache that no longer fits in HBM to the host's DRAM and secondary storage with Google Cloud Managed Lustre, and brings it back when a later request reuses the prefix. Either way, the transfer has to overlap computation or it adds directly to latency.\n\nBoth engines use [TPU Sync](https://github.com/google/tpu-sync) for this. It is an open-source C++ library for KV cache transfer and management on TPU. It reads the device buffers behind PyTorch tensors directly, so a transfer needs no extra staging copy on the host, and it runs transfers on its own threads outside Python while prefill and decode continue. vLLM-TorchTPU exposes it through vLLM's KV connector interface. SGLang-TorchTPU uses it as a disaggregation transfer backend and as the I/O backend for HiCache.\n\nBoth engines are already running at scale in production with preview customers. To provide transparent, reproducible proof of our performance, we benchmark in public through the community-standard SemiAnalysis InferenceX project.\n\nBuilding on our [initial submission](https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam) of Qwen 3.5 on Ironwood TPU (TPU 7x), our engineering teams have further optimized the stack, bringing the lowest cost per million tokens overall at 2.01 cents at high-concurrency configurations. Alongside these improvements, we have also published new [state-of-the-art (SOTA) performance results to AgentX](https://inferencex.semianalysis.com/inference/qwen-3-5?g_model=Qwen-3.5-397B-A17B&g_rundate=2026-10-19&i_seq=agentic-traces&g_tco=external&i_prec=fp8&i_xmode=interactivity), the project's dedicated track for evaluating agentic long-context and multi-turn workloads. Ironwood TPU leads across most of the Pareto frontier. We look forward to continuously updating our public performance profiles as we bring more models and configurations to the leaderboard.\n\nWe did not build these backends in a vacuum. By embracing PyTorch and working directly with open-source project maintainers, we ensure these engines remain upstream-aligned and community-first:\n\n**SGLang-TorchTPU was** **co-developed** **with RadixArk (founded by the creators of SGLang)**\n\nThis partnership is bringing SGLang's stellar team to build for TPU, delivering an engine that is already running in production at Meta and battle-tested at scale across demanding real-world scenarios.\n\n*<sup>\"SGLang-TorchTPU is running in production at Meta today, and getting there was a close three-way collaboration between Meta, Google, and the SGLang team to make the stack production-ready. It gives us a remarkably mature, PyTorch-native serving stack that integrates effortlessly into our existing AI infrastructure, with the performance, reliability, and flexibility we need to run frontier workloads efficiently on TPUs. \" said Zongwei Zhou, PyTorch-TPU lead at Meta.</sup>*\n\n*<sup>\"At RadixArk, TorchTPU has supercharged our development velocity as we roll out SGLang support across diverse silicon. It’s given us a painless path to native TPU acceleration, passing massive efficiency gains and flexible compute choices directly down to our customers,\" said Banghua Zhu, CTO and co-founder of RadixArk.</sup>*\n\n**vLLM-TorchTPU was** **co-developed** **with Inferact (founded by the creators of vLLM)**\n\nTogether, we integrated a high-performance Pallas [megakernel](https://inferact.ai/blog/tpu-megakernels) for the 2.8T-parameter Kimi K3 model that runs the entire decode step as one kernel, serving more than 700 tokens per second to a single user on Ironwood TPU. Building on this work, Inferact recently led our first preliminary [AgentX submission](https://inferencex.semianalysis.com/inference/kimi-k3) for Kimi K3 on the vLLM-TorchTPU stack. This initial milestone already demonstrates exceptional performance per TCO in ultra-high-interactivity workloads, and the rapid increase in the maturity of the vLLM and PyTorch ecosystem on TPU for doing new frontier OSS models bring-up, with follow-up submissions set to unlock even greater efficiency through larger scale prefill/decode disaggregation.\n\n*<sup>\"Our mission at Inferact has always been to make high-performance inference accessible, reliable, and cost-effective across the entire AI ecosystem. We are thrilled to partner on the launch of TorchTPU and bring first-class support for vLLM to TPUs. By bridging PyTorch’s optimized TPU capabilities directly into vLLM, developers can immediately run production workloads on TPUs with the high throughput, low latency, and memory efficiency they expect from our stack,\" said Simon Mo, CEO and co-founder of Inferact.</sup>*\n\nTo foster continuous open-source contribution, vLLM is also launching a first-of-its-kind community contribution TPU capacity pool to provide free TPU compute to developers who actively contribute to both engines. Request [here](https://docs.google.com/forms/d/1Jm0uUXvcwYJRrvRojfS2RzMLjaqlt2a7lNyALFZs-nQ/viewform?edit_requested=true).\n\nAs the models get larger, we are evolving both engines to make large-scale TPU inference deployments performant and efficient. The major features for production-grade agentic workloads are already supported. More performance optimizations and advanced features are under way – open a Github issue to request collaboration on yours!\n\nAcross the two engines, most of the popular open model families are supported, including Qwen, DeepSeek, GLM, Kimi, GPT-OSS, Gemma4 and more. The team is investing in making more models available to run natively on TPUs, including Day 0 support for new open models. We also want to give the community the tools and TPU capacity to tune performance for their own workloads and configurations.\n\n[vLLM-Jax](https://github.com/vllm-project/tpu-inference), and [SGLang-JAX](https://github.com/sgl-project/sglang-jax) remain important projects. Both will continue to be maintained, supported, and available for the community to use and adapt.\n\nGet started today:\n\nRelated announcements:", "url": "https://wpnews.pro/news/announcing-new-vllm-and-sglang-inference-engines-on-tpu", "canonical_source": "https://developers.googleblog.com/announcing-new-vllm-and-sglang-inference-engines-on-tpu/", "published_at": "2026-10-20 15:17:46.658465+00:00", "updated_at": "2026-10-20 15:17:48.674543+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "developer-tools", "large-language-models", "ai-chips"], "entities": ["Google", "vLLM", "SGLang", "TorchTPU", "Inferact", "RadixArk", "Pallas", "TPU Sync"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/announcing-new-vllm-and-sglang-inference-engines-on-tpu", "markdown": "https://wpnews.pro/news/announcing-new-vllm-and-sglang-inference-engines-on-tpu.md", "text": "https://wpnews.pro/news/announcing-new-vllm-and-sglang-inference-engines-on-tpu.txt", "jsonld": "https://wpnews.pro/news/announcing-new-vllm-and-sglang-inference-engines-on-tpu.jsonld"}}