Announcing New vLLM and SGLang Inference Engines on TPU Google launched native PyTorch backends for the vLLM and SGLang inference engines on TPU, built on its TorchTPU project that runs PyTorch natively on TPU, with vLLM developed in partnership with Inferact and SGLang with RadixArk. The engines keep each framework's worker model, scheduler, KV cache manager and API server while adding TPU implementations and custom Pallas kernels, and use the open-source C++ TPU Sync library for KV cache transfer in disaggregated serving and tiered KV offload. Google said both engines are already running at scale in production with preview customers. As part of our broader announcements https://cloud.google.com/blog/topics/ai-infrastructure/torchtpu-is-now-open-source-run-native-pytorch-on-tpus for PyTorch today, we are excited to launch native PyTorch backends for vLLM developed in partnership with Inferact and SGLang developed in partnership with RadixArk . For years, TPUs have powered large-scale AI workloads with high performance and efficiency. With the rise of agentic and multimodal workloads, serving efficiently is more important than ever. vLLM and SGLang already run on TPU through JAX-based backend, but we kept hearing from developers that they wanted to customize and debug their serving stack in PyTorch. So we started in earnest on TorchTPU https://developers.googleblog.com/torchtpu-running-pytorch-natively-on-tpus-at-google-scale/ , which runs PyTorch natively on TPU. We then built new vLLM https://github.com/vllm-project/vllm-torchtpu and SGLang https://github.com/sgl-project/sglang-torchtpu engines on top of it. Our goal is for the PyTorch community to bring models to TPU and make them fast using the tools developers are already familiar with. The engines come with optimized, production-ready serving features and custom Pallas kernels. Serving at scale requires co-design across every layer of the technology stack, from silicon to compilers to orchestration. Today's launch of new inference engines is at the central part of an integrated, open TPU ecosystem. TorchTPU makes the TPU a native PyTorch device. Tensors are managed as standard PyTorch tensors on a tpu device with no wrapper types, code runs eagerly or compiles via torch.compile using XLA, and collectives go through torch.distributed . Because each device runs its own PyTorch process, both engines keep each framework’s worker model, scheduler, KV cache manager and API server etc, and add TPU implementations underneath. Most models use the standard upstream PyTorch model definitions in vLLM or SGLang, with TPU-specific optimizations in kernels and layer implementations. Custom kernels written in Pallas are called directly from PyTorch as custom operators inside the compiled graph. This platform-plugin design reuses as much of each framework as possible while letting TPU optimizations move at their own pace Both engines are built for the full range of production workloads. Here’s the list of supported features: Agentic workloads in particular demand several of these features simultaneously. Because agents repeatedly resend expanding multi-turn conversations, serving them efficiently requires finding the cached prefix instead of recomputing it. Both engines keep reusable prefixes in HBM and, with host offload turned on, automatically move older prefixes to host memory as a second tier. Two of these features move the KV cache away from the device that computed it. Disaggregated serving runs prefill and decode on separate TPU hosts, so each request's KV cache has to travel from the prefill host to the decode host.Tiered KV offload moves the part of the KV cache that no longer fits in HBM to the host's DRAM and secondary storage with Google Cloud Managed Lustre, and brings it back when a later request reuses the prefix. Either way, the transfer has to overlap computation or it adds directly to latency. Both engines use TPU Sync https://github.com/google/tpu-sync for this. It is an open-source C++ library for KV cache transfer and management on TPU. It reads the device buffers behind PyTorch tensors directly, so a transfer needs no extra staging copy on the host, and it runs transfers on its own threads outside Python while prefill and decode continue. vLLM-TorchTPU exposes it through vLLM's KV connector interface. SGLang-TorchTPU uses it as a disaggregation transfer backend and as the I/O backend for HiCache. Both engines are already running at scale in production with preview customers. To provide transparent, reproducible proof of our performance, we benchmark in public through the community-standard SemiAnalysis InferenceX project. Building on our initial submission https://newsletter.semianalysis.com/p/tpu-inferencex-full-steam of Qwen 3.5 on Ironwood TPU TPU 7x , our engineering teams have further optimized the stack, bringing the lowest cost per million tokens overall at 2.01 cents at high-concurrency configurations. Alongside these improvements, we have also published new state-of-the-art SOTA performance results to AgentX https://inferencex.semianalysis.com/inference/qwen-3-5?g model=Qwen-3.5-397B-A17B&g rundate=2026-10-19&i seq=agentic-traces&g tco=external&i prec=fp8&i xmode=interactivity , the project's dedicated track for evaluating agentic long-context and multi-turn workloads. Ironwood TPU leads across most of the Pareto frontier. We look forward to continuously updating our public performance profiles as we bring more models and configurations to the leaderboard. We did not build these backends in a vacuum. By embracing PyTorch and working directly with open-source project maintainers, we ensure these engines remain upstream-aligned and community-first: SGLang-TorchTPU was co-developed with RadixArk founded by the creators of SGLang This partnership is bringing SGLang's stellar team to build for TPU, delivering an engine that is already running in production at Meta and battle-tested at scale across demanding real-world scenarios.