Google Open-Sources TPU Sync for KV Cache and Weight Transfer on TPUs

Google Cloud open-sourced TPU Sync, an Apache 2.0 C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage and the data center network, alongside the announced TPU 8tx and TPU 8i Max systems. Google reports median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency, and up to 90% lower average TTFT on 64K-context prefix-caching workloads. The library ships Python wheels for JAX and PyTorch with KV cache manager, KV store and weight synchronizer APIs, acts as the KV connector for vLLM-TorchTPU and the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and is validated on TPU v7x and v6e, though only development builds are on PyPI and the RL weight-sync module remains experimental.

Google Cloud has open-sourced TPU Sync https://cloud.google.com/blog/topics/ai-infrastructure/introducing-tpu-sync-fast-kv-cache-and-weight-transfer-on-tpu , a native C++ library that moves KV cache blocks and model weights between TPU HBM, host DRAM, shared storage, and the data center network. GCP also announced the TPU 8tx and TPU 8i Max systems https://blog.google/innovation-and-ai/infrastructure-and-cloud/global-network/tpu-8i-max-and-8tx/ , the project targets two workloads where accelerators often wait on data: long-context and disaggregated LLM serving, and reinforcement learning loops that push updated weights from trainers to rollout samplers. For prefill/decode disaggregation, Google says median KV transfer latency for Qwen3 235B fell from several seconds to about 230 ms, with average TTFT reductions of 60% for that model and 90% for a 480B model at low concurrency. For prefix caching, it reports up to 90% lower average TTFT on 64K-context workloads. Developers get Python wheels for JAX and PyTorch, with KV cache manager, KV store, and weight synchronizer APIs. TPU Sync acts as the KV connector for vLLM-TorchTPU through vLLM's KVConnector interface, as the disaggregation and hierarchical-cache backend for SGLang-TorchTPU, and pairs with llm-d https://llm-d.ai/ 1.0 on GKE for prefix-cache-aware routing. The RL weight-sync module for Pathways and multi-controller JAX is experimental. The GitHub repository https://github.com/google/tpu-sync , which absorbed the earlier TPU Raiden codebase, warns that APIs are changing rapidly and that only development builds are on PyPI, and recommends pinning to its last-known-good commit. It is validated on TPU v7x and v6e, and the PyTorch wheel requires a matching torch tpu build that is not on a public index. On GPUs, similar problems are handled by NVIDIA Dynamo https://github.com/ai-dynamo/dynamo and its NIXL https://github.com/ai-dynamo/nixl transfer library, LMCache https://github.com/LMCache/LMCache , which offloads KV cache to CPU memory, disk, or remote stores, Mooncake https://github.com/kvcache-ai/Mooncake , the KVCache-centric serving architecture behind Moonshot AI's Kimi, and vLLM's own disaggregated prefill connectors https://docs.vllm.ai/en/latest/features/disagg prefill.html . TPU Sync differs mainly in being tied to PJRT and TPU tile layouts, and in covering RL weight transfer in the same library. Early reactions focused on data movement rather than the new chips, with developer Anand Butani https://x.com/AnandButani/status/2119830708516745520 sharing: TPU sync cutting kv transfer latency 90% matters more to me than the chip names, long-context serving mostly dies on moving cache between nodes. Another developer posting under the pseudonym @lgoodbookd https://x.com/lgoodbookd/status/2120081156624507050 ran rough numbers on why bandwidth alone does not solve the problem: 8i Max doubles HBM bandwidth to ~17.2TB/s, yet one 64K session on a 235B MoE still eats ~12.6GB of KV cache. That's why open-sourcing TPU Sync host-DRAM KV tiering, ~223GB/s per v7x host may matter as much as the new package. Bandwidth helps; capacity is still the wall. TPU Sync is available under the Apache 2.0 license, and its repository https://github.com/google/tpu-sync includes architecture documentation, C++ and Python APIs, and examples for single-host and multi-host disaggregated serving and host KV offloading. Until a stable release is cut, the JAX https://pypi.org/project/tpu-sync-jax/ and PyTorch https://pypi.org/project/tpu-sync-torch/ packages are published only as development builds, while the storage tier and RL weight synchronization remain experimental