Google gooses Python AI performance by... removing Python

Google has open sourced XSE (XLA Serving Engine), a C++ serving runtime that compiles JAX-exported StableHLO computations into native code so models run without a Python interpreter in the serving path. Google director of product management Nathan Beach said XSE already handles more than six million queries per second across YouTube Shorts, Google Ads and Search, and in Google's tests on short requests it delivered two to five times the throughput of its best optimized Python server using a quarter of the memory. Google also plans integration with TorchTPU so PyTorch users can export models for C++ serving on TPUs, with XLA SavedModels running on GPUs, CPUs and Google's TPUs.

Google gooses Python AI performance by... removing Python The Register https://www.theregister.com Newly open sourced XSE handles six million queries a second across services including Search, Ads, and YouTube Shorts "Train in Python, serve in C++" might as well be Google's motto when it comes to much of its ML work. Python is easy for data scientists to wield, but keeping its interpreter in the serving pipeline can add overhead and deployment headaches. The industry-backed OpenXLA Project https://openxla.org/ provides tools for compiling machine learning https://www.machinebrief.com/glossary/machine-learning computations into native code for different processors. The latest component of this setup to be open sourced https://opensource.googleblog.com/2026/11/supercharging-xla-inference-xla-serving-engine.html is XSE https://xla-serving.readthedocs.io/en/latest/ , short for the XLA Serving Engine. One-shot inference https://www.machinebrief.com/glossary/inference XSE is not so much suited for large-scale LLM inference which would be better served by vLLM https://vllm.ai/ or SGLang https://www.sglang.io/ , but rather for smaller, more sharply task-oriented models, of which there are plenty at Google. "LLM serving gets most of the attention https://www.machinebrief.com/glossary/attention right now, but a significant amount of production inference is still ranking, recommendation, embedding https://www.machinebrief.com/glossary/embedding , and vision models. These tend to be short requests with very high queries-per-second and tight latency budgets," Nathan Beach https://www.linkedin.com/in/ndbeach/?lipi=urn%3Ali%3Apage%3Ad flagship3 feed%3Bg7EHPuI9RwyparIXdYRiBw%3D%3D , Google director of product management, wrote in a LinkedIn missive Tuesday. At Google, XSE already handles more than six million queries per second across services including YouTube Shorts, Google Ads, and Search. These models aren't there for a leisurely chat. They rank results, recommend videos, and make other decisions on the fly, often without users noticing. A Python interpreter mulling each request would be a bottleneck, if not the end of Google itself. In Google's tests on short requests, XSE delivered two to five times the throughput of its best optimized Python server, using a quarter of the memory. No Python here In this setup, the ML boffins write their model code in Python, using Google's JAX https://docs.jax.dev/en/latest/notebooks/thinking in jax.html to help with the sums. Google also plans integration with TorchTPU so PyTorch https://www.theregister.com/devops/2026/11/06/pytorch-215-tackles-the-one-bad-gpu-problem/5308227 users can export models for C++ serving on TPUs. An export tool packages the model's computations in StableHLO, a portable intermediate representation, alongside its trained weights in an "XLA SavedModel." Ready to precognate, the C++ serving runtime uses XLA to compile those computations into native code for the target silicon. No Python interpreter to see here. XLA SavedModels can run on GPUs and CPUs – Google partnered with both Arm and Intel to develop the XLA ecosystem. But Google obviously thinks it'd be a lot cooler if you ran your models on its own line of TPU accelerators https://www.nextplatform.com/compute/2026/11/09/google-rejiggers-tpu-accelerators-for-broader-ai-inference-work/5308534 . Python's sluggishness continues to be an issue for the ML community. Beyond XSE, other approaches have been taken, many of which involve jettisoning the Python code as quickly as possible. PyTorch https://www.machinebrief.com/glossary/pytorch offers a native C++ runtime called LibTorch https://docs.pytorch.org/cppdocs/installing.html . Also, Microsoft and its collaborators offer the ONNX Runtime https://onnxruntime.ai/ , which supports an open graph format with Python bindings. Even Python's own reference implementation https://www.theregister.com/devops/2026/09/14/cpython-eases-rust-requirements-as-assimilation-continues/5296010 relies heavily on C, with Rust now getting a foot in the door. ® Get AI news in your inbox Daily digest of what matters in AI.