Google / Wikimedia Commons (Public domain)
The C++ runtime already serves over 6 million queries per second inside Google, and now anyone can use it
Google has open-sourced the XLA Serving Engine, or XSE, a C++ runtime that serves compiled machine learning models with no Python involved.
Google announced the release on November 5, 2026, under the Apache 2.0 license. The company says the engine is already battle-tested at scale, handling over 6 million queries per second internally across products including YouTube Shorts and Google Ads.
Train in Python, serve in C++ #
XSE tackles that split directly. Developers can train deep learning models in JAX or PyTorch, then export them into a standalone C++ environment. Once exported, the model runs without the Python interpreter tagging along.
The engine sits inside the OpenXLA Project, an industry-backed effort that compiles machine learning computations into native code for different processors. XSE plugs into existing OpenXLA tooling, including the XLA compiler and StableHLO.
The performance numbers #
Google is making some hefty claims about efficiency. According to the company’s figures, XSE delivers 2–5 times the throughput of the best-optimized Python servers.
Against basic setups, the gap widens to as much as 36 times higher throughput.
Resource savings are the other headline. Google says XSE uses 30–70% less CPU per prediction than Python implementations.
AI, tech, and the markets they move—in one daily briefing.
Daily. Free. Join 34,000+ readers across crypto, finance, and policy.
Memory footprint drops by about 75%, meaning the engine needs roughly one-quarter of the memory a Python setup would.
Hardware portability is also part of the package. A single exported artifact can run on CPUs, NVIDIA and AMD GPUs, and Google’s Cloud TPU. Arm has contributed through its Neoverse work, extending the engine’s reach to Arm-based server chips.
Not built for chatbots #
XSE is not aimed at large language models. Google is targeting workloads like recommendation engines and vision applications.
YouTube Shorts and Google Ads, the two internal examples Google cited, are exactly that kind of workload.
Where OpenXLA fits #
The OpenXLA Project launched in 2023 with a goal of unifying the fragmented landscape of machine learning frameworks and accelerators.
OpenXLA’s answer is a shared compilation layer. Models from different frameworks get compiled into a common form, then optimized for whatever hardware they land on.
XSE fills in the serving side, giving the ecosystem a production-grade runtime. Releasing it under Apache 2.0 is a meaningful choice. That license is permissive, allowing commercial use and modification without forcing companies to open their own code in return.
Disclosure: This article was edited by Diego Almada Lopez. For more information on how we create and review content, see our Editorial Policy.