Google has introduced the LiteRT Tensor API, a lightweight C++ API that allows developers to build, customize, and optimize machine learning models directly for on-device execution. According to Google, it combines the flexibility of native model authoring with cross-platform portability.
The traditional approach, Google says, is for developers to write computations manually in C++ for maximum control, which comes with a significant drawback:
Implementations are tightly coupled to specific backends (such as Metal, OpenCL, or Vulkan) and must be rewritten for each new hardware target. Furthermore, managing zero-copy buffer interoperability and synchronization across CPU, GPU, and NPU, while gluing together pre-processing, inference, and post-processing, demands substantial manual engineering effort.
The LiteRT Tensor API aims to address these challenges by allowing developers to build computation graphs operation by operation in C++. This provided greater control over model architecture, optimization, and accelerator execution, while enabling the same graphs to run through LiteRT on CPU, GPU, and NPU hardware without requiring a rewrite for each new target.
Beyond providing full control over model graphs, other key capabilities of the API include direct weight ingestion, allowing developers to load weights directly from checkpoints, including Hugging Face .safetensors files; zero-copy buffer interoperability, which avoids copying accelerator buffers across pre-processing, inference, and post-processing; and compatibility with AI coding agents, which can construct, modify, validate, and benchmark model graphs by generating C++ code.
Google used a coding agent to create a real-world app demonstrating how to use the LiteRT Tensor API for multi-object segmentation and tracking. The application also served as the basis for benchmarking performance across platforms:
We ran the same C++ code for the SAM 2 model on the GPUs of three devices: a macOS laptop (Metal), an iOS phone (Metal), and an Android phone (OpenCL). In our tests on the MacBook Pro, it ran about 1.5x faster than Apple MLX (fp16, image encoder plus mask decoder).
Google emphasized the fact that the coding agent could make a number of optimizations thanks to the model graph being defined in plain C++. Specifically, it "moved SAM 2's per-frame decisions, such as whether the object is still visible, from CPU code into the graph" and rewrote operations that triggered CPU fallback on WebGPU using GPU-compatible arithmetics.
Commenting on the announcement on LinkedIn, Arm director of AI Laurence Moroney emphasized that developers should always start with model conversion and turn to the C++ API only "when you need graph-level control or want pre and post-processing to stay on the accelerator". He also highlighted the benefits of defining model graphs in plain C++:
Going from under 10 to over 30 frames per second at 512x512 on WebGPU, with a coding agent doing the iteration, shows how much structured primitives help agents do real optimization work.
Google is also working on several planned improvmeents, including direct ML Drift backend support, additional NPU-friendly operations, automatic full-graph quantization, and expanded dynamic-shape for different image sizes, batch sizes, and sequence lengths.