Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

Siri Expressive Voices synthesize rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced, Apple’s most powerful on-device foundation model. This work presents the memory-efficient audio synthesis architecture behind that capability: a detokenizer that converts the semantic audio tokens emitted by the foundation model into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). We convert semantic audio tokens to a residual vector quantization (RVQ) representation with a three-component design—a streaming encoder, a temporal decoder, and a depth decoder—that systematically decouples temporal and depth processing. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing the dedicated per-level decoders of prior multi-decoder architectures, while causal sliding window attention with fixed-window key-value caching yields constant memory complexity independent of sequence length. Deployed on the AMX, the detokenizer sustains roughly 10ms per generation step—about 16x faster than real time—with a peak runtime memory of only ∼21MB and 329MB of on-device assets, enabling continuous streaming synthesis of 20–320 seconds of audio alongside the on-device foundation model. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Comprehensive ablation studies validate the effectiveness of key architectural components, including DiT conditioning mechanisms, temporal lookahead processing, and unified depth decoding strategies. Audio quality assessment through phonetic discriminability analysis, perceptual quality metrics, and neural quality estimation confirms that the proposed architecture maintains synthesis fidelity while achieving computational efficiency gains over existing methodologies. The proposed architecture is deployed in production as part of Siri Expressive Voices, powering a voice overhaul with Pace and Expressivity customizations sliders in Apple Devices and support for custom assistant voices. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score (MOS) by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.