ZONOS3-Mini

Zyphra released ZONOS3-Mini, a zero-shot voice-cloning text-to-speech model with 100 million active parameters and 800 million total parameters under an Apache 2.0 license, trained on 112 Nvidia H100 GPUs in 2.5 days. The sparse Mixture of Experts model runs in real time with under 570 ms Time To First Audio in a browser on a Pixel 9 and under 150 ms on an M3 MacBook Air, with runtimes for WebGPU browsers, SGLang and vLLM servers, and GGML, plus weights on Hugging Face and code on GitHub. Zyphra said the model offers controllability across identity, accent, quality, mood and delivery, with word-level delivery control via alignment attention head monitoring during streaming generation and a separate edit model after generation.

ZONOS3-Mini Introducing ZONOS3-Mini, a compact zero-shot voice-cloning Text To Speech TTS model with state-of-the-art performance as well as leading voice control capabilities which can run in real-time on-device. Introduction We are excited to announce ZONOS3-Mini, our next-generation real-time text-to-speech TTS model with high-fidelity voice cloning. ZONOS3-Mini is a sparse Mixture of Experts MoE model with 100 million active parameters and 800 million total parameters licensed under Apache 2.0. We trained ZONOS3-Mini on 112 Nvidia H100s in 2.5 days. You can access it on Zyphra Inference https://cloud.zyphra.com served on Nvidia H200s, on your own GPU, or in your browser on-device. With ZONOS2, we established that the tradeoff between quality and inference performance can be mitigated by using MoE architectures for TTS. However, ZONOS2 did so at the cost of 8B parameters on a codec running at 86 Hz which limited the deployment of the model. ZONOS3-Mini sidesteps these limitations thanks to large efficiency gains at multiple layers of the stack. Thanks to its size, ZONOS3-Mini can run on a multitude of devices. We release runtimes for browsers via WebGPU, servers via SGLang and vLLM, as well as many others through GGML. ZONOS3-Mini runs in real-time with <570 ms of Time To First Audio in the browser on a Pixel 9 and <150 ms on an M3 MacBook Air. Moreover the inference server comes with controllability across identity, accent, quality, mood, and delivery. Word level delivery control is accomplished through alignment attention head monitoring during streaming generation and separately with an edit model after generation. We encourage you to listen to the samples below in the relevant section. The ZONOS3-Mini model weights are available on Hugging Face, and our runtimes are available on our GitHub. Model Comparisons We present qualitative comparison of ZONOS3-Mini against a number of state-of-the-art proprietary and open-source models using our new ZTTS2-Eval benchmark described in detail below . These results capture the intelligibility, quality, voice-cloning performance and prosodic variation of model generations. We believe the best way to evaluate a TTS model is qualitatively, by listening to the generated samples. With that in mind, we present a series of ZONOS3-Mini samples alongside outputs from comparable open-source and proprietary TTS models: Qualitative Comparisons Trailer Voice “I must not overfit. Overfitting is the model-killer. Overfitting is the little loss that brings total generalization failure. I will face my loss. I will permit the gradient to pass backward over me and through me. And when it has propagated, I will turn the validation eye to see its path. Where the loss has gone there will be convergence. Only the learned representation will remain.” Parks and Recreation Guy “You leak your test set? Straight to jail. No validation, no nothing. Data scientists, we have a special jail for data scientists. You overfit? Jail You underfit? Also jail Overfit, underfit. You normalize the training data but not the test data? Jail, right away. You tune hyperparameters on the leaderboard? Believe it or not, jail You call it AI when it’s just logistic regression? Jail You use a neural net for a problem solved by a decision tree? Jail You forget to set a random seed? Jail You set the seed but don’t log the experiment? Also jail You say 'the model is 99% accurate' on an imbalanced dataset? Right to jail You deploy without monitoring drift? Jail, immediately.” Dwarkesh “I guess what’s interesting is that it no longer has that synthetic, slop-y cadence you expect from these models. It actually feels like you're talking to a person, right? And maybe the M.O.E thing is actually doing a lot of work here. You get the feel of a much bigger model but with great realtime performance at the same time.” Quantitative Comparisons Control Examples Voice Design Each comparison uses identical text across A, B and C. These 15 samples use the same narrator reference voice in embedding mode with neutral delivery. The gender and voice texture sets use seed 27182; the other sets use seed 2027. Only the listed design controls change. The settings are generation targets; perceived traits and speaker consistency should be assessed by listening. 1. Same speaker, three accents “I parked the car near the water, just past the coffee shop. I’ll meet you there after half past four.” Listen for changes in pronunciation while the voice remains recognizable as the same speaker. England steering is used as an approximation of the Southern English target. - british1.5 - australian1.5 2. Same voice at different ages “I found the old camera in a box upstairs. There’s still one photograph left, so let’s make it a good one.” Listen for a progression in perceived age through vocal texture and resonance. The ages are targets, not measured ages. - age−1.5 - age+1.5 3. Gender presentation “The next train leaves in ten minutes. We have just enough time to grab a coffee and find our seats.” Listen for changes in resonance, texture and pitch, while the base speaker, accent and delivery remain fixed. The intermediate gender setting targets an androgynous presentation. - gender−1.5 - gender−1 4. Voice texture “The wooden box is on the top shelf, beside the green bottle and the little brass clock.” Listen for moderate texture changes at ordinary speaking volume. Soft texture approximates breathiness; there is no dedicated breathiness control. - soft+5 - raspy+5 5. Compose a cast from multiple controls “The map ends here. Beyond that ridge, we’re going to have to find our own way.” Listen for whether age, gender presentation, accent and texture come through together as three distinct casting choices. - age−1.5 - bright+2.25 - australian1.5 - gender−1.5 - deep+3 - age+1.5 - soft+2.25 - raspy+1.5 - british1.5 Delivery Steering These 15 samples keep the narrator reference speaker and accent fixed. The whisper span set uses seed 22361; the other delivery sets use seed 2027. Whole-line delivery is steered unless a span of words is specified. Each A/B/C set uses identical text so the performance changes can be compared directly. 6. Whispering on a span “I didn’t say she took the keys.” B targets “I didn’t say” and C targets “she took the keys” with local whisper steering +4 . The neutral take uses the same sentence and seed. - whisper+4 - whisper+4 7. Same words, three emotional intentions “You came back. I wasn’t sure you would. I’ve been waiting here all evening.” Listen for different interpretations of the same situation. These are emotional targets; the controls do not guarantee a specific emotion or fully independent changes in pace and projection. - happy+3.5 - arousal+1 - angry+3.5 - arousal−1 - scared+8 8. Pace without losing clarity “The next train leaves from platform four at seven thirty. Please have your ticket ready before boarding.” Listen for a clear speed range with every word preserved, particularly the platform number and departure time. Pace steering can also affect loudness and pitch. - pace−2 - pace+2 9. Projection: nearby to across the room “Can you hear me? I’m right here, beside the gate. Follow my voice, and I’ll wait for you.” Listen for changes in breath, resonance and vocal effort as well as loudness. The emotional target remains calm across the set. - projection−3 - projection+3 10. Liveliness: restrained to animated “Inside the box were a silver key, a folded map, and a tiny doorbell. Naturally, I pressed it.” Listen for increasing pitch variation, rhythmic interest and expressive phrasing. Liveliness steering can also affect pitch level and duration. - lively−8 - lively+8 Training and Architecture Details For ZONOS3-Mini, we made substantial improvements at the level of the codec, the architecture, and the speaker encoder. The first major improvement is at the representation level. Descript Audio Codec DAC despite its extreme fidelity is incredibly hard to model and limits real-time speech generation due to its high framerate 86 Hz . To improve upon these problems we trained our own codec Zyphra Audio Codec 1 ZAC-1 . Under similar data constraints, a model trained with ZAC-1 achieves a 10x reduction in word error rate compared with one trained with DAC while achieving better acoustic similarity. When compared to leading TTS codecs like S1-DAC and MOSS v2 it achieves comparable results at better compression or better results at similar compression. It achieves this by creating a more favorable representation for the TTS backbone to model. Second, we integrate architecture improvements that have been developed in LLMs. The sequence mixers in ZONOS3-Mini follow a 3:1 hybrid of Gated DeltaNet GDN and global attention which helps us achieve stronger performance as well as higher throughput at longer context lengths. We also integrate latent MoE, Q balancing, and Attention Residuals which significantly improved performance at our model scale. Third, we change our voice representation from the Qwen3-TTS speaker encoder to the MECT-B2 speaker verification model. This both improves downstream speaker similarity and makes speaker manipulation more tractable. ZONOS3-Mini supports two modes of speaker conditioning. The first mode is trained from the start during pretraining with a separate clip in context. This achieves a more natural output at the cost of not copying prosody exactly. The second mode uses a direct voice representation of the target clip and is finetuned in a short stage after pretraining. Finally, we introduce two post-trained stages first using direct preference optimization DPO and then with Group Relative Policy Optimization GRPO to enhance robustness. Evaluation Following our previous release of ZTTS1-Eval, we now release the Zyphra TTS Evaluation v2 ZTTS2-Eval . ZTTS2-Eval expands the clean set of ZTTS1 to cover 35 languages and its in-the-wild set to cover 20 languages, both at a higher sample count per language. Qwen3-ASR is supplemented by Whisper Large v3 in lower-resource languages as well as English where it advantages Qwen TTS models. The TTSDS2 prosody metrics used in ZTTS1-Eval are superseded by new metrics for rhythm and pitch distribution matching as well as an adjusted Masked Prosody Model embedding distance with a scrambled input floor expanding the range spanned by models. With the increased number of metrics, especially across languages and conditions, we include a new Zyphra TTS Capability Index v1 based on the Epoch Capability Index using item response theory over 420 items metric x test set x language , each with its own difficulty and slope. ZONOS3-Mini ranks first among open models followed by Fish S2 Pro and MOSS TTS v1.5. Controllability In addition to state-of-the-art intelligibility and speaker similarity, ZONOS3-Mini features extensive control of speaker identity and delivery. Our control axes span identity across age, gender, accent, and quality through speaker embedding vectors and task vectors for accents specifically. Delivery control including pace, emotion, projection, and liveliness is achieved through Activation Steering. Along with steering at the utterance level, we leverage learned alignment heads in attention to steer delivery at the word level. Furthermore, an edit model is released based on a VoiceCraft recipe, allowing users to modify sentences seamlessly with the same control as normal generations. Inference In order to serve ZONOS3-Mini, we provide day 0 support through SGL. Additionally, we release a native C++/ggml runtime as well as a WebGPU runtime for running ZONOS3-Mini on devices ranging from top of the line server GPUs to multiple years old phones.