{"slug": "how-we-prepared-voice-cloning-tts-to-run-on-device", "title": "How we prepared voice cloning TTS to run on-device", "summary": "A developer detailed how they trained a text-to-speech model with voice cloning that fits into 100 MB and runs on-device, from smartphones to a Raspberry Pi. The model, about 45M parameters, uses a single-step flow sampler instead of iterative diffusion to achieve faster-than-realtime synthesis on budget hardware. The developer shared challenges with single-step sampling and mixed-precision training, including a fix for gradient poisoning that could invalidate thousands of training iterations.", "body_md": "We trained a text-to-speech model with voice cloning that fits into 100 MB (fp16) and runs on-device for synthesis, from smartphones to a Raspberry Pi. The model is in a file, generation happens on the device. No API, no cloud backend at inference time.\n\nEarly tests on a budget Android phone hit faster-than-realtime on a single core. Full benchmarks with numbers and different devices are coming in a separate post. Here is how we got there.\n\nThe model is about 45M parameters (a language model plus an audio decoder). For comparison, typical server-side TTS systems are hundreds of MB to several GB of weights. We trained both stages from scratch and packaged them into a single binary.\n\nThe road here was not smooth. This post is a short architecture overview (without it, it is hard to explain what broke) and two concrete stories of what broke and how we fixed it. If you end up training something similar, this should save you a few surprises.\n\nWe use the classic two-stage scheme. The first stage is a language model (LM) that, given text and a short voice sample, generates audio latents autoregressively, roughly the way an LLM generates text, except the outputs are continuous audio representations instead of words. The second stage is the decoder of our own neural codec (24 kHz, compact continuous latent space), which turns those latents back into a waveform.\n\nThe key design decision is a single-step flow sampler instead of iterative diffusion. Classical diffusion TTS often needs many forward passes per frame to sound right. That is fine on a server. On mobile or a Raspberry Pi, that does not work. One step per frame gives you speed on weak hardware. But you pay for it in training complexity, described below.\n\nTwo more things for context. The architecture is causal, so it streams by construction. You do not have to wait for the end of the input, and audio comes out with sub-second time-to-first-audio. Voice cloning works from a short sample of around 5 seconds. The cloning part runs off-device, the synthesis is on-device.\n\nDiffusion models work well partly because stochasticity accumulates over many steps. Each step adds a bit of noise and the model learns to remove it. By the end of the process you get a quality sample.\n\nWhen you do this in a single step, that stochasticity disappears. A naively trained one-step flow sampler tends to converge to an \"averaged\" sound. The model finds an output that minimizes loss on average across the training set, rather than a diverse, characteristic sound.\n\nWe saw this on early iterations as a lifeless \"muddled\" voice, identical in intonation regardless of input. Test recordings sounded like a robotic monotone from the nineties.\n\nWhat helped: a special sampler training scheme. I will skip the architectural details (that part of the recipe stays internal), but the general idea is to force the sampler to account for the target variance of the distribution, not just its mean. Verification was through a separate synthetic test set specifically for this failure mode, where voice diversity is expected. The model passed, it started to sound like real speech.\n\nThe takeaway: single-step sampling is not free. You save on inference and pay in training complexity.\n\nWe trained in mixed precision, some operations in fp16 for speed, some in fp32 for stability. Standard practice for large neural nets on GPUs.\n\nThe problem is that fp16 has a narrow range of representable numbers. Sometimes gradients overflow and you get a NaN. The usual protection (loss scaling and gradient clipping) works most of the time. But sometimes a rare combination of input data plus current weight state causes one operation to produce a NaN. That NaN gradient then poisons every parameter it touches at the next optimizer step.\n\nOne such \"poisoned\" step can invalidate 5000 previous training iterations. The model starts emitting noise, loss shoots to infinity, and everything since the last checkpoint is gone.\n\nThe fix was multi-layer protection. Per-batch NaN check, per-epoch sanity check, running-loss anomaly detection, then automatic rollback to the last good checkpoint plus a human alert. Not free compute-wise, but cheaper than losing a week of training. Mixed precision is a real engineering discipline, not a flag you enable and forget, especially on long runs where accumulated risk grows linearly with time.\n\nThe design constraints were tight.\n\nStreaming by construction, with sub-second time-to-first-audio on weak hardware. No \"wait a moment while the model thinks\".\n\nNumbers, dates, and currency without special-case hacks. The model reads \"March 21st, 2026\" and \"$19.99\" correctly. To get there, we added synthetic speech to the training mix with quality control specifically on those edge cases.\n\nAround 45M parameters, because 100 MB is what you can bundle into an app without a noticeable impact on bundle size. Any bigger and it gets uncomfortable for mobile apps.\n\nWhat we did not prioritize: language diversity (English only for now, the training corpus was around 3000 hours for the codec and 1700 hours for the LM), and extreme quality benchmarks for professional use. We aim for natural-sounding output on default use cases, not to beat top commercial TTS in blind studio tests.\n\nQuality is judged via blind listening tests on a fixed text set: numbers, dates, tongue twisters, long narratives. Not perfect, but reproducible, and raters agree with each other.\n\nThe model is going through final polishing (we are fine-tuning the decoder for the production generation mode) and packaging into the VoxRT runtime. The delivery format will be the same as our other SDKs: a single binary model file plus native SDKs for iOS, Android, and Raspberry Pi.\n\nRelease in September 2026. Code, model, and release notes will land on GitHub. Star the repo if you want a ping when it drops.\n\nMore about our other on-device SDKs (VAD, wake-word, KWS, streaming ASR) at [voxrt.com](https://voxrt.com?utm_source=devto&utm_medium=article&utm_campaign=tts-on-device). Models and SDKs on [GitHub](https://github.com/VoxRT).", "url": "https://wpnews.pro/news/how-we-prepared-voice-cloning-tts-to-run-on-device", "canonical_source": "https://dev.to/voxrtio/how-we-prepared-voice-cloning-tts-to-run-on-device-184", "published_at": "2026-09-07 14:05:00+00:00", "updated_at": "2026-09-07 14:28:30.632240+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "generative-ai", "ai-research", "ai-infrastructure"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/how-we-prepared-voice-cloning-tts-to-run-on-device", "markdown": "https://wpnews.pro/news/how-we-prepared-voice-cloning-tts-to-run-on-device.md", "text": "https://wpnews.pro/news/how-we-prepared-voice-cloning-tts-to-run-on-device.txt", "jsonld": "https://wpnews.pro/news/how-we-prepared-voice-cloning-tts-to-run-on-device.jsonld"}}