Choosing TTS Based on Sound Quality Was Too Slow for Conversations — Separating 'Design' and 'Production' with a Measured 2.5x RTF Difference

📝 Originally published (in Japanese) at

[forge.workstyle.tech].

"Calm adult female narrator voice. Slow, warm, and trustworthy delivery,
carefully reading long sentences."

Pass this description, and it will speak in exactly that voice. You can generate any Japanese voice without preparing even one second of speaker audio. Moreover, the same caption and random seed will always produce the same voice.

I thought this could be useful for interactive avatars—eliminating the hassle of preparing voices for each character.

In short: it didn’t work for conversations. But it was too valuable to discard, so I repurposed it.

Our existing system used a pre-trained TTS (Style-Bert-VITS2-based). We synthesized the same 7.5-second sentence on the same GPU slice to compare.

RTF (Real Time Factor) = generation time ÷ audio length. Lower is faster.

At 40 steps, it’s 2.5× slower. Even cutting steps to 16 still leaves a 1.5× gap. And below 16 steps, quality collapses (6 steps was immediately rejected on listening).

On a smaller GPU slice, the gap widens further.

In conversation, audio is synthesized and played sentence-by-sentence. The delay until the first sentence plays determines the user experience. A 1–2 second lag per turn is prohibitive.

I considered allocating more GPU resources—from 2g.20gb to 7g.80gb (4× the compute).

But measurements showed a fixed overhead of ~1.1 seconds. Even cutting steps to 12, generation time bottomed out at 1.1–1.4 seconds. This is due to model and text processing—parts that don’t benefit from parallelism.

So even with 4× the resources, the fastest possible generation would be ~1.0–1.2 seconds—still slower than the pre-trained model’s 0.8 seconds. The cost multiplies. We decided not to adopt it.

A useful takeaway: measure fixed overhead first. By reducing steps to the point where performance stops improving, you can estimate how much of the bottleneck is parallelizable.

Speed was the only downside. But diffusion TTS offers something pre-trained models can’t: generating voices from captions alone. Pre-trained models can only reproduce voices seen during training.

So we split the pipeline:

[Design Phase] Diffusion TTS ── caption + seed → voice ── Training Corpus (~200 clips)
                                                          ↓
                                                        Training
                                                          ↓
[Runtime] Pre-trained Model ────────────────── Voice heard by users

During design, diffusion TTS creates “this kind of voice.” We record ~200 lines with that voice to build a training corpus, then train a lightweight model for runtime use.

Since diffusion TTS runs only once per voice, its slowness isn’t a problem. Generating one voice takes ~70 minutes, but it’s a batch process.

Deterministic generation. The same caption and seed must always produce the same voice. Without this, we couldn’t reproduce the designed voice, and every retraining would yield a different speaker.

Thanks to this property, we only need to store the design ledger to recreate the model. Even if we lose the trained model file, we can regenerate the exact voice from the caption and seed. (We once lost the driver script, but recovered the voice by restoring the caption and seed from logs: [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]).

Audio quality must survive training. Since generated audio becomes training data, diffusion TTS’s audio quality sets the ceiling. We verified this by actually training and listening—no issues here.

Speaker identity must remain consistent. Over 200 clips, the voice must stay consistent. This was tricky. Starting sentences with exclamations sometimes changed the speaker at the beginning, and strong emotional expressions could break speaker identity. We solved this by carefully managing reference audio—but that’s another story.

After splitting the workflow, we discovered a constraint: captions don’t just define timbre—they also dictate speech rate and intonation.

We tried designing a male counselor voice with this caption:

A calm, gentle male voice. Low and slow, enveloping the listener with reassurance.

The result was criticized as “too low and too slow,” so we revised the caption:

A calm, gentle male voice. A moderately low voice that reassures the listener,
spoken at a natural pace with careful, warm delivery.

F0 rose from 105–159Hz to 136–159Hz—good. But speech rate jumped from 5.5–6.7 to 7.3–7.7 syllables per second.

Even adding “a bit slower” brought it only to 6.4–7.1. F0 and speech rate are coupled in the caption—you can’t adjust one independently.

After testing five variants, we concluded: separating pitch and tempo via caption alone is difficult. Raising pitch inevitably increases tempo.

Moreover, speech rate cannot be changed after synthesis (see [[speaking-style-is-baked-into-the-corpus|Speech Rate Is Baked Into the Corpus]] for measurements). This means the combination of pitch and tempo is fixed at caption-writing time, with no post-hoc adjustment.

Our new workflow: decide upfront, generate multiple candidates, and compare. Since generating one candidate takes only a few minutes, running 5 variants × 12 seeds to build a comparison table was feasible.

A record of designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 1: Design.

(This article is the entry point.)

→ Next: [[deterministic-voice-gacha-and-design-ledger|Voice Gacha and the Design Ledger]]

Full 18-part series

1. [[diffusion-tts-too-slow-for-conversation|The TTS We Chose for Quality Was Too Slow for Conversation]] ← You are here

All insights are consolidated in [[拡散TTSから実用ボイスを量産する製造パイプライン|Mass-Producing Practical Voices from Diffusion TTS: Manufacturing Pipeline]].