# Teaching Kokoro / StyleTTS2 to Speak Yoruba Without Resizing Its Embedding Table

> Source: <https://ainexusdaily.vercel.app/article/2026-11-20-teaching-kokoro-styletts2-to-speak-yoruba-without-resizing-its-embedding-table>
> Published: 2026-11-20 14:51:40+00:00

**Recommendation:** Keep the embedding table at 178 tokens by mapping tone-marked vowels to unused slots, and trim silence and filter clips to 1.2–5.8 seconds so audio matches the dataloader's 1.44-second crop.
More in the playbook "Fine-tuning and data": <https://wpnews.pro/playbooks/training-data>

# Teaching Kokoro / StyleTTS2 to Speak Yoruba Without Resizing Its Embedding Table

Most open-weight TTS models treat pitch like seasoning. Stress a syllable here, drop your voice at the end of a question, and English listeners still understand you. Yorùbá does not work like that. Pitch is part of the spelling. Take the letters o-k-o (with underdots ọkọ): ọkọ (mid-mid) means husban

d ọkọ̀ (mid-low) means car or vehicle ọ̀kọ̀ (low-low) means spear If an English-trained model guesses a rising or falling melody on ọkọ, it says the wrong word. Over the past few weeks, I adapted the Kokoro / StyleTTS2 architecture to speak Yorùbá using a single Google Colab NVIDIA L4 GPU (22 GB VRAM) and 2.70 hours of cleaned audio. A lot of things broke along the way: dropped tones, raspy noise, speaker mix-ups, PyTorch 2.6 crashes, flat robotic speech, and even a 402 Payment Required surprise when I tried to deploy a free demo. This is the exact path I took, including the mistakes I made and where v1 still falls short.

oro-g2p and hunting for unused token slots Before touching any neural network weights, I needed a phonemizer that actually understood Yorùbá orthography. English phonemizers fail immediately here because gb is a single sound (/ɡb/), p is pronounced /kp/ (kp), and consonants like n or m nasalize vowels that follow them. Syllabic nasals (m, n, ŋ) also shift based on the next consonant, including stacked cases like nǹkan. I did not want people to install heavy system packages like espeak-ng, so I wrote oro-g2p, a zero-dependency Python library (pip install oro-g2p): from oro_g2p import g2p print(g2p("Ẹ kú àárọ̀")) # 'ɛˉ kuˊ aˋaˊɾɔˋ' Getting the text into IPA with tone marks (ˊ High, ˉ Mid, ˋ Low) was the easy part.

Feeding it into Kokoro was where I got stuck. My first attempt emitted the vowel and its tone as two separate tokens, like ["a", "ˋ"]. That sounded terrible in StyleTTS2. The duration predictor treated the tone mark like an extra letter, stretching trailing tones at the end of phrases or losing the link between a vowel and the pitch it was supposed to hold.

I did not want to resize Kokoro's embedding table (n_token = 178) because doing that breaks direct weight loading from pretrained checkpoints. So I printed out BASE_KOKORO_VOCAB to see what was actually inside those 178 IDs (0 to 177). Slot 0 is reserved (PAD/BOS/EOS), and 114 slots are already taken by Kokoro's base tokens. That left 63 completely empty slots sitting unused inside 1..177.

So I stopped sending vowels and tones separately. I packed every Yorùbá (vowel + tone) pair, (syllabic_nasal + tone) pair, and digraph directly into those 63 spare slots as single atomic IDs ("kp": 122, "ɡb": 124, "dʒ": 82, "aˊ": 7, "aˋ": 26, "ũˋ": 105, etc.), matched with a greedy longest-first tokenizer. Because n_token stayed 178, all pretrained tensor shapes lined up without touching the model architecture. Early training runs sounded awful, and I spent too long blaming hyperparameters before inspecting my dataset.

The biggest culprit was StyleTTS2's default dataloader crop. By default, it crops long audio down to max_len=120 (around 1.44 seconds) while still feeding the full transcript. A lot of my source clips were 6 to 11 seconds long. I was basically asking the model to listen to 1.4 seconds of audio while guessing how it aligned to a 10-second sentence.

Once I trimmed leading and trailing silence (top_db=35) and filtered my training set down to clips between 1.2s and 5.8s, alignment snapped into place. The second mess came from speaker IDs. StyleTTS2 grabs another random clip from the same speaker_id during training to estimate pitch baseline and voice style. Early on, I had multiple voices lumped under speaker_id = 0, which meant the pitch predictor was getting conflicting references on every batch.

Separating speaker IDs strictly cleared up a lot of random pitch jumps. I also learned a strange quirk about iSTFTNet the hard way at inference time: if you try to strip predicted silence frames before passing features into the decoder, InstanceNorm1d calibration gets thrown off and the audio comes out covered in raspy static. Waveform silence trimming only works after the decoder produces audio. After cleaning, my final Gold training split came out to 2.70 hours (2,168 clips) drawn from AfriSpeech/open-bible-speech-african (CC BY-SA 4.0) and google/WaxalNLP (CC-BY / CC-BY-SA-4.0).

Trying to run Stage 2 training on Google Colab with modern PyTorch 2.6+ threw a handful of upstream crashes unrelated to Yorùbá: models.py failed to load checkpoints without weights_only=False, choked on module. DataParallel key prefixes, and threw spectral_norm errors if .eval() ran on style_encoder or predictor_encoder before a 1-step .train() warmup. Modules/slmadv.py had 17 .squeeze() calls (wav.detach().squeeze(), y_pred.squeeze(), y_rec_gt.detach().squeeze(), y_rec_gt_pred.detach().squeeze(), real_GP.detach().squeeze(), etc.) that dropped the batch dimension completely if batch_size=1, crashing tensor shapes. I replaced those with a _wav2d(x) helper (x.reshape(x.shape[0], -1)) and added 1D/3D shape guards in Modules/istftnet.py.

train_second.py called pdb.set_trace() on NaN/Inf batches. If a batch blew up overnight on Colab, the script did not skip it or crash loudly, it just sat waiting for keyboard input while burning GPU units. I disabled set_detect_anomaly(True), wrapped y_rec_gt_pred in torch.no_grad(), added clip_grad_norm_(..., 5.0) on bert, predictor, and diffusion, and skipped NaN batches safely. I pushed these fixes upstream in semidark/StyleTTS2#4 (+47 / -19 across 4 files) and left a breakdown in semidark/kikiri-tts#46.

My Stage 2 runs on the L4 (batch_size=4) proved just how much diffusion and style loss weights matter for tonal speech: Round 1: I accidentally left lambda_diff=0.0 and lambda_sty=0.0. My F0 loss stalled at 2.245 and the voice sounded completely flat. Round 2: Turned both back on (lambda_diff=1.0, lambda_sty=1.0, lambda_F0=3.0, bert_lr=5e-5). Validation loss dropped to 0.313 and F0 loss fell to 1.921.

Round 3: Ran 15 epochs with lambda_F0=4.5, lambda_sty=1.5, lambda_diff=1.0, and bert_lr=4e-5. F0 loss reached a low of 1.758 along the way, finishing Epoch 15 (epoch_2nd_00014.pth) at Validation loss: 0.305, Dur loss: 0.974, F0 loss: 1.823. epoch_2nd_00014.pth was 2,091.5 MB on disk because it carried training discriminators and optimizer state. Filtering down strictly to INFERENCE_MODULES compressed the release checkpoint (kokoro_yoruba_gold.pth) to 526.0 MB.

During listening tests, I still noticed two annoying glitches: Vocal shaking on multi-tone vowels like láàárín. Locking diffusion parameters to alpha=0.0, beta=0.55, embedding_scale=1.15, diffusion_steps=15, and speed=1.0 stopped the tremor. Rushing on long text. Passing 400+ character paragraphs all at once made the model speed up unnaturally.

Splitting text by punctuation (0.18s pause on commas, 0.35s pause on periods, plus appending a trailing . if missing) fixed the pacing. In Python, running inference looks like this: from tts_yoruba import YorubaTTS tts = YorubaTTS(styletts2_dir="./StyleTTS2") tts.speak( "Ẹ kú àárọ̀ o! Báwo ni àwọn ará ilé?", voice="spk8", out_path="greeting.wav", ) When I tried to deploy a live Gradio Space on Hugging Face's free cpu-basic tier, it threw 402 Payment Required.

Since I did not want to pay for idle GPU hosting or keep people from testing it, I split the public demo into two pieces: a 24/7 static HTML audio showcase on Hugging Face Spaces so anyone can hear real v1 outputs right away, paired with a 1-click Google Colab Gradio notebook (which I redesigned after checking live browser screenshots so all 9 preset phrases fit without scrolling or clipped sliders). If you try this model today, keep its v1 limits in mind: Small, lopsided training data (2.70h total): Speaker 8 (BibleTTS) makes up ~92% of the audio and matches the reference voice closely (~98% in my A/B listening checks). Speaker 2 (Waxal) is only ~1% of the data, so that voice is noticeably weaker with slight static. English-pretrained PLBERT: Because the text encoder never saw native Yorùbá during pretraining, rare 4-to-6 syllable compound words (like Ṣàwòròidẹ or ògbóǹtarìgì) can lose tone accuracy.

Full diacritics required: There is no auto-diacritizer in v1 yet. Plain ASCII Yorùbá without tone marks defaults to mid-tones. Pure Yorùbá only: Raw English words get read with Yorùbá spelling rules (for example, "phone" maps p to /kp/ as kphoˉneˉ). Rather than leaving everything jammed inside one repo, I broke the project into three standalone parts: oro-g2p (MIT) — Just the pure-Python Yorùbá G2P and Kokoro atomic vocab (pip install oro-g2p).

kokoro-tonal-kit (MIT) — The starter kit (patch_styletts2.py, vocab_template.py, export_release.py, Stage 2 config notes) in case you want to adapt Kokoro / StyleTTS2 for another tonal language like Igbo, Twi, Hausa, or Vietnamese. kokoro-yoruba — The Yorùbá inference wrapper (tts_yoruba.py) and 1-click Colab Gradio demo wired up to the CC BY-SA 4.0 weights. Audio Showcase (HF Space): huggingface.co/spaces/Akintunde102/kokoro-yoruba-demo Live Gradio Demo (Colab): Open in Google Colab Model Weights (CC BY-SA 4.0): huggingface.co/Akintunde102/kokoro-yoruba-v1 Yorùbá TTS Repo: github.com/Akintunde102/kokoro-yoruba Tonal Starter Kit: github.com/Akintunde102/kokoro-tonal-kit Yorùbá G2P: github.com/Akintunde102/oro-g2p | PyPI Upstream StyleTTS2 PR: semidark/StyleTTS2#4 For v2, I want to scale past 2.70h toward 15h+ of balanced multi-speaker audio, train a native Yorùbá PLBERT so long compounds hold their contour tones, and add auto-diacritization plus English code-switching before G2P runs. If you are building TTS for a tonal or low-resource language and hit similar walls, drop a comment below.

## Key Takeaways

- •Over the past few weeks, I adapted the Kokoro / StyleTTS2 architecture to speak Yorùbá using a single Google Colab NVIDIA L4 GPU (22 GB VRAM) and 2.70 hours of cleaned audio.
- •This story was reported by **Dev.to** , covering developments in the**dev** space.
- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.

📖 Continue reading the full article:

[Read Full Article on Dev.to →](https://dev.to/akintunde102/teaching-kokoro-styletts2-to-speak-yoruba-without-resizing-its-embedding-table-4k3n)
