Teaching Kokoro / StyleTTS2 to Speak Yoruba Without Resizing Its Embedding Table A developer adapted the Kokoro/StyleTTS2 text-to-speech architecture to speak Yorùbá on a single Google Colab NVIDIA L4 GPU (22 GB VRAM) using 2.70 hours of cleaned audio, without resizing the model's 178-token embedding table. The developer packed each Yorùbá vowel-plus-tone and syllabic-nasal-plus-tone pair into the 63 unused slots among token IDs 1–177, using a new zero-dependency phonemizer, oro-g2p, and a greedy longest-first tokenizer, so pretrained tensor shapes still loaded. Training quality improved only after trimming silence (top_db=35), filtering clips to 1.2–5.8 seconds to match StyleTTS2's default max_len=120 crop, and separating speaker IDs that had been lumped under speaker_id = 0. Teaching Kokoro / StyleTTS2 to Speak Yoruba Without Resizing Its Embedding Table Most open-weight TTS models treat pitch like seasoning. Stress a syllable here, drop your voice at the end of a question, and English listeners still understand you. Yorùbá does not work like that. Pitch is part of the spelling. Take the letters o-k-o with underdots ọkọ : ọkọ mid-mid means husban d ọkọ̀ mid-low means car or vehicle ọ̀kọ̀ low-low means spear If an English-trained model guesses a rising or falling melody on ọkọ, it says the wrong word. Over the past few weeks, I adapted the Kokoro / StyleTTS2 architecture to speak Yorùbá using a single Google Colab NVIDIA L4 GPU 22 GB VRAM and 2.70 hours of cleaned audio. A lot of things broke along the way: dropped tones, raspy noise, speaker mix-ups, PyTorch 2.6 crashes, flat robotic speech, and even a 402 Payment Required surprise when I tried to deploy a free demo. This is the exact path I took, including the mistakes I made and where v1 still falls short. oro-g2p and hunting for unused token slots Before touching any neural network weights, I needed a phonemizer that actually understood Yorùbá orthography. English phonemizers fail immediately here because gb is a single sound /ɡb/ , p is pronounced /kp/ kp , and consonants like n or m nasalize vowels that follow them. Syllabic nasals m, n, ŋ also shift based on the next consonant, including stacked cases like nǹkan. I did not want people to install heavy system packages like espeak-ng, so I wrote oro-g2p, a zero-dependency Python library pip install oro-g2p : from oro g2p import g2p print g2p "Ẹ kú àárọ̀" 'ɛˉ kuˊ aˋaˊɾɔˋ' Getting the text into IPA with tone marks ˊ High, ˉ Mid, ˋ Low was the easy part. Feeding it into Kokoro was where I got stuck. My first attempt emitted the vowel and its tone as two separate tokens, like "a", "ˋ" . That sounded terrible in StyleTTS2. The duration predictor treated the tone mark like an extra letter, stretching trailing tones at the end of phrases or losing the link between a vowel and the pitch it was supposed to hold. I did not want to resize Kokoro's embedding table n token = 178 because doing that breaks direct weight loading from pretrained checkpoints. So I printed out BASE KOKORO VOCAB to see what was actually inside those 178 IDs 0 to 177 . Slot 0 is reserved PAD/BOS/EOS , and 114 slots are already taken by Kokoro's base tokens. That left 63 completely empty slots sitting unused inside 1..177. So I stopped sending vowels and tones separately. I packed every Yorùbá vowel + tone pair, syllabic nasal + tone pair, and digraph directly into those 63 spare slots as single atomic IDs "kp": 122, "ɡb": 124, "dʒ": 82, "aˊ": 7, "aˋ": 26, "ũˋ": 105, etc. , matched with a greedy longest-first tokenizer. Because n token stayed 178, all pretrained tensor shapes lined up without touching the model architecture. Early training runs sounded awful, and I spent too long blaming hyperparameters before inspecting my dataset. The biggest culprit was StyleTTS2's default dataloader crop. By default, it crops long audio down to max len=120 around 1.44 seconds while still feeding the full transcript. A lot of my source clips were 6 to 11 seconds long. I was basically asking the model to listen to 1.4 seconds of audio while guessing how it aligned to a 10-second sentence. Once I trimmed leading and trailing silence top db=35 and filtered my training set down to clips between 1.2s and 5.8s, alignment snapped into place. The second mess came from speaker IDs. StyleTTS2 grabs another random clip from the same speaker id during training to estimate pitch baseline and voice style. Early on, I had multiple voices lumped under speaker id = 0, which meant the pitch predictor was getting conflicting references on every batch. Separating speaker IDs strictly cleared up a lot of random pitch jumps. I also learned a strange quirk about iSTFTNet the hard way at inference time: if you try to strip predicted silence frames before passing features into the decoder, InstanceNorm1d calibration gets thrown off and the audio comes out covered in raspy static. Waveform silence trimming only works after the decoder produces audio. After cleaning, my final Gold training split came out to 2.70 hours 2,168 clips drawn from AfriSpeech/open-bible-speech-african CC BY-SA 4.0 and google/WaxalNLP CC-BY / CC-BY-SA-4.0 . Trying to run Stage 2 training on Google Colab with modern PyTorch 2.6+ threw a handful of upstream crashes unrelated to Yorùbá: models.py failed to load checkpoints without weights only=False, choked on module. DataParallel key prefixes, and threw spectral norm errors if .eval ran on style encoder or predictor encoder before a 1-step .train warmup. Modules/slmadv.py had 17 .squeeze calls wav.detach .squeeze , y pred.squeeze , y rec gt.detach .squeeze , y rec gt pred.detach .squeeze , real GP.detach .squeeze , etc. that dropped the batch dimension completely if batch size=1, crashing tensor shapes. I replaced those with a wav2d x helper x.reshape x.shape 0 , -1 and added 1D/3D shape guards in Modules/istftnet.py. train second.py called pdb.set trace on NaN/Inf batches. If a batch blew up overnight on Colab, the script did not skip it or crash loudly, it just sat waiting for keyboard input while burning GPU units. I disabled set detect anomaly True , wrapped y rec gt pred in torch.no grad , added clip grad norm ..., 5.0 on bert, predictor, and diffusion, and skipped NaN batches safely. I pushed these fixes upstream in semidark/StyleTTS2 4 +47 / -19 across 4 files and left a breakdown in semidark/kikiri-tts 46. My Stage 2 runs on the L4 batch size=4 proved just how much diffusion and style loss weights matter for tonal speech: Round 1: I accidentally left lambda diff=0.0 and lambda sty=0.0. My F0 loss stalled at 2.245 and the voice sounded completely flat. Round 2: Turned both back on lambda diff=1.0, lambda sty=1.0, lambda F0=3.0, bert lr=5e-5 . Validation loss dropped to 0.313 and F0 loss fell to 1.921. Round 3: Ran 15 epochs with lambda F0=4.5, lambda sty=1.5, lambda diff=1.0, and bert lr=4e-5. F0 loss reached a low of 1.758 along the way, finishing Epoch 15 epoch 2nd 00014.pth at Validation loss: 0.305, Dur loss: 0.974, F0 loss: 1.823. epoch 2nd 00014.pth was 2,091.5 MB on disk because it carried training discriminators and optimizer state. Filtering down strictly to INFERENCE MODULES compressed the release checkpoint kokoro yoruba gold.pth to 526.0 MB. During listening tests, I still noticed two annoying glitches: Vocal shaking on multi-tone vowels like láàárín. Locking diffusion parameters to alpha=0.0, beta=0.55, embedding scale=1.15, diffusion steps=15, and speed=1.0 stopped the tremor. Rushing on long text. Passing 400+ character paragraphs all at once made the model speed up unnaturally. Splitting text by punctuation 0.18s pause on commas, 0.35s pause on periods, plus appending a trailing . if missing fixed the pacing. In Python, running inference looks like this: from tts yoruba import YorubaTTS tts = YorubaTTS styletts2 dir="./StyleTTS2" tts.speak "Ẹ kú àárọ̀ o Báwo ni àwọn ará ilé?", voice="spk8", out path="greeting.wav", When I tried to deploy a live Gradio Space on Hugging Face's free cpu-basic tier, it threw 402 Payment Required. Since I did not want to pay for idle GPU hosting or keep people from testing it, I split the public demo into two pieces: a 24/7 static HTML audio showcase on Hugging Face Spaces so anyone can hear real v1 outputs right away, paired with a 1-click Google Colab Gradio notebook which I redesigned after checking live browser screenshots so all 9 preset phrases fit without scrolling or clipped sliders . If you try this model today, keep its v1 limits in mind: Small, lopsided training data 2.70h total : Speaker 8 BibleTTS makes up ~92% of the audio and matches the reference voice closely ~98% in my A/B listening checks . Speaker 2 Waxal is only ~1% of the data, so that voice is noticeably weaker with slight static. English-pretrained PLBERT: Because the text encoder never saw native Yorùbá during pretraining, rare 4-to-6 syllable compound words like Ṣàwòròidẹ or ògbóǹtarìgì can lose tone accuracy. Full diacritics required: There is no auto-diacritizer in v1 yet. Plain ASCII Yorùbá without tone marks defaults to mid-tones. Pure Yorùbá only: Raw English words get read with Yorùbá spelling rules for example, "phone" maps p to /kp/ as kphoˉneˉ . Rather than leaving everything jammed inside one repo, I broke the project into three standalone parts: oro-g2p MIT — Just the pure-Python Yorùbá G2P and Kokoro atomic vocab pip install oro-g2p . kokoro-tonal-kit MIT — The starter kit patch styletts2.py, vocab template.py, export release.py, Stage 2 config notes in case you want to adapt Kokoro / StyleTTS2 for another tonal language like Igbo, Twi, Hausa, or Vietnamese. kokoro-yoruba — The Yorùbá inference wrapper tts yoruba.py and 1-click Colab Gradio demo wired up to the CC BY-SA 4.0 weights. Audio Showcase HF Space : huggingface.co/spaces/Akintunde102/kokoro-yoruba-demo Live Gradio Demo Colab : Open in Google Colab Model Weights CC BY-SA 4.0 : huggingface.co/Akintunde102/kokoro-yoruba-v1 Yorùbá TTS Repo: github.com/Akintunde102/kokoro-yoruba Tonal Starter Kit: github.com/Akintunde102/kokoro-tonal-kit Yorùbá G2P: github.com/Akintunde102/oro-g2p | PyPI Upstream StyleTTS2 PR: semidark/StyleTTS2 4 For v2, I want to scale past 2.70h toward 15h+ of balanced multi-speaker audio, train a native Yorùbá PLBERT so long compounds hold their contour tones, and add auto-diacritization plus English code-switching before G2P runs. If you are building TTS for a tonal or low-resource language and hit similar walls, drop a comment below. Key Takeaways - •Over the past few weeks, I adapted the Kokoro / StyleTTS2 architecture to speak Yorùbá using a single Google Colab NVIDIA L4 GPU 22 GB VRAM and 2.70 hours of cleaned audio. - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/akintunde102/teaching-kokoro-styletts2-to-speak-yoruba-without-resizing-its-embedding-table-4k3n