Fine-tuning was the easy part: what shipping ASR for a low-resource language taught me

A developer building speech recognition for Crimean Tatar found that a large Whisper model fine-tuned on the language scored 34.6% word error rate on clean audio, improving to 20.1% WER after roughly 1.5 hours of fine-tuning on a single consumer GPU and to 17.0% WER through decode-time tuning alone. The engineer also discovered that four audiobooks in the corpus were stored twice under unrelated filenames, with 96.9% of clips in the intended evaluation book having a near-duplicate in training, invalidating prior measurements. The project recommends splitting data by speaker, session, or document rather than randomly, and verifying held-out sets with word n-gram overlap checks.

The fine-tune took an hour and a half on a single consumer GPU. Everything that made that hour and a half mean anything took months. This is the second post in a series about building speech tech for Crimean Tatar, a language with no datasets to speak of. The first one https://ai.ana-yurt.dev/blog/2026-09-kak-rodilas-ideya/ covered synthesis and how the project started; this one is the recogniser — much less a story about training than about measurement. The headline, so you can decide whether to keep reading: | | WER | CER | |---|---|---| | the model I started from | 34.6% | 11.9% | | + fine-tuning ≈1.5 h, one GPU | 20.1% | 9.4% | | + tuning how it decodes | 17.0% | 7.0% | Left to right: starting model, after fine-tuning, after decode-time tuning. The third bar cost zero training. Now the parts worth your time. I already had a "working" Crimean Tatar recogniser: a large Whisper, fine-tuned on our language, which I'd been using on clean studio audio where it looked fine. I had read its transcripts and nodded. I had never scored it. So the first thing I did here was not improve anything. I measured. 34.6% WER — every third word wrong. That's not "rough": it can't back archive search, can't produce training pairs for synthesis, and an assistant built on it mishears you constantly. I only knew that because I finally scored the thing I'd been eyeballing for a year. One footnote on that baseline: 34.6% slightly understates it, because the model writes 1954 where the book spells out Biñ doquz yüz elli dört and the scorer counts four substitutions. Small, but if I don't say it, the number is dressed up. Standard practice is a random 90/10 split. Here that would have been meaningless. Adjacent clips come from the same recording, same reader, same microphone, same afternoon — and the segmenter routinely cuts one sentence across two neighbouring clips, so a random split drops half a sentence in train and the other half in eval. What you measure then isn't language ability. It's whether the model memorised a voice and a room. So I held out two entire books, two readers the model had never heard : 893 clips, 1 h 52 min. One move removes the voice, the mic, the recording era and the vocabulary of a particular text at once. If your data has structure — speaker, session, document, device — split on it. Random splitting is only safe when clips are genuinely independent, and in low-resource audio they essentially never are. That feels like enough caution. It wasn't. Before launching anything I checked the obvious thing: is any of my held-out eval material also sitting in training? By filename: zero overlap. Not one collision. The two name sets looked like they came from different universes. Then I compared the text instead: for every eval clip, does any training clip contain the same run of six consecutive words? Word 6-grams — cheap to build, cheap to hash, long enough that natural coincidence is rare at this corpus size. Four audiobooks were in my material twice. Same recordings, segmented by two different tools at two different times, under completely unrelated names. | book | share of its clips with a twin in training | |---|---| | book I'd picked for eval | 96.9% 651 / 672 | | second eval book | 58.8% 130 / 221 | | third book | 82.1% 769 / 937 | | fourth book | 69.8% 549 / 787 | Four books stored twice. Zero overlap by filename; the collision only showed up in the text. Top row is the book I was about to grade the model on. I wasn't panicked. I just understood two things at once: every measurement I'd ever taken on this data was void, and thank God it surfaced before training rather than after. The counterfactual is the point. If I hadn't checked, nothing would have gone wrong. No crash, no warning, no weird loss curve. The model trains on recordings it will later be examined on — different cuts, same sentences, same voices — posts a great number, and I publish it. The number means nothing: the exam questions were handed out in advance. That's the nasty property of this bug class. It has no symptoms. Everything works, the result is better than expected — which is exactly what should make you suspicious. And it generalises: in a low-resource language, re-cut copies of the same recordings are the norm, not the exception. There is little material, so you keep returning to it with new tooling, and every pass produces fresh filenames wrapping old content. Deduplicate by content. Filenames are not identity. After cleaning I re-verified rather than trusting myself: zero eval clips in training, zero 6-gram matches. Measured and written down, not "I'm pretty sure I got them all". Now the part everyone thinks is the project, which was the fastest and dullest bit of it. I didn't retrain the model. I froze it and trained a small adapter alongside — 2% of parameters , 31M trainable against ~1.5B frozen. 15.5 hours of speech survived the cleanup, 3 epochs, 705 steps, 1 h 30 min on one home GPU , 10.4 GB peak VRAM, and a finished adapter of 126 MB . Result: 20.1% WER, 9.4% CER — a 42% relative cut in word errors. I felt nothing at the time; a number moved as designed. The pride arrived later, when I ran the model over material it had no business handling — films, podcasts, noisy field recordings — and the transcripts came back near-perfect. That's when it stopped being a table row and became working ears. Two things from the run are more useful than the headline number. Checkpoint 100 was worse than the base model. I saved every 100 steps and scored all of them. The first checkpoint came out noticeably worse than what I started from: the adapter has half-learned the new behaviour, it's already interfering with the old one, and it isn't giving anything back yet. If I'd glanced at it and concluded the approach was a dud, I'd have killed a run that ended up cutting error by 42%. A dip at the first checkpoint is what partial adaptation looks like, not what failure looks like. All the gains landed in epoch 1 of 3. Past roughly step 200 the numbers stopped moving; epochs two and three contributed nothing measurable. So "1.5 hours" is itself padded — half an hour would have done. Checkpoint often and score often; runs converge long before the schedule says so. One detail that reassured me more than the average: both eval books improved, and the smaller, later, different-reader one improved more. That's the signal you want — the model didn't just learn one narrator's delivery. Then error dropped again with no training at all. A trained model has knowledge, but it also has a decoding policy . The default is greedy — take the locally most probable token, never reconsider. Fast, occasionally catastrophic: take one wrong turn and it can't back out, so it goes in circles. That's exactly what the worst eval clips were. Not mishearing — looping: Qıznıñ qıznıñ qıznıñ qıznıñ … – dedi Akimoviç, – dedi Akimoviç, – dedi Akimoviç, … The fix is beam search: hold several candidate sequences and score whole hypotheses rather than committing token by token. Costs decode time, costs zero training. I swept 24 configurations on a separate selection set, not on the exam more below , then ran the winner on the test set once. 17.0% WER, 7.0% CER. Not one weight changed. Decoding got 3.1× slower. Here it would have been easy to write "beam search: −3 WER points" and move on. True, and half the truth. So I broke the win down. 409 word errors were recovered in total. 160 of them came from 3 clips out of 893. 0.34% of the material delivered 39% of the gain. Those are the loopers: on 41 reference words the greedy model emitted 175 words of repeated garbage. The remaining 890 clips improved by a modest, even 10%. Both halves are real, and neither alone justifies a 3.1× slowdown. But they mean different things. The broad 10% is "the model got more careful". The three clips are "it stopped occasionally losing its mind". For archive transcription the second matters far more, because a clip containing 500 bytes of one repeated token isn't merely bad — it poisons anything you train on it later. After the change: zero looping clips, 0 of 893. Decompose your aggregate wins. If most of your improvement lives in a handful of examples, you didn't improve accuracy; you fixed a failure mode. Those call for different follow-up work, and averaging hides the difference. There's a more direct tool against repetition: forbid the model from emitting the same n-gram twice. On paper, exactly the right hammer. I tested it. It was net harmful: across 255 selection clips it broke 17 and fixed 1 , and 12 of the 17 it broke had been essentially perfect. The transcripts explain themselves. The constraint doesn't stop the model repeating — it forces it to repeat incorrectly : book – Qartbaba, qartbaba, – tez-tez çapıp kelgen kiçkene Aziz unrestricted– Qartbaba Qartbaba – tez-tez çapıp kelgen kiçkene Aziz restricted – Qartbaba Qartbabaa – tez-tez çapıp kelgen kiçkene Aziz harder – Qartbaba Qartbava – tez-tez çapıp kelgen kiçkene Aziz A child runs up calling "Qartbaba, qartbaba " — grandpa, grandpa. The model heard it perfectly. The ban wouldn't let it write down what it heard, so it corrupted the second word to avoid matching the first. Elsewhere bağışla… bağışla… "forgive me… forgive me…" came back the second time as the non-word Bağuşla ; in a third case the model routed around the constraint by inventing a suffix, parçalarnıñ parça instead of parçalar . Doubled address and doubled pleading are ordinary Crimean Tatar. People talk that way. This is my favourite finding of the stage, and it isn't really about decoding. You cannot fix a language with prohibitions. Any rule of the form "this never occurs" eventually collides with how people actually speak, and it breaks precisely that spot while leaving everything else intact — so you won't notice. A decoding default borrowed from an English-language tutorial encodes assumptions about English. Test it against your own language's rhetoric first. I've mentioned a "selection set" twice. Every number above rests on it. There are three sets, not two. Why so strict? Because if you run the test set repeatedly and take the best result each time, you're no longer selecting the best model — you're selecting the luckiest one. The number comes out pretty and never reproduces. The dev set got the same cleanliness audit: zero shared clips, zero 6-gram matches. After the duplicate story I don't file that under paranoia. The dev set was much easier than the test set. Same model: 17% WER on dev, 34.6% on test. Both honestly held out, neither seen in training. But the dev material comes from the same family of recordings as training, while the test books are two entirely foreign readers. So dev is good for "config A beats config B" and useless for "the model errs at X%". Report the dev number as your result and you're not lying exactly — you're reporting the easier exam. Keep both in their lanes: dev ranks, test reports. One consequence of that discipline. Whisper ships a temperature-fallback mechanism against repetition: it detects circular output and retries. Precisely the tool this problem wants, and nearly free at decode time. But my dev set contained zero looping clips. Nothing for it to act on — it looked like a config that does nothing at all. By my own selection rule I couldn't pick it, and I refused to sneak it onto the test set afterwards to see whether it would have helped. So I have no number for it, and I'm not inventing one. If you couldn't validate a tool on the set you're allowed to validate on, write down that it's unvalidated and keep it out of the results table. Everything above was measured on one kind of audio: one person, one mic, quiet room, reading a book. Honest for its own world, and it says little about the world a voice assistant lives in. So I took the same 893 clips and corrupted them 17 ways — noise, music, competing speech, reverb, telephone-band filtering, time-stretching. The reference text is known, so error is measurable in each. That points somewhere specific: the highest-leverage investment in robustness is not more data. It's source separation and microphone placement. More training hours will not fix a model whose input holds two people at equal volume. One last check that made me happy: on an open Crimean Tatar audiobook set the model had never trained on, clean chapters landed at essentially the same number as my own exam. It didn't memorise my readers. It hears the language. Training was 90 minutes. The months went into auditing the eval set by content, splitting by recording, checkpointing and scoring instead of trusting the schedule, decomposing wins instead of quoting averages, refusing to let an English-shaped default constrain our grammar, and keeping a test set clean enough that its number means something. Fine-tuning is the part with a progress bar. It is not the part where the work is. Second post in a series. Coming up: a synthesised voice reading an entire book — sixteen chapters, nearly two hours of audio — and what had to be verified chapter by chapter. Then the one where I rented somebody else's GPUs and set money on fire, computing almost nothing. Audio samples and project page: https://ai.ana-yurt.dev/ https://ai.ana-yurt.dev/en/blog/2026-09-mashina-nauchilas-slyshat/ . Two ways to help. If you speak Crimean Tatar, the highest-value contribution is fifteen minutes with headphones — listen to a clip, say whether the machine wrote it down correctly, fix a transcript. No programming needed, and ears like that caught things here that no automated check did; write to support@ana-yurt.com mailto:support@ana-yurt.com . If you'd rather fuel the work directly, there's ko-fi.com/anayurt https://ko-fi.com/anayurt .