{"slug": "fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me", "title": "Fine-tuning was the easy part: what shipping ASR for a low-resource language taught me", "summary": "A developer building speech recognition for Crimean Tatar found that a large Whisper model fine-tuned on the language scored 34.6% word error rate on clean audio, improving to 20.1% WER after roughly 1.5 hours of fine-tuning on a single consumer GPU and to 17.0% WER through decode-time tuning alone. The engineer also discovered that four audiobooks in the corpus were stored twice under unrelated filenames, with 96.9% of clips in the intended evaluation book having a near-duplicate in training, invalidating prior measurements. The project recommends splitting data by speaker, session, or document rather than randomly, and verifying held-out sets with word n-gram overlap checks.", "body_md": "The fine-tune took an hour and a half on a single consumer GPU. Everything that made that hour and a half mean anything took months.\n\nThis is the second post in a series about building speech tech for Crimean Tatar, a language with no datasets to speak of. [The first one](https://ai.ana-yurt.dev/blog/2026-09-kak-rodilas-ideya/) covered synthesis and how the project started; this one is the recogniser — much less a story about training than about measurement.\n\nThe headline, so you can decide whether to keep reading:\n\n|  | WER | CER | \n|---|---|---|\n| the model I started from | 34.6% | 11.9% | \n| + fine-tuning (≈1.5 h, one GPU) | 20.1% | 9.4% | \n| + tuning how it decodes | **17.0%** | **7.0%** | \n\n*Left to right: starting model, after fine-tuning, after decode-time tuning. The third bar cost zero training.*\n\nNow the parts worth your time.\n\nI already had a \"working\" Crimean Tatar recogniser: a large Whisper, fine-tuned on our language, which I'd been using on clean studio audio where it looked fine. I had read its transcripts and nodded. I had never scored it.\n\nSo the first thing I did here was not improve anything. I measured. 34.6% WER — every third word wrong. That's not \"rough\": it can't back archive search, can't produce training pairs for synthesis, and an assistant built on it mishears you constantly. I only knew that because I finally scored the thing I'd been eyeballing for a year.\n\nOne footnote on that baseline: 34.6% slightly *understates* it, because the model writes `1954` where the book spells out `Biñ doquz yüz elli dört` and the scorer counts four substitutions. Small, but if I don't say it, the number is dressed up.\n\nStandard practice is a random 90/10 split. Here that would have been meaningless. Adjacent clips come from the same recording, same reader, same microphone, same afternoon — and the segmenter routinely cuts one sentence across two neighbouring clips, so a random split drops half a sentence in train and the other half in eval. What you measure then isn't language ability. It's whether the model memorised a voice and a room.\n\nSo I held out **two entire books, two readers the model had never heard**: 893 clips, 1 h 52 min. One move removes the voice, the mic, the recording era and the vocabulary of a particular text at once. If your data has structure — speaker, session, document, device — split on it. Random splitting is only safe when clips are genuinely independent, and in low-resource audio they essentially never are.\n\nThat feels like enough caution. It wasn't.\n\nBefore launching anything I checked the obvious thing: is any of my held-out eval material also sitting in training?\n\nBy filename: zero overlap. Not one collision. The two name sets looked like they came from different universes.\n\nThen I compared the *text* instead: for every eval clip, does any training clip contain the same run of six consecutive words? Word 6-grams — cheap to build, cheap to hash, long enough that natural coincidence is rare at this corpus size.\n\nFour audiobooks were in my material twice. Same recordings, segmented by two different tools at two different times, under completely unrelated names.\n\n| book | share of its clips with a twin in training | \n|---|---|\n| **book I'd picked for eval** | **96.9%** (651 / 672) | \n| second eval book | 58.8% (130 / 221) | \n| third book | 82.1% (769 / 937) | \n| fourth book | 69.8% (549 / 787) | \n\n*Four books stored twice. Zero overlap by filename; the collision only showed up in the text.*\n\nTop row is the book I was about to grade the model on.\n\nI wasn't panicked. I just understood two things at once: every measurement I'd ever taken on this data was void, and thank God it surfaced before training rather than after.\n\nThe counterfactual is the point. If I hadn't checked, nothing would have gone wrong. No crash, no warning, no weird loss curve. The model trains on recordings it will later be examined on — different cuts, same sentences, same voices — posts a great number, and I publish it. The number means nothing: the exam questions were handed out in advance. That's the nasty property of this bug class. **It has no symptoms.** Everything works, the result is better than expected — which is exactly what should make you suspicious.\n\nAnd it generalises: **in a low-resource language, re-cut copies of the same recordings are the norm, not the exception.** There is little material, so you keep returning to it with new tooling, and every pass produces fresh filenames wrapping old content. Deduplicate by content. Filenames are not identity.\n\nAfter cleaning I re-verified rather than trusting myself: zero eval clips in training, zero 6-gram matches. Measured and written down, not \"I'm pretty sure I got them all\".\n\nNow the part everyone thinks is the project, which was the fastest and dullest bit of it.\n\nI didn't retrain the model. I froze it and trained a small adapter alongside — **2% of parameters**, 31M trainable against ~1.5B frozen. 15.5 hours of speech survived the cleanup, 3 epochs, 705 steps, **1 h 30 min on one home GPU**, 10.4 GB peak VRAM, and a finished adapter of **126 MB**. Result: 20.1% WER, 9.4% CER — a 42% relative cut in word errors.\n\nI felt nothing at the time; a number moved as designed. The pride arrived later, when I ran the model over material it had no business handling — films, podcasts, noisy field recordings — and the transcripts came back near-perfect. That's when it stopped being a table row and became working ears.\n\nTwo things from the run are more useful than the headline number.\n\n**Checkpoint 100 was worse than the base model.** I saved every 100 steps and scored all of them. The first checkpoint came out noticeably *worse* than what I started from: the adapter has half-learned the new behaviour, it's already interfering with the old one, and it isn't giving anything back yet. If I'd glanced at it and concluded the approach was a dud, I'd have killed a run that ended up cutting error by 42%. A dip at the first checkpoint is what partial adaptation looks like, not what failure looks like.\n\n**All the gains landed in epoch 1 of 3.** Past roughly step 200 the numbers stopped moving; epochs two and three contributed nothing measurable. So \"1.5 hours\" is itself padded — half an hour would have done. Checkpoint often and score often; runs converge long before the schedule says so.\n\nOne detail that reassured me more than the average: both eval books improved, and the *smaller, later, different-reader* one improved more. That's the signal you want — the model didn't just learn one narrator's delivery.\n\nThen error dropped again with no training at all.\n\nA trained model has knowledge, but it also has a **decoding policy**. The default is greedy — take the locally most probable token, never reconsider. Fast, occasionally catastrophic: take one wrong turn and it can't back out, so it goes in circles. That's exactly what the worst eval clips were. Not mishearing — looping:\n\n```\nQıznıñ qıznıñ qıznıñ qıznıñ …\n– dedi Akimoviç, – dedi Akimoviç, – dedi Akimoviç, …\n```\n\nThe fix is beam search: hold several candidate sequences and score whole hypotheses rather than committing token by token. Costs decode time, costs zero training. I swept 24 configurations on a separate selection set, not on the exam (more below), then ran the winner on the test set once. **17.0% WER, 7.0% CER.** Not one weight changed. Decoding got 3.1× slower.\n\nHere it would have been easy to write \"beam search: −3 WER points\" and move on. True, and half the truth. So I broke the win down.\n\n409 word errors were recovered in total. **160 of them came from 3 clips out of 893.** 0.34% of the material delivered 39% of the gain. Those are the loopers: on 41 reference words the greedy model emitted 175 words of repeated garbage. The remaining 890 clips improved by a modest, even 10%.\n\nBoth halves are real, and neither alone justifies a 3.1× slowdown. But they mean different things. The broad 10% is \"the model got more careful\". The three clips are \"it stopped occasionally losing its mind\". For archive transcription the second matters far more, because a clip containing 500 bytes of one repeated token isn't merely bad — it poisons anything you train on it later. After the change: zero looping clips, 0 of 893.\n\n**Decompose your aggregate wins.** If most of your improvement lives in a handful of examples, you didn't improve accuracy; you fixed a failure mode. Those call for different follow-up work, and averaging hides the difference.\n\nThere's a more direct tool against repetition: forbid the model from emitting the same n-gram twice. On paper, exactly the right hammer.\n\nI tested it. It was net harmful: across 255 selection clips it **broke 17 and fixed 1**, and 12 of the 17 it broke had been essentially perfect. The transcripts explain themselves. The constraint doesn't stop the model repeating — it forces it to repeat *incorrectly*:\n\n```\nbook        – Qartbaba, qartbaba, – tez-tez çapıp kelgen kiçkene Aziz\nunrestricted– Qartbaba! Qartbaba! – tez-tez çapıp kelgen kiçkene Aziz\nrestricted  – Qartbaba! Qartbabaa! – tez-tez çapıp kelgen kiçkene Aziz\nharder      – Qartbaba! Qartbava! – tez-tez çapıp kelgen kiçkene Aziz\n```\n\nA child runs up calling \"Qartbaba, qartbaba!\" — grandpa, grandpa. The model heard it perfectly. The ban wouldn't let it write down what it heard, so it corrupted the second word to avoid matching the first. Elsewhere `bağışla… bağışla…` (\"forgive me… forgive me…\") came back the second time as the non-word `Bağuşla`; in a third case the model routed around the constraint by inventing a suffix, `parçalarnıñ parça` instead of `parçalar`. Doubled address and doubled pleading are ordinary Crimean Tatar. People talk that way.\n\nThis is my favourite finding of the stage, and it isn't really about decoding. **You cannot fix a language with prohibitions.** Any rule of the form \"this never occurs\" eventually collides with how people actually speak, and it breaks precisely that spot while leaving everything else intact — so you won't notice. A decoding default borrowed from an English-language tutorial encodes assumptions about English. Test it against your own language's rhetoric first.\n\nI've mentioned a \"selection set\" twice. Every number above rests on it. There are three sets, not two.\n\nWhy so strict? Because if you run the test set repeatedly and take the best result each time, you're no longer selecting the best model — you're selecting the luckiest one. The number comes out pretty and never reproduces.\n\nThe dev set got the same cleanliness audit: zero shared clips, zero 6-gram matches. After the duplicate story I don't file that under paranoia.\n\nThe dev set was **much easier** than the test set. Same model: **17% WER on dev, 34.6% on test.** Both honestly held out, neither seen in training. But the dev material comes from the same family of recordings as training, while the test books are two entirely foreign readers.\n\nSo dev is good for \"config A beats config B\" and useless for \"the model errs at X%\". Report the dev number as your result and you're not lying exactly — you're reporting the easier exam. Keep both in their lanes: dev ranks, test reports.\n\nOne consequence of that discipline. Whisper ships a temperature-fallback mechanism against repetition: it detects circular output and retries. Precisely the tool this problem wants, and nearly free at decode time.\n\nBut my dev set contained **zero** looping clips. Nothing for it to act on — it looked like a config that does nothing at all. By my own selection rule I couldn't pick it, and I refused to sneak it onto the test set afterwards to see whether it would have helped. So I have no number for it, and I'm not inventing one. If you couldn't validate a tool on the set you're allowed to validate on, write down that it's unvalidated and keep it out of the results table.\n\nEverything above was measured on one kind of audio: one person, one mic, quiet room, reading a book. Honest for its own world, and it says little about the world a voice assistant lives in. So I took the same 893 clips and corrupted them 17 ways — noise, music, competing speech, reverb, telephone-band filtering, time-stretching. The reference text is known, so error is measurable in each.\n\nThat points somewhere specific: the highest-leverage investment in robustness is not more data. It's **source separation and microphone placement.** More training hours will not fix a model whose input holds two people at equal volume.\n\nOne last check that made me happy: on an open Crimean Tatar audiobook set the model had never trained on, clean chapters landed at essentially the same number as my own exam. It didn't memorise my readers. It hears the language.\n\nTraining was 90 minutes. The months went into auditing the eval set by content, splitting by recording, checkpointing and scoring instead of trusting the schedule, decomposing wins instead of quoting averages, refusing to let an English-shaped default constrain our grammar, and keeping a test set clean enough that its number means something.\n\nFine-tuning is the part with a progress bar. It is not the part where the work is.\n\nSecond post in a series. Coming up: a synthesised voice reading an entire book — sixteen chapters, nearly two hours of audio — and what had to be verified chapter by chapter. Then the one where I rented somebody else's GPUs and set money on fire, computing almost nothing.\n\nAudio samples and project page: [https://ai.ana-yurt.dev/](https://ai.ana-yurt.dev/en/blog/2026-09-mashina-nauchilas-slyshat/).\n\nTwo ways to help. If you speak Crimean Tatar, the highest-value contribution is fifteen minutes with headphones — listen to a clip, say whether the machine wrote it down correctly, fix a transcript. No programming needed, and ears like that caught things here that no automated check did; write to [support@ana-yurt.com](mailto:support@ana-yurt.com). If you'd rather fuel the work directly, there's [ko-fi.com/anayurt](https://ko-fi.com/anayurt).", "url": "https://wpnews.pro/news/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me", "canonical_source": "https://dev.to/servin_osmanov/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-taught-me-4331", "published_at": "2026-09-23 19:49:45+00:00", "updated_at": "2026-09-23 20:29:08.078735+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-research", "ai-tools"], "entities": ["Crimean Tatar", "Whisper"], "alternates": {"html": "https://wpnews.pro/news/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me", "markdown": "https://wpnews.pro/news/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me.md", "text": "https://wpnews.pro/news/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me.txt", "jsonld": "https://wpnews.pro/news/fine-tuning-was-the-easy-part-what-shipping-asr-for-a-low-resource-language-me.jsonld"}}