I set out to deploy a fine-tuned LLM fully offline, on a mid-range Android phone, answering medical questions in Yoruba, Hausa, Igbo, and Nigerian Pidgin. No internet connection required, because that's the reality for a lot of the people this was meant to help.
The model worked. Getting there broke three times, in three completely different ways, and left me with one problem I still haven't solved.
I fine-tuned unsloth/Llama-3.2-3B-Instruct
, Unsloth's 4-bit-optimized derivative of Meta's Llama 3.2 3B, in two stages: QLoRA supervised fine-tuning on a curated dataset of 3,917 medical question-answer pairs across the four languages, followed by direct preference optimization (DPO) to sharpen response quality. SFT converged to a loss of 1.099. DPO landed a reward margin of 18.40.
Then I merged to 16-bit and tried to convert to GGUF, the format llama.cpp
needs to run the model on-device. That's where things started breaking.
First conversion attempt. Model loads. First inference call. Instant crash:
terminating due to uncaught exception of type std::out_of_range:
unordered_map::at: key not found
Turns out the conversion had written tokenizer.ggml.model = "llama"
into the GGUF file. That field tells the runtime which tokenizer code path to use, and "llama"
routes to SentencePiece. Llama 3.2 doesn't use SentencePiece. It uses BPE. The runtime was trying to read BPE tokens through a SentencePiece parser, and predictably, it found nothing where it expected something.
Fix: manually set the field to "gpt2"
, which routes to the correct BPE path.
Fixed crash 1, tried again. New failure, before the model even finished converting:
TypeError: Llama 3 must be converted with BpeVocab
followed, after the code's fallback path kicked in, by:
ValueError: Cannot instantiate this tokenizer from a slow version
This one took longer to trace. Unsloth's saved tokenizer_config.json
includes "from_slow": true
and "backend": "tokenizers"
, fields that made transformers
try to instantiate the tokenizer as if it were a slow SentencePiece-style tokenizer, when it's actually a fast BPE tokenizer. The auto-detection logic just picked the wrong branch.
Fix: skip the auto-detection entirely. Route Llama 3's vocabulary directly through BpeVocab
, and load the tokenizer directly via PreTrainedTokenizerFast
instead of letting AutoTokenizer
guess.
Two down. Converted again. This time the model failed to load, not just infer:
error model vocabulary: cannot find tokenizer merges in model file
BPE needs a merges table, basically the rulebook for how to combine subword tokens back together. By default, the conversion path wasn't passing load_merges=True
, so the 280,147-entry merges table for this tokenizer never made it into the GGUF file at all.
Fix: pass load_merges=True
explicitly.
Three fixes in, the model finally converted cleanly and ran on-device.
Here's where I want to be straight with you, because it would be easy to wrap this up as "and then everything worked perfectly."
After all three fixes, the model runs. Most output is fine. But on a subset of Yoruba and Igbo prompts, it produces fluent-looking output that isn't real language. I asked it, in Yoruba, what to take for a headache:
ogun ori fifo?
And got back:
B��ọpọ nile, ounllo ni bi lojijoe obi mii. Odi kurosi ti wa (o)
oju, ekuruji nira naa rẹ lọwalo ninu ile ounllo.
That's not grammatical Yoruba. It reads like Yoruba if you don't look closely, invented words with plausible Yoruba shape, dropped into something that isn't a sentence.
I have four candidate explanations, and I want to be honest that I haven't isolated which one, or which combination, is actually responsible:
Sparse coverage. The base model almost certainly saw very little Yoruba and Igbo text during pretraining, which alone could produce exactly this kind of surface-plausible nonsense.
Not enough fine-tuning data. 3,917 QA pairs across four languages isn't a lot, especially split across languages with very different scripts and morphology.
Quantization damage. I quantized down to Q2_K, an aggressive compression level. It's plausible whatever capability the model had for these languages took a disproportionate hit.
My own patch, possibly. The fix for Crash 2 changed which code path writes the vocabulary, from LlamaHfVocab
to BpeVocab
. Those two paths can handle byte-level encoding and word-boundary markers differently. I can't rule out that this difference is quietly scrambling morpheme boundaries specifically for low-frequency languages.
The test that would actually tell me which one it is: run the same prompts against the unquantized model, and separately against an unpatched conversion, and compare. I didn't run either comparison. The unpatched conversion crashed unconditionally (that's Crash 1), so there was never a clean unpatched baseline to compare against on-device, and I didn't benchmark the full-precision model before quantizing.
So I don't know yet. I'm saying so, instead of picking the most plausible-sounding explanation and presenting it as settled.
The three crash fixes are concrete and verified, I confirmed each one by watching the corresponding failure disappear. I also built a small diagnostic tool: a round-trip checker that encodes prompts with the original tokenizer, compares against what the converted GGUF vocabulary actually contains at every token ID, and flags any mismatch. It won't tell you why output is wrong, but it will tell you fast whether your conversion silently corrupted something, before you find out the hard way in production.
Open source, MIT license: github.com/Afolabi-cyber/tinymed-tokenizer-fix
I've also filed the three crash bugs upstream against llama.cpp
, since two of them turned out to have partial precedent in older, unresolved issue reports going back to 2024. If you're converting anything derived from an Unsloth checkpoint, you may hit the same three walls I did.
It would be a cleaner story to only post the three fixes and call it a win. But the more important finding here might be the one I can't close: this class of failure, quiet, fluent-sounding, morphologically wrong output, doesn't show up in standard perplexity evaluation, because the language patterns it affects are exactly the ones underrepresented in most evaluation data for these languages to begin with. A metric that looks healthy can be hiding a real problem, and you won't know until you go looking specifically for it.
If you've hit something like this deploying quantized models for low-resource or morphologically-rich languages, I'd genuinely like to compare notes.