By Arun Prabhakaran · @LatentAILabs
TL;DR
Parts of this article were drafted with AI assistants; I checked every number against the model card and the evaluation logs.
Here is a made-up comment of the kind you see under any Tamil film trailer:
“padam semma mass bro, second half konjam slow but climax vera level”
Half of it is Tamil, half English, all of it in Roman letters. Swap the film for cricket and you get the same thing in Hinglish, Manglish (Malayalam) or Kanglish (Kannada). Many people in India type like this on their phones.
Most language-ID tools give one label per message. That is fine for routing a whole message, but not when you need to know which words are Tamil:
There are good word-level models for Hinglish. For Tamil, Malayalam and Kannada mixed with English, I found only two downloadable checkpoints, both without a licence: the 2022 shared-task winner for Kannada (a DistilBERT that judges one word at a time) and a Malayalam model with no model card. Newer MIT models such as MixLID cover these languages only in their native scripts. Nothing I found handles romanised Tamil, so I built a model. Bhasha simply means “language”.
It works well for Kannada and Malayalam. Tamil taught me something I did not expect: on human-labelled comments, Bhasha calls about a third of Tamil words Telugu, including every Tamil word in the comment above. Most of this post is about how I found that, and why my own test set missed it.
It is the fourth model in a small open set for Indian languages, after Yama (zero-shot classification), Chitragupta (PII detection) and Nandi (complaint routing).
Besides a label per word, you get a message summary: the main language, the script (native, roman or mixed), whether the message is mixed, and the share of each language. Romanised Hindi and Urdu are often the same words, so when the model cannot tell them apart it returns hi/ur instead of guessing.
The langs argument matters, and the Tamil section explains why. If you know which language pair your text is in, pass it.
No openly licensed, word-labelled code-mixed corpus covers these languages, so every training row was built by code:
The result is 1.65 million rows across 11 languages: romanised, mixed, native, native mixed, mixed script and plain English. No training row overlapped an evaluation set (two native-script test sentences that matched training text were dropped from that test), and a later check found no overlap with the human-labelled sets either. Then I fine-tuned XLM-RoBERTa-base for one epoch, which took about an hour on one L4 GPU.
There are no hand labels and no LLM-written text in the training data. That keeps the licence clean, but it has a cost: all of the Indian-language text started as news and web writing. That cost shows up in the Tamil section.
Before training, I wrote down the evaluation plan and a bar for each language pair: beat the better of two GlotLID baselines, with a margin of 0.10 over GlotLID applied word by word. Then I built a real test set: 425 YouTube comments from DravidianCodeMix (CC-BY): 150 Tamil, 150 Malayalam and 125 Kannada, each mixed with English. Each word was labelled by two Claude passes and by Gemini, and a 2-of-3 vote decided. The three labellers agreed on 95–99% of words.
These labels are LLM-reviewed, not human-verified. I scored the frozen model once.
The released Hinglish and Indic word-level models were built for Hindi, Bengali and Gujarati. They have no Tamil, Malayalam or Kannada label, so they fail here.
(Macro-F1 here averages two F1 scores, one for the Indian language and one for English. 1.0 is perfect, and a model that never finds the Indian language tops out near 0.5.)
At this point I would have shipped with the Tamil miss stated. Then I found human-labelled data.
I found three sets of Dravidian code-mixed text with word-level language labels written by people:
I used them for evaluation only, with the model frozen, and scored each set once. I report totals only and do not redistribute them (the CoLI files state no licence). The CoLI files have one word per row and no sentence breaks, so I scored them in 12-word windows.
Malayalam and Kannada hold up on human labels. Tamil does not.
I also ran the released models I had found on the same sets:
So Bhasha is not the best Kannada model on every test. But it is the only one I found that scores 0.84 or higher on every Kannada and Malayalam set, and the only licensed word-level model I found for romanised Tamil, Malayalam or Kannada.
Malayalam and Kannada matched my LLM-labelled test, which is reassuring. Tamil fell from 0.87 to 0.72. A drop that large is a finding, not noise.
One caveat: of the three CoLI files, the Tamil one kept its original word order least well, and context helped it least. So part of the drop may come from scoring in windows, not from the model. The error pattern in the next section is there either way.
I counted, for every Tamil word in the human-labelled set, which label the model gave it. I counted labels only and did not read the comments, so the evaluation stayed blind.
Telugu is as common an answer as Tamil itself.
It is not a labelling mismatch: names, numbers and English barely move. The model calls Tamil words Telugu or another South Indian language. The same pull shows up, more weakly, elsewhere: 9% of Kannada words and 5.5% of Malayalam words are also tagged Telugu.
My own 425-comment set did not show this, probably because its Tamil comments were easier. The comment at the top of this post is an example: without a language hint, Bhasha tags padam, semma, konjam and vera as Telugu.
My best explanation is the kind of Tamil the model learned. Written Tamil, the kind found in news, differs a lot from spoken Tamil: a newspaper writes irukkiradhu where a commenter writes irukku. Bhasha only ever saw news Tamil, converted to Roman letters. When it meets spoken forms it has never seen, it falls back on the nearest South Indian language it has.
Most users already know their language pair: a Tamil film channel gets Tamil and English. So the wrapper has a langs option that limits the output to the expected language plus English, name and other. It is a fixed rule with nothing tuned on the test sets. I scored it once on every set.
It helps every human-labelled Dravidian set. For Tamil it mostly moves the errors to English.
For Tamil the details matter more than the total. With Telugu ruled out, the Tamil score itself rises from 0.53 to 0.78. But 2,188 Tamil words then come out as English, and the English score falls from 0.92 to 0.73. Among the allowed labels, the model ranks English above Tamil for those words (another 653 become name), so it does not recognise them as Tamil at all. On my own 425-comment set the hint does not help Tamil either (0.873 to 0.865).
That supports the spoken-Tamil explanation, and the one inference-time rule I tried does not fix it. The model needs to see spoken Tamil in training. That is the plan for v2: real CC-BY YouTube comments from DravidianCodeMix’s training split, with word labels from a simple rule, plus chat-style text written by open-weights (Apache-2.0) models. That would be the first LLM-written text in Bhasha’s training data, and the card will say so. The human-labelled sets stay evaluation-only.
Good uses:
Use with care:
How is it different from GlotLID or IndicLID? GlotLID and IndicLID give one language per text. Bhasha gives one per word, which is what you need when a sentence mixes two languages. On whole messages, it also beats GlotLID at finding the main language of real code-mixed comments.
Is it free for commercial use? Yes. The weights and code are Apache-2.0, and the base model (XLM-RoBERTa-base) is MIT. The training data was built by code from CC0, CC-BY, MIT and ODC-By sources. The wordfreq word lists (CC-BY-SA) were used only as a lookup while building the data and are not in the model; the NOTICE file says so.
Was it trained on real comments? No. Training rows were built by code from news and web sentences. Real comments were used only for evaluation and are not redistributed.
If you speak Tamil, Telugu or any of the untested languages and Bhasha gets your text wrong, open a discussion on the model page. Real failure cases are the most useful thing anyone can send me.
Bhasha, Nandi, Chitragupta and Yama are independent personal releases by Arun Prabhakaran, licensed Apache-2.0. Evaluation data: CoLI-Dravidian 2024 (FIRE 2024 shared task); CoLI-Kanglish (ICON 2022 shared task); WoLLaI Mal-Eng (Pondicherry University, Mendeley Data, CC BY 4.0); DravidianCodeMix (CC BY 4.0); COMI-LINGUA; LinCE; IndicTriMix; Bhasha-Abhijnaanam (CC0). Training data: IndicCorp v2 (CC0), Aksharantar (CC BY 4.0), IndicXlit (MIT), Wikidata (CC0), FineWeb-2 word counts (ODC-By).
My Model Thought Tamil Was Telugu: Open-Source Language ID for Hinglish, Tanglish and Kanglish was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.