A free model that sorts complaints in 13 Indian languages into 53 categories based on RBI, IRDAI and CPGRAMS lists. What a synthetic test hid, and what real posts showed.
By Arun Prabhakaran · @LatentAILabs
TL;DR
Parts of this article were drafted with AI assistants; I checked every number against the model card and the evaluation logs.
RBI’s ombudsman scheme received 13.34 lakh complaints in 2024–25. The central government’s grievance portal, CPGRAMS, redressed over 26 lakh in 2024. Behind those official numbers are many more complaints that never reach a regulator: app reviews, tweets, emails and support chats sent to banks, insurers, telecom companies and government offices.
Every one of those messages has to land on someone’s desk first. Here is a typical one (a made-up example):
“UPI se 2,000 rupaye kat gaye par receiver ko nahi mile, aaj subah ki baat hai. Bank helpline koi jawab nahi de raha.” (₹2,000 left my account by UPI this morning but never reached the receiver. The bank helpline isn’t answering.)
Before a support agent can act, triage has to answer three questions: which desk owns this (digital payments, not fraud or loans), how urgent it is (money lost today), and what the fallback categories are if the first guess is wrong.
In Shiva temples, people whisper their wishes into the ear of Nandi, the bull who keeps Shiva’s gate. Nandi does not grant the wish; he makes sure it reaches the right place. That is Nandi’s entire scope: it does not resolve complaints or judge whether they are valid; it routes each message to the right desk.
Nandi is the third model in a small open set for Indian languages. Yama is a zero-shot classifier: it judges text against labels you write at run time. Chitragupta keeps the records: it finds and masks personal data. Nandi exists because on routing into a fixed set of categories, a fine-tuned 560M encoder beats a zero-shot generalist of the same size by 25.9 points on real text.
route_batch(texts) handles lists. Under the hood, Nandi is an XLM-RoBERTa-large encoder with mean pooling and two linear heads: a 53-way category head and a 3-way urgency head.
One warning belongs right next to the code: the confidence number is not calibrated. On real text Nandi often outputs 0.99 on wrong predictions. Treat top3 as suggestions for a human, never as a threshold for fully automatic routing.
I did not want to invent a taxonomy. Indian regulators already publish what people complain about, so the categories follow their lists:
Each category records where it came from: 23 are verbatim items on an official complaint form, 17 are CPGRAMS categories, 4 are National Consumer Helpline sectors, 4 are derived from official buckets, and 5 (railways, ration cards, two telecom groups, coaching) have no published list, so I defined them. The category names belong to the regulators; the mapping is mine, not an official one.
A few boundary rules matter more than they look:
No openly licensed dataset of labelled Indian complaints exists, and I had a hard rule against training on real complaints, production data or customer data. So every training text is synthetic, written and filtered only by open-weights models (Gemma 4 and Qwen3.6, both Apache-2.0):
The result: 116,199 texts in 13 languages, of which 15,902 are non-complaints and 24,638 are urgent (P1). Every name, account, phone and policy number is a dummy value. No output of a proprietary model is in the training data.
I wrote down the ship rules before generating any data: on a synthetic test, macro-F1 at least 0.10 above Yama, urgent recall at least 0.90, and no more than 10% of non-complaints routed to a category.
To keep the test honest, it was written by a different model (Qwen3.6, not Gemma), with channels and personas held out from training. Nandi passed easily:
Even with the test written by a different model family and unseen personas, the 560M specialist beat the 35B LLM. Stopping here would have hidden a 30-point drop on real text.
Even when a different LLM writes it, a synthetic test shares the definitions, prompt structure and tidy detail of the training data. So the real decision was made on real text, scored once.
I built a blind set of 443 real public posts: Play Store and App Store reviews from India and posts from a public forum, collected only after the model was frozen. 38% are not in English, mostly Hindi and Hinglish. Apps and companies from my two earlier real sets were excluded. Three independent LLM labellers (Claude agents, not humans) labelled each post from the text alone, using the same written definitions the training teachers used, and a 2-of-3 vote decided. I wrote the evaluation plan and the pass rules before collecting a single post, and scored the set once.
Two cautions about those labels. They are LLM-reviewed, not human-verified. And all three labellers are the same model, so their 91% agreement measures one model’s consistency, not true agreement between independent annotators, and overstates how reliable the labels are.
Every model drops on real text, and Nandi drops the most (30 points).
On real posts the 35B LLM leads on accuracy and, by a lot, on rare categories.
What the real numbers say:
Real complaints are shorter, vaguer and messier than anything an LLM writes when you ask it for a complaint. An imagined one-line review such as app is fraud, money gone carries less signal than three tidy sentences about a failed UPI transfer.
A correction, disclosed. After scoring, I found a mistake in my own labelling guide. It told the labellers to put recovery-agent threats under “loans” instead of the taxonomy’s own recovery-harassment category, so models that picked the correct category were marked wrong. On those 35 posts I now accept either label. Before the correction Nandi scored 0.571 accuracy and 0.397 macro-F1 (Qwen 0.607 and 0.593, Yama 0.352 and 0.256). Urgency numbers did not change, and neither did any pass-or-fail decision.
Before collecting the real set, I fixed the bar for calling urgency reliable: the lower end of the 95% interval for urgent (P1) recall had to reach 0.50.
While the set was being labelled, a review of the plan pointed out that about 40% of its urgent posts were loan-app harassment, which might be easy to spot. So before scoring, I added a second condition: the bar also had to hold without the loan-app posts.
The pre-registered bar is the dashed line. Nandi’s intervals sit well to its left.
So the urgency head ships, but the model card calls it experimental: a hint for a human, not a triage decision. That split was the most useful step in the whole evaluation. Without it, a 0.43 recall on the easier loan-app slice would have hidden a 0.10 recall on every other kind of urgent complaint.
Before freezing the model, I used a public set of tweets as a development set: the Mendeley Enhanced Dataset of Citizen Centric Complaints and Grievances on Twitter (Agarwal, Mittal and Sureka, IIIT-Delhi, CC BY 4.0). It has English tweets to Indian Railways, the Income Tax Department, Delhi Police and Delhi Traffic Police. I relabelled 2,249 of them with this taxonomy and masked the official @handles, which would otherwise give the category away.
The shipped model, v5b, routed 40% of the non-complaint tweets (news, live traffic updates, opinions) to a category. That looked fixable. I generated about 5,600 new look-alike texts, such as traffic updates next to synthetic traffic complaints, and retrained twice.
Both retrained versions cut false alarms, and both started rejecting real complaints.
I kept v5b and closed that line of work. A model that sometimes routes a news tweet costs an agent a few seconds. A model that drops a real complaint costs the person who wrote it. The model card now lists “about 1 in 5 non-complaints routed” as a known limit, not a solved problem.
Mask, route, then a human decides. All three steps run locally.
Nandi works best as a suggestion layer: show the agent the top 3 categories, let them confirm with one click, and log the disagreements. If the text holds personal data, run Chitragupta’s redact() first so far less of it reaches your logs. Nandi's accuracy on redacted text has not been measured, so check the chained pipeline on a sample of your own first.
Where it fits:
Where it does not fit:
What is Nandi? Nandi (arunp0/nandi-grievance-indic) is an open-source complaint classification model for Indian languages and Hinglish. It reads a complaint and returns one of 53 categories based on RBI, IRDAI, CPGRAMS and National Consumer Helpline lists, the top 3 categories with probabilities, and an experimental urgency level.
Which languages does it support? Hindi, Hinglish, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and Indian English, in native scripts and in Roman letters.
How accurate is it on real complaints? On 443 real public posts labelled blind by three LLM labellers, it picked the right category 62.5% of the time and had it in the top 3 81.3% of the time.
Is it better than using an LLM? Not on accuracy. Qwen3.6–35B-A3B zero-shot scored 66.4% on the same posts and is much better on rare categories. Nandi is about 60 times smaller, runs locally on a CPU, and sends no text to an API.
Can it detect urgent complaints? Only as a hint. It catches about 1 in 4 real urgent posts, so it failed its own reliability rule and is marked experimental.
Is it free for commercial use? Yes. The weights and code are Apache-2.0, the base model (XLM-RoBERTa-large) is MIT, and the training data was generated only with open-weights models (Gemma 4 and Qwen3.6, both Apache-2.0).
Was it trained on real complaints? No. All training text is synthetic. Real posts were used only for evaluation, and they are not released.
If you run Nandi on your own complaints and it gets them wrong, open a discussion on the model page. Real failure cases are the most useful thing anyone can send me.
Nandi, Chitragupta and Yama are independent personal releases by Arun Prabhakaran, licensed Apache-2.0. The development tweets come from Agarwal, Mittal and Sureka (IIIT-Delhi), Enhanced Dataset of Citizen Centric Complaints and Grievances on Twitter, Mendeley Data (CC BY 4.0). Complaint volumes: RBI, Annual Report of the Ombudsman Scheme 2024–25; DARPG, CPGRAMS 2024.
Nandi Carries Every Complaint to the Right Desk: Open-Source AI for Indian Languages was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.