TL;DR
Here is a typical support message in India:
“Hi, main Rohit Verma bol raha hoon. Mera Aadhaar XXXX XXXX 5246 hai aur UPI rohit.example@upi, refund isi pe bhej do. Call: 98765 43210”
Before this message goes anywhere useful, two things have to happen:
In Indian tradition, these are two different jobs. Yama is the judge who weighs what is placed before him and gives a verdict. Chitragupta, his record-keeper, writes down the details of every person’s life.
I built one open model for each job. My previous article, Before You Invent a New Architecture, Try a Bigger Encoder, covered Yama. This one introduces Chitragupta and shows how the two work together.
Personal data in Indian text is hard to catch with English tools, for four reasons:
Most open PII models are English-first. Of the Indian-focused ones I tested, few cover Indian IDs, 13 languages and a permissive licence in one model.
Chitragupta tags 18 types of personal data:
It works in Hindi, Hinglish, Bengali, Tamil, Telugu, Marathi, Gujarati, Kannada, Malayalam, Punjabi, Urdu, Odia and Indian English, in native scripts and in Roman letters, including native digits and numbers spoken out as words.
detect() returns character offsets, so you can mask, hash or tokenise the values however your system needs. Ordinary numbers are left alone: m.detect("Order #4410293 delivered on 12/03/2025 at 10:30. Thanks!") returns nothing.
On a multi-core server CPU it takes about 24 ms per short message in batches of 32. You can also try it in the box on the model page.
Chitragupta has two parts.
1. A tagger. XLM-RoBERTa-large (560M parameters) fine-tuned to label every token as the beginning, inside or outside of a personal-data span.
2. A small rule layer. Most Indian IDs carry their own proof of validity:
When the tagger sees an ID format it has never met, it often splits it into pieces with mixed labels. The rules glue the pieces back together and let the checksum decide the label. They never touch names or addresses, and they never relabel bank account or driving licence numbers, which have no checksum.
The effect is large where it matters:
“Strict” means separator styles never seen in training. They turned out easier than unseen digit groupings, which is why that slice scores higher.
So the rules mostly fix labels, not coverage. The model already finds almost every value; the checksum decides whether it is an Aadhaar or a card number. A small model cannot memorise every way people type a GSTIN, but a few lines of arithmetic can verify one.
No real personal data, production data or public PII dataset went into training. Every training text is synthetic:
The lesson I would pass on: when you generate training data with an LLM, let code produce the labels. LLMs are good at writing realistic text and unreliable at reporting exact character offsets.
A model trained on synthetic data can score well simply by memorising its own generator. I built the test set to make that impossible:
The test set has 1,582 texts: 1,289 with personal data (6,123 spans) and 293 without. On the texts with no personal data, Chitragupta raised a false alarm on 2.4% (7 of 293).
After v1, I trained three v2 variants to fix two weaknesses the public benchmarks exposed: short one-line addresses and look-alike numbers such as timestamps and invoice IDs. On Ylemis, v2b jumped from 0.878 to 0.998. That gain is not independent: I designed v2’s data after reading the benchmark’s errors.
Before looking at the results, I had written down a release rule: ship v2 only if it is not significantly worse than v1 on the fresh test set, overall or on driving licences.
Both variants I tested with the bootstrap significantly hurt driving licences (v2b −3.5, v2c −1.6 points), which fails the rule. v2c also lost 3.6 points on voter IDs, and the first v2 had already broken slash-separated IDs. Teaching the model that invoice numbers are not personal data made it more cautious about ID formats it had never seen. A missed voter ID leaks real data; a masked timestamp is only over-redaction. So I shipped v1, and the model card lists short addresses and look-alike numbers as known limits.
I ran nine public alternatives on the same texts. To compare models with different label sets, the score is character-level redaction F1: whatever label a model uses, did it mask the personal-data characters?
Two caveats belong next to this chart:
On two public benchmarks built by other people, Chitragupta is not the best model.
If your text has short addresses, test both models on a sample of your own data before choosing. Benchmarks tell you where to look, not what to deploy.
The pipeline from the diagram fits in a few lines. Chitragupta masks the personal data; Yama decides what the message is about. Far less personal data reaches your logs or a hosted LLM.
import thx01 # pip install thx01 (the open-source library Yama uses)yama = thx01.load("arunp0/yama-indic")safe = m.redact(text) # Chitragupta, from the snippet aboveout = yama.system_one(safe, { "topic": {"type": "choice", "instructions": "What is the customer writing about?", "criteria": {"refund": "asking for a refund", "failed_payment": "payment failed", "account": "login or account access", "other": None}}, "urgent": {"type": "noul", "instructions": "Does the customer need help urgently?"},})# log `safe` and `out`, never `text`
Yama is a zero-shot classifier: you write the labels and questions at run time, with no fine-tuning. On 33 tasks it never trained on, across 19 languages, it scores 0.814 accuracy, against 0.770 for the next best zero-shot model I tested. I have not measured Yama’s accuracy on redacted text, so check it on your own data.
What is Chitragupta? Chitragupta (arunp0/chitragupta-pii-indic) is an open-source PII detection and redaction model for Indian languages and Hinglish. It finds 18 types of personal data, including Aadhaar, PAN, UPI IDs, phone numbers and names, in 13 languages, and returns their character offsets.
How do I mask Aadhaar or PAN numbers in Python? Download the model with snapshot_download("arunp0/chitragupta-pii-indic"), then call Chitragupta(path).redact(text). It replaces each value with its label, such as [AADHAAR] or [PAN]. The full snippet is above.
Is it free for commercial use? Yes. The weights and code are Apache-2.0, the base model (XLM-RoBERTa-large) is MIT, and the training data was generated only with open-weights models (Gemma 4 and Qwen3.6, both Apache-2.0).
Does it run offline? Yes. It runs locally on CPU or GPU with PyTorch, and no text leaves your machine.
How is it different from Microsoft Presidio or regex rules? Regex finds well-formed IDs but misses names, addresses, unusual formats, native digits and spoken numbers. On the fresh test set, Chitragupta masked 99.0% of personal-data characters, against 55.0% for Presidio with its Indian recognisers.
Does it make a system DPDP-compliant? No model does that on its own. It helps you keep personal data out of logs, analytics and LLM prompts, as one part of a privacy process.
What is Yama? Yama (arunp0/yama-indic) is a companion zero-shot text classifier for the same 13 languages. You describe the labels in plain English, and it returns a calibrated probability for each, without fine-tuning.
If you run Chitragupta on real text and it misses something, open a discussion on the model page. Real failure cases are the most useful thing anyone can send me.
Chitragupta and Yama are independent personal releases by Arun Prabhakaran, licensed Apache-2.0.
Drafted with help from Claude (Anthropic); experiments, numbers and final edits by the author.
Yama Judges, Chitragupta Keeps the Records: Open-Source AI for Indian Languages was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.