Indicate provides high-quality transliteration between Indic languages and English using both a traditional PyTorch model and state-of-the-art LLMs (Large Language Models).
🔀 Composable Backends: Chain a word table, a local model and an LLM in any order🌍 Multi-Language: 12+ Indic languages, with the source script auto-detected🔄 Bidirectional: Supports both Indic→English and English→Indic transliteration🛡️ Production Ready: Safe file handling, atomic writes, backup support📊 Structured Output: Rich JSON format with metadata and error handling⚡ Batch Processing: Efficient processing of large files with progress tracking
Hindi • Tamil • Telugu • Bengali • Gujarati • Kannada • Malayalam • Punjabi • Marathi • Odia • Urdu • Sanskrit ↔ English
We strongly recommend installing indicate
inside a Python virtual environment (see venv documentation)
Requirements: Python 3.13+
pip install indicate
pip install indicate
export OPENAI_API_KEY=your-key
export ANTHROPIC_API_KEY=your-key
export GOOGLE_API_KEY=your-key
pip install indicate
The Bengali word table downloads from the pinned model-assets repository on first use and is then cached. It is compiled from a shared, LLM-labeled electoral-name corpus into one deterministic native-to-Latin lookup; the multi-million-row source CSV is not duplicated in this repository or package.
Hindi and Punjabi tables are different: they derive from
data/hindi.csv.gz
(which blends CC-BY-NC IIT Bombay pairs) and
data/punjabi.csv.gz
(from a restricted electoral-roll deposit), neither of which is ours to redistribute under MIT. Build those from a checkout:
export INDICATE_DATA_DIR=~/.local/share/indicate # where your tables live
uv run --group train python training/build_lookup.py --lang hindi
uv run --group train python training/build_lookup.py --lang punjabi
INDICATE_DATA_DIR
is where the builder writes and where an installed package
looks first. Without it the table lands inside the checkout, which a
pip install
ed copy in site-packages
will never read. Keep it exported and
indicate languages
flips that row from unavailable
to ready
:
Direction Backend Status
bengali -> english lookup downloads on first use
llm needs an API key
punjabi -> english lookup ready
model ready
Without a Hindi or Punjabi table nothing breaks: lookup
declines every word
and model
answers them. Bengali is lookup-only locally, so an unavailable table is reported as an error instead of silently returning blank text.
One command, one function. The language and the backend are arguments, not separate entry points.
indicate transliterate "राजशेखर चिंतालपति"
indicate transliterate "ਰਵਿ ਸ਼ਰਮਾ"
indicate transliterate "বৰুৱা"
indicate transliterate "नमस्ते" --from marathi --engine llm
indicate transliterate --input names.txt --output roman.txt --format json --backup
indicate transliterate --input names.txt --output roman.txt --dry-run
indicate languages
indicate info
python -m indicate
does the same as the indicate
script, for when the
console script is not on PATH
.
import indicate
indicate.transliterate("राजशेखर चिंतालपति") # "rajshekhar chintalpati"
indicate.transliterate("ਰਵਿ", source="punjabi") # "ravi"
indicate.transliterate("नमस्ते", n=3) # 3 ranked candidates
indicate.transliterate_batch(["हिंदी", "मुंबई"]) # ["hindi", "mumbai"]
indicate.supported() # {(source, target): (backends...)}
A word is answered by the first backend that will answer it. The chain is an argument, so you decide how much machinery each word is worth:
| chain | what it does |
|---|---|
lookup, model |
|
| default — read the table, decode the rest locally | |
model |
|
| decode everything; what a benchmark must use | |
lookup |
|
table only, "" on a miss — "is my corpus already covered?" |
|
lookup, llm |
|
| the table intercepts the paid path | |
lookup, model, llm |
|
| escalate to a provider only what both decline | |
llm |
|
| ask a provider for everything |
indicate transliterate "मुंबई" --engine model
indicate transliterate "मुंबई" --engine lookup,llm --provider openai
indicate.transliterate("मुंबई", engine=["lookup", "llm"])
indicate.transliterate("मुंबई", engine="model")
A backend that cannot serve a direction is skipped; if none remain you get an error naming what would work, rather than a silent fallback onto something that costs money:
$ indicate transliterate "வணக்கம்"
Error: no backend in ['lookup', 'model'] supports tamil->english;
try engine=['llm'] or see indicate.supported()
That is UnsupportedPairError
. A different failure gets its own type, because the two mean opposite things:
- a backend that
declined— it loaded its table and had no entry for that word — is ordinary and silent.
engine=["lookup"]
over an uncovered corpus declines everything and returns""
, which is the whole point of asking. - a backend that was
unavailable— no table built, no weights, no network — answers nothing because it could not run. Wheneverybackend in the chain is in that state you getBackendsUnavailableError
naming each one and what to do about it, rather than an empty string that looks like an answer.
try:
indicate.transliterate("राजशेखर")
except indicate.BackendsUnavailableError as exc:
print(exc) # nothing could answer 1 word(s): lookup has no table (build ...
Known words are answered from the word table and never reach the decoder. On
Punjab electoral-roll text that covers 99.1% of tokens, so the model handles the
tail: 42x the end-to-end throughput (10,937 tok/s against 258), and an input
that hits entirely never even imports torch, which is worth 4.4x on cold
start (0.10s to first answer against 0.44s). training/bench_lookup.py
reproduces both.
It is also more accurate than either component alone, because the builder declines to answer where the training corpus has no majority and lets those words fall through: on the Dakshina test set, 78.8% exact against the model's 76.2% for Hindi, 77.6% against 77.0% for Punjabi.
Two caveats worth knowing before you rely on those numbers. They are measured on
electoral-roll names; on general Wikipedia prose the same table covers 56.9%
of tokens, not 99.1%, and the cold-start win largely disappears because a
sentence almost always contains a miss. And the shipped table contains 908 of
the 2,500 Dakshina Hindi test words, so the Hindi accuracy figure is optimistic
by an unknown amount. training/build_lookup.py --eval-clean
builds a table with every eval word withheld.
Use --engine model
(or engine=["model"]
) to measure the model by itself —
benchmarks must, or they score memorization. training/seam_check.py
checks that mixing table and model output in one string stays stylistically consistent.
For whole-sentence transliteration with context, use the client rather than the engine chain — the chain resolves word by word:
from indicate import IndicLLMTransliterator
transliterator = IndicLLMTransliterator("hindi", "english")
transliterator.transliterate("राजशेखर चिंतालपति")
transliterator.transliterate_batch(["राजेश", "गौरव", "प्रिया"])
For millions of tokens, indicate.batch
submits to a provider's async Batch API with checkpointing, and answers what it can locally first:
from indicate.batch import transliterate_tokens_batched
pairs = transliterate_tokens_batched(
tokens,
"punjabi",
"english",
checkpoint_path="run.jsonl",
engine=("lookup", "llm"), # default; ("lookup","model","llm") goes further
)
--format json
works with every backend, not just the LLM. One line of input in, one entry out, with the chain that answered it recorded per row:
{
"metadata": {
"source_language": "hindi",
"target_language": "english",
"timestamp": "2026-08-14T07:40:08.697757+00:00",
"total_lines": 1,
"successful_lines": 1,
"failed_lines": 0,
"format_version": "1.0",
"encoding": "utf-8",
"description": "Indic language transliteration results from indicate package"
},
"results": [
{
"line_number": 1,
"input_text": "राजेश कुमार",
"output_text": "rajesh kumar",
"source_lang": "hindi",
"target_lang": "english",
"confidence": "lookup,model",
"error": null,
"processing_time": 0.07029390335083008,
"timestamp": "2026-08-14T07:40:08.697423+00:00"
}
]
}
confidence
holds the engine chain, not a probability — the local model's beam scores are not calibrated, so publishing one would invite a comparison it cannot support.
🔒 Input/Output Validation: Prevents accidental file overwrites⚛️ Atomic Writing: Safe file operations using temporary files💾 Automatic Backups: Optional timestamped backups of existing files👁️ Dry Run Mode: Preview operations before execution
Resumable runs live in indicate.batch
, which checkpoints every resolved token to disk and picks up where it left off.
indicate transliterate "text" --engine llm --provider anthropic --model claude-3-opus
indicate transliterate --input results.json --from english --to hindi --engine llm
indicate transliterate --input names.txt --engine lookup
lookup |
model |
llm |
|
|---|---|---|---|
Directions |
Bengali, Hindi, Punjabi → English | Hindi, Punjabi → English | 12+ languages, any Indic pair |
Setup |
Bengali downloads; build Hindi/Punjabi | none | API key |
Speed |
10,937 tok/s end to end | 258 tok/s | network-bound |
Cost |
free | free | per API call |
Offline |
✅ | ✅ | ❌ |
Coverage |
only what is in the table | every word | every word |
Answers with |
the corpus label | a decode | the provider |
Both speeds are end-to-end on roll names, measured back to back on one machine, so the ratio is the meaningful part. The table itself serves 16.9M reads/s once loaded; that number describes the dictionary, not the pipeline, and quoting it as throughput would overstate the win by three orders of magnitude.
indicate languages
prints which of these are available for a direction on your machine.
Clone and install:
git clone https://github.com/in-rolls/indicate.git
cd indicate
uv sync # or pip install -e .
Run tests:
uv run pytest # everything
uv run pytest tests/test_engine.py # one file
Model weights and lookup tables are gitignored, so a fresh clone skips the tests that need them and prints what is missing with the command that builds it. To make those skips into failures instead — which is what CI does, after building the tables from the committed corpora:
uv run pytest --require-artifacts
Test the backends:
indicate transliterate "हिंदी" --engine lookup,model
export OPENAI_API_KEY=your-key
indicate transliterate "हिंदी" --engine llm
The datasets used to train the model:
Indian Election affidavitsGoogle Dakshina datasetESPN Cric Infofor hindi version of theenglish scorecardIIT Bombay English-Hindi Corpus
The v2 models (trained on our data + the public Aksharantar corpus) are benchmarked against AI4Bharat IndicXlit — the same direction (native→Latin), the same test sets, the same metric (Top-1 exact-match, match-any-reference). Training is leakage-filtered so no eval word appears in it.
| Model | Dakshina (gold) | Held-out-own names¹ |
|---|---|---|
| Hindi → English | 74.4% (IndicXlit 73.2%) | |
| 52.8% (IndicXlit 49.7%) | ||
| Punjabi → English | 71.9% (IndicXlit 73.2%) | 56.9% (IndicXlit 53.5%) |
¹ Held-out slice of our own electoral/affidavit names — the cleanest comparison,
since IndicXlit never trained on it. v2 matches or edges IndicXlit on the gold
benchmark and beats it on the deployment domain. Primary metric is Top-1
exact-match; CER (character error rate) is the soft companion. Reproduce with
training/eval.py
and training/compare.py
.
Below is the edit-distance distribution on the test set (0 = exact match):
Rajashekar Chintalapati and Gaurav Sood
The project welcomes contributions from everyone! In fact, it depends on it. To maintain this welcoming atmosphere, and to collaborate in a fun and productive way, we expect contributors to the project to abide by the Contributor Code of Conduct.
The package is released under the MIT License.