Show HN: Indicate: Transliterate Indic Languages with PyTorch and LLMs

Indicate, a new open-source tool for transliterating 12+ Indic languages to and from English, was released on Hacker News, offering both a PyTorch-based local model and LLM backends with auto-detection of source scripts. The tool supports bidirectional transliteration, batch processing, and structured JSON output, with Python 3.13+ required and weights downloaded from Hugging Face on first use.

Indicate provides high-quality transliteration between Indic languages and English using both a traditional PyTorch model and state-of-the-art LLMs Large Language Models . 🔀 Composable Backends : Chain a word table, a local model and an LLM in any order 🌍 Multi-Language : 12+ Indic languages, with the source script auto-detected 🔄 Bidirectional : Supports both Indic→English and English→Indic transliteration 🛡️ Production Ready : Safe file handling, atomic writes, backup support 📊 Structured Output : Rich JSON format with metadata and error handling ⚡ Batch Processing : Efficient processing of large files with progress tracking Hindi • Tamil • Telugu • Bengali • Gujarati • Kannada • Malayalam • Punjabi • Marathi • Odia • Urdu • Sanskrit ↔ English We strongly recommend installing indicate inside a Python virtual environment see venv documentation https://docs.python.org/3/library/venv.html creating-virtual-environments Requirements: Python 3.13+ pip install indicate pip install indicate Set your API key choose one : export OPENAI API KEY=your-key export ANTHROPIC API KEY=your-key export GOOGLE API KEY=your-key pip install indicate No API key needed. The PyTorch weights are downloaded once from Hugging Face gojiberries/indicate on first transliterate and cached locally; tokenizers ship in the wheel. After the first run it works fully offline. The Bengali word table downloads from the pinned model-assets repository on first use and is then cached. It is compiled from a shared, LLM-labeled electoral-name corpus into one deterministic native-to-Latin lookup; the multi-million-row source CSV is not duplicated in this repository or package. Hindi and Punjabi tables are different: they derive from data/hindi.csv.gz which blends CC-BY-NC IIT Bombay pairs and data/punjabi.csv.gz from a restricted electoral-roll deposit , neither of which is ours to redistribute under MIT. Build those from a checkout: export INDICATE DATA DIR=~/.local/share/indicate where your tables live uv run --group train python training/build lookup.py --lang hindi uv run --group train python training/build lookup.py --lang punjabi INDICATE DATA DIR is where the builder writes and where an installed package looks first. Without it the table lands inside the checkout, which a pip install ed copy in site-packages will never read. Keep it exported and indicate languages flips that row from unavailable to ready : php Direction Backend Status bengali - english lookup downloads on first use llm needs an API key punjabi - english lookup ready model ready Without a Hindi or Punjabi table nothing breaks: lookup declines every word and model answers them. Bengali is lookup-only locally, so an unavailable table is reported as an error instead of silently returning blank text. One command, one function. The language and the backend are arguments, not separate entry points. Source language auto-detected from the script indicate transliterate "राजशेखर चिंतालपति" rajshekhar chintalpati indicate transliterate "ਰਵਿ ਸ਼ਰਮਾ" ravi sharma indicate transliterate "বৰুৱা" barua Devanagari carries several languages and detection picks Hindi, so say it explicitly when it is not. Marathi has no local model — hence --engine llm indicate transliterate "नमस्ते" --from marathi --engine llm Files, with the usual safety options indicate transliterate --input names.txt --output roman.txt --format json --backup indicate transliterate --input names.txt --output roman.txt --dry-run What can this install actually do? indicate languages Model architecture, training sources, where the weights come from indicate info python -m indicate does the same as the indicate script, for when the console script is not on PATH . python import indicate indicate.transliterate "राजशेखर चिंतालपति" "rajshekhar chintalpati" indicate.transliterate "ਰਵਿ", source="punjabi" "ravi" indicate.transliterate "नमस्ते", n=3 3 ranked candidates indicate.transliterate batch "हिंदी", "मुंबई" "hindi", "mumbai" indicate.supported { source, target : backends... } A word is answered by the first backend that will answer it. The chain is an argument, so you decide how much machinery each word is worth: | chain | what it does | |---|---| lookup, model | default — read the table, decode the rest locally | model | decode everything; what a benchmark must use | lookup | table only, "" on a miss — "is my corpus already covered?" | lookup, llm | the table intercepts the paid path | lookup, model, llm | escalate to a provider only what both decline | llm | ask a provider for everything | indicate transliterate "मुंबई" --engine model indicate transliterate "मुंबई" --engine lookup,llm --provider openai indicate.transliterate "मुंबई", engine= "lookup", "llm" indicate.transliterate "मुंबई", engine="model" A backend that cannot serve a direction is skipped; if none remain you get an error naming what would work, rather than a silent fallback onto something that costs money: bash $ indicate transliterate "வணக்கம்" Error: no backend in 'lookup', 'model' supports tamil- english; try engine= 'llm' or see indicate.supported That is UnsupportedPairError . A different failure gets its own type, because the two mean opposite things: - a backend that declined — it loaded its table and had no entry for that word — is ordinary and silent. engine= "lookup" over an uncovered corpus declines everything and returns "" , which is the whole point of asking. - a backend that was unavailable — no table built, no weights, no network — answers nothing because it could not run. When every backend in the chain is in that state you get BackendsUnavailableError naming each one and what to do about it, rather than an empty string that looks like an answer. try: indicate.transliterate "राजशेखर" except indicate.BackendsUnavailableError as exc: print exc nothing could answer 1 word s : lookup has no table build ... Known words are answered from the word table and never reach the decoder. On Punjab electoral-roll text that covers 99.1% of tokens, so the model handles the tail: 42x the end-to-end throughput 10,937 tok/s against 258 , and an input that hits entirely never even imports torch, which is worth 4.4x on cold start 0.10s to first answer against 0.44s . training/bench lookup.py reproduces both. It is also more accurate than either component alone, because the builder declines to answer where the training corpus has no majority and lets those words fall through: on the Dakshina test set, 78.8% exact against the model's 76.2% for Hindi, 77.6% against 77.0% for Punjabi. Two caveats worth knowing before you rely on those numbers. They are measured on electoral-roll names ; on general Wikipedia prose the same table covers 56.9% of tokens, not 99.1%, and the cold-start win largely disappears because a sentence almost always contains a miss. And the shipped table contains 908 of the 2,500 Dakshina Hindi test words, so the Hindi accuracy figure is optimistic by an unknown amount. training/build lookup.py --eval-clean builds a table with every eval word withheld. Use --engine model or engine= "model" to measure the model by itself — benchmarks must, or they score memorization. training/seam check.py checks that mixing table and model output in one string stays stylistically consistent. For whole-sentence transliteration with context, use the client rather than the engine chain — the chain resolves word by word: python from indicate import IndicLLMTransliterator transliterator = IndicLLMTransliterator "hindi", "english" transliterator.transliterate "राजशेखर चिंतालपति" transliterator.transliterate batch "राजेश", "गौरव", "प्रिया" For millions of tokens, indicate.batch submits to a provider's async Batch API with checkpointing, and answers what it can locally first: python from indicate.batch import transliterate tokens batched pairs = transliterate tokens batched tokens, "punjabi", "english", checkpoint path="run.jsonl", engine= "lookup", "llm" , default; "lookup","model","llm" goes further --format json works with every backend, not just the LLM. One line of input in, one entry out, with the chain that answered it recorded per row: { "metadata": { "source language": "hindi", "target language": "english", "timestamp": "2026-08-14T07:40:08.697757+00:00", "total lines": 1, "successful lines": 1, "failed lines": 0, "format version": "1.0", "encoding": "utf-8", "description": "Indic language transliteration results from indicate package" }, "results": { "line number": 1, "input text": "राजेश कुमार", "output text": "rajesh kumar", "source lang": "hindi", "target lang": "english", "confidence": "lookup,model", "error": null, "processing time": 0.07029390335083008, "timestamp": "2026-08-14T07:40:08.697423+00:00" } } confidence holds the engine chain, not a probability — the local model's beam scores are not calibrated, so publishing one would invite a comparison it cannot support. 🔒 Input/Output Validation : Prevents accidental file overwrites ⚛️ Atomic Writing : Safe file operations using temporary files 💾 Automatic Backups : Optional timestamped backups of existing files 👁️ Dry Run Mode : Preview operations before execution Resumable runs live in indicate.batch , which checkpoints every resolved token to disk and picks up where it left off. Pick an LLM provider and model indicate transliterate "text" --engine llm --provider anthropic --model claude-3-opus Read JSON produced by an earlier run indicate transliterate --input results.json --from english --to hindi --engine llm Table only: how much of this file does the table already cover? indicate transliterate --input names.txt --engine lookup lookup | model | llm | | |---|---|---|---| Directions | Bengali, Hindi, Punjabi → English | Hindi, Punjabi → English | 12+ languages, any Indic pair | Setup | Bengali downloads; build Hindi/Punjabi | none | API key | Speed | 10,937 tok/s end to end | 258 tok/s | network-bound | Cost | free | free | per API call | Offline | ✅ | ✅ | ❌ | Coverage | only what is in the table | every word | every word | Answers with | the corpus label | a decode | the provider | Both speeds are end-to-end on roll names, measured back to back on one machine, so the ratio is the meaningful part. The table itself serves 16.9M reads/s once loaded; that number describes the dictionary, not the pipeline, and quoting it as throughput would overstate the win by three orders of magnitude. indicate languages prints which of these are available for a direction on your machine. - Clone and install : git clone https://github.com/in-rolls/indicate.git cd indicate uv sync or pip install -e . - Run tests : uv run pytest everything uv run pytest tests/test engine.py one file Model weights and lookup tables are gitignored, so a fresh clone skips the tests that need them and prints what is missing with the command that builds it. To make those skips into failures instead — which is what CI does, after building the tables from the committed corpora: uv run pytest --require-artifacts - Test the backends : Local, no API key indicate transliterate "हिंदी" --engine lookup,model LLM set an API key first export OPENAI API KEY=your-key indicate transliterate "हिंदी" --engine llm The datasets used to train the model: Indian Election affidavits https://affidavit.eci.gov.in/CandidateCustomFilter Google Dakshina dataset https://github.com/google-research-datasets/dakshina ESPN Cric Info https://www.espncricinfo.com/hindi/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard for hindi version of the english scorecard https://www.espncricinfo.com/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard IIT Bombay English-Hindi Corpus https://www.cfilt.iitb.ac.in/iitb parallel/ The v2 models trained on our data + the public Aksharantar https://huggingface.co/datasets/ai4bharat/Aksharantar corpus are benchmarked against AI4Bharat IndicXlit — the same direction native→Latin , the same test sets, the same metric Top-1 exact-match, match-any-reference . Training is leakage-filtered so no eval word appears in it. | Model | Dakshina gold | Held-out-own names¹ | |---|---|---| | Hindi → English | 74.4% IndicXlit 73.2% | 52.8% IndicXlit 49.7% | | Punjabi → English | 71.9% IndicXlit 73.2% | 56.9% IndicXlit 53.5% | ¹ Held-out slice of our own electoral/affidavit names — the cleanest comparison, since IndicXlit never trained on it. v2 matches or edges IndicXlit on the gold benchmark and beats it on the deployment domain. Primary metric is Top-1 exact-match; CER character error rate is the soft companion. Reproduce with training/eval.py and training/compare.py . Below is the edit-distance distribution on the test set 0 = exact match : Rajashekar Chintalapati and Gaurav Sood The project welcomes contributions from everyone In fact, it depends on it. To maintain this welcoming atmosphere, and to collaborate in a fun and productive way, we expect contributors to the project to abide by the Contributor Code of Conduct http://contributor-covenant.org/version/1/0/0/ . The package is released under the MIT License https://opensource.org/licenses/MIT .