{"slug": "show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms", "title": "Show HN: Indicate: Transliterate Indic Languages with PyTorch and LLMs", "summary": "Indicate, a new open-source tool for transliterating 12+ Indic languages to and from English, was released on Hacker News, offering both a PyTorch-based local model and LLM backends with auto-detection of source scripts. The tool supports bidirectional transliteration, batch processing, and structured JSON output, with Python 3.13+ required and weights downloaded from Hugging Face on first use.", "body_md": "**Indicate** provides high-quality transliteration between Indic languages and English using both a traditional PyTorch model and state-of-the-art LLMs (Large Language Models).\n\n**🔀 Composable Backends**: Chain a word table, a local model and an LLM in any order**🌍 Multi-Language**: 12+ Indic languages, with the source script auto-detected**🔄 Bidirectional**: Supports both Indic→English and English→Indic transliteration**🛡️ Production Ready**: Safe file handling, atomic writes, backup support**📊 Structured Output**: Rich JSON format with metadata and error handling**⚡ Batch Processing**: Efficient processing of large files with progress tracking\n\nHindi • Tamil • Telugu • Bengali • Gujarati • Kannada • Malayalam • Punjabi • Marathi • Odia • Urdu • Sanskrit ↔ English\n\nWe strongly recommend installing `indicate`\n\ninside a Python virtual environment (see [venv documentation](https://docs.python.org/3/library/venv.html#creating-virtual-environments))\n\n**Requirements:** Python 3.13+\n\n```\npip install indicate\npip install indicate\n\n# Set your API key (choose one):\nexport OPENAI_API_KEY=your-key\nexport ANTHROPIC_API_KEY=your-key  \nexport GOOGLE_API_KEY=your-key\npip install indicate\n# No API key needed. The PyTorch weights are downloaded once from Hugging Face\n# (gojiberries/indicate) on first transliterate and cached locally; tokenizers ship\n# in the wheel. After the first run it works fully offline.\n```\n\nThe Bengali word table downloads from the pinned model-assets repository on first use and is then cached. It is compiled from a shared, LLM-labeled electoral-name corpus into one deterministic native-to-Latin lookup; the multi-million-row source CSV is not duplicated in this repository or package.\n\nHindi and Punjabi tables are different: they derive from\n`data/hindi.csv.gz`\n\n(which blends CC-BY-NC IIT Bombay pairs) and\n`data/punjabi.csv.gz`\n\n(from a restricted electoral-roll deposit), neither of\nwhich is ours to redistribute under MIT. Build those from a checkout:\n\n```\nexport INDICATE_DATA_DIR=~/.local/share/indicate     # where your tables live\nuv run --group train python training/build_lookup.py --lang hindi\nuv run --group train python training/build_lookup.py --lang punjabi\n```\n\n`INDICATE_DATA_DIR`\n\nis where the builder writes and where an installed package\nlooks first. Without it the table lands inside the checkout, which a\n`pip install`\n\ned copy in `site-packages`\n\nwill never read. Keep it exported and\n`indicate languages`\n\nflips that row from `unavailable`\n\nto `ready`\n\n:\n\n``` php\nDirection                 Backend   Status\nbengali -> english        lookup    downloads on first use\n                          llm       needs an API key\npunjabi -> english        lookup    ready\n                          model     ready\n```\n\nWithout a Hindi or Punjabi table nothing breaks: `lookup`\n\ndeclines every word\nand `model`\n\nanswers them. Bengali is lookup-only locally, so an unavailable\ntable is reported as an error instead of silently returning blank text.\n\nOne command, one function. The language and the backend are arguments, not separate entry points.\n\n```\n# Source language auto-detected from the script\nindicate transliterate \"राजशेखर चिंतालपति\"\n# rajshekhar chintalpati\n\nindicate transliterate \"ਰਵਿ ਸ਼ਰਮਾ\"\n# ravi sharma\n\nindicate transliterate \"বৰুৱা\"\n# barua\n\n# Devanagari carries several languages and detection picks Hindi, so say it\n# explicitly when it is not. Marathi has no local model — hence --engine llm\nindicate transliterate \"नमस्ते\" --from marathi --engine llm\n\n# Files, with the usual safety options\nindicate transliterate --input names.txt --output roman.txt --format json --backup\nindicate transliterate --input names.txt --output roman.txt --dry-run\n\n# What can this install actually do?\nindicate languages\n\n# Model architecture, training sources, where the weights come from\nindicate info\n```\n\n`python -m indicate`\n\ndoes the same as the `indicate`\n\nscript, for when the\nconsole script is not on `PATH`\n\n.\n\n``` python\nimport indicate\n\nindicate.transliterate(\"राजशेखर चिंतालपति\")  # \"rajshekhar chintalpati\"\nindicate.transliterate(\"ਰਵਿ\", source=\"punjabi\")  # \"ravi\"\nindicate.transliterate(\"नमस्ते\", n=3)  # 3 ranked candidates\nindicate.transliterate_batch([\"हिंदी\", \"मुंबई\"])  # [\"hindi\", \"mumbai\"]\n\nindicate.supported()  # {(source, target): (backends...)}\n```\n\nA word is answered by the first backend that will answer it. The chain is an argument, so you decide how much machinery each word is worth:\n\n| chain | what it does |\n|---|---|\n`lookup, model` |\ndefault — read the table, decode the rest locally |\n`model` |\ndecode everything; what a benchmark must use |\n`lookup` |\ntable only, `\"\"` on a miss — \"is my corpus already covered?\" |\n`lookup, llm` |\nthe table intercepts the paid path |\n`lookup, model, llm` |\nescalate to a provider only what both decline |\n`llm` |\nask a provider for everything |\n\n```\nindicate transliterate \"मुंबई\" --engine model\nindicate transliterate \"मुंबई\" --engine lookup,llm --provider openai\nindicate.transliterate(\"मुंबई\", engine=[\"lookup\", \"llm\"])\nindicate.transliterate(\"मुंबई\", engine=\"model\")\n```\n\nA backend that cannot serve a direction is skipped; if none remain you get an error naming what would work, rather than a silent fallback onto something that costs money:\n\n``` bash\n$ indicate transliterate \"வணக்கம்\"\nError: no backend in ['lookup', 'model'] supports tamil->english;\ntry engine=['llm'] or see indicate.supported()\n```\n\nThat is `UnsupportedPairError`\n\n. A different failure gets its own type, because\nthe two mean opposite things:\n\n- a backend that\n**declined**— it loaded its table and had no entry for that word — is ordinary and silent.`engine=[\"lookup\"]`\n\nover an uncovered corpus declines everything and returns`\"\"`\n\n, which is the whole point of asking. - a backend that was\n**unavailable**— no table built, no weights, no network — answers nothing because it could not run. When*every*backend in the chain is in that state you get`BackendsUnavailableError`\n\nnaming each one and what to do about it, rather than an empty string that looks like an answer.\n\n```\ntry:\n    indicate.transliterate(\"राजशेखर\")\nexcept indicate.BackendsUnavailableError as exc:\n    print(exc)  # nothing could answer 1 word(s): lookup has no table (build ...\n```\n\nKnown words are answered from the word table and never reach the decoder. On\nPunjab electoral-roll text that covers 99.1% of tokens, so the model handles the\ntail: **42x** the end-to-end throughput (10,937 tok/s against 258), and an input\nthat hits entirely never even imports torch, which is worth **4.4x** on cold\nstart (0.10s to first answer against 0.44s). `training/bench_lookup.py`\n\nreproduces both.\n\nIt is also more accurate than either component alone, because the builder declines to answer where the training corpus has no majority and lets those words fall through: on the Dakshina test set, 78.8% exact against the model's 76.2% for Hindi, 77.6% against 77.0% for Punjabi.\n\nTwo caveats worth knowing before you rely on those numbers. They are measured on\n**electoral-roll names**; on general Wikipedia prose the same table covers 56.9%\nof tokens, not 99.1%, and the cold-start win largely disappears because a\nsentence almost always contains a miss. And the shipped table contains 908 of\nthe 2,500 Dakshina Hindi test words, so the Hindi accuracy figure is optimistic\nby an unknown amount. `training/build_lookup.py --eval-clean`\n\nbuilds a table\nwith every eval word withheld.\n\nUse `--engine model`\n\n(or `engine=[\"model\"]`\n\n) to measure the model by itself —\nbenchmarks must, or they score memorization. `training/seam_check.py`\n\nchecks\nthat mixing table and model output in one string stays stylistically consistent.\n\nFor whole-sentence transliteration with context, use the client rather than the engine chain — the chain resolves word by word:\n\n``` python\nfrom indicate import IndicLLMTransliterator\n\ntransliterator = IndicLLMTransliterator(\"hindi\", \"english\")\ntransliterator.transliterate(\"राजशेखर चिंतालपति\")\ntransliterator.transliterate_batch([\"राजेश\", \"गौरव\", \"प्रिया\"])\n```\n\nFor millions of tokens, `indicate.batch`\n\nsubmits to a provider's async Batch API\nwith checkpointing, and answers what it can locally first:\n\n``` python\nfrom indicate.batch import transliterate_tokens_batched\n\npairs = transliterate_tokens_batched(\n    tokens,\n    \"punjabi\",\n    \"english\",\n    checkpoint_path=\"run.jsonl\",\n    engine=(\"lookup\", \"llm\"),  # default; (\"lookup\",\"model\",\"llm\") goes further\n)\n```\n\n`--format json`\n\nworks with every backend, not just the LLM. One line of input in,\none entry out, with the chain that answered it recorded per row:\n\n```\n{\n  \"metadata\": {\n    \"source_language\": \"hindi\",\n    \"target_language\": \"english\",\n    \"timestamp\": \"2026-08-14T07:40:08.697757+00:00\",\n    \"total_lines\": 1,\n    \"successful_lines\": 1,\n    \"failed_lines\": 0,\n    \"format_version\": \"1.0\",\n    \"encoding\": \"utf-8\",\n    \"description\": \"Indic language transliteration results from indicate package\"\n  },\n  \"results\": [\n    {\n      \"line_number\": 1,\n      \"input_text\": \"राजेश कुमार\",\n      \"output_text\": \"rajesh kumar\",\n      \"source_lang\": \"hindi\",\n      \"target_lang\": \"english\",\n      \"confidence\": \"lookup,model\",\n      \"error\": null,\n      \"processing_time\": 0.07029390335083008,\n      \"timestamp\": \"2026-08-14T07:40:08.697423+00:00\"\n    }\n  ]\n}\n```\n\n`confidence`\n\nholds the engine chain, not a probability — the local model's beam\nscores are not calibrated, so publishing one would invite a comparison it cannot\nsupport.\n\n**🔒 Input/Output Validation**: Prevents accidental file overwrites**⚛️ Atomic Writing**: Safe file operations using temporary files**💾 Automatic Backups**: Optional timestamped backups of existing files**👁️ Dry Run Mode**: Preview operations before execution\n\nResumable runs live in `indicate.batch`\n\n, which checkpoints every resolved token\nto disk and picks up where it left off.\n\n```\n# Pick an LLM provider and model\nindicate transliterate \"text\" --engine llm --provider anthropic --model claude-3-opus\n\n# Read JSON produced by an earlier run\nindicate transliterate --input results.json --from english --to hindi --engine llm\n\n# Table only: how much of this file does the table already cover?\nindicate transliterate --input names.txt --engine lookup\n```\n\n`lookup` |\n`model` |\n`llm` |\n|\n|---|---|---|---|\nDirections |\nBengali, Hindi, Punjabi → English | Hindi, Punjabi → English | 12+ languages, any Indic pair |\nSetup |\nBengali downloads; build Hindi/Punjabi | none | API key |\nSpeed |\n10,937 tok/s end to end | 258 tok/s | network-bound |\nCost |\nfree | free | per API call |\nOffline |\n✅ | ✅ | ❌ |\nCoverage |\nonly what is in the table | every word | every word |\nAnswers with |\nthe corpus label | a decode | the provider |\n\nBoth speeds are end-to-end on roll names, measured back to back on one machine, so the ratio is the meaningful part. The table itself serves 16.9M reads/s once loaded; that number describes the dictionary, not the pipeline, and quoting it as throughput would overstate the win by three orders of magnitude.\n\n`indicate languages`\n\nprints which of these are available for a direction on your\nmachine.\n\n-\n**Clone and install**:\n\n```\ngit clone https://github.com/in-rolls/indicate.git\ncd indicate\nuv sync  # or pip install -e .\n```\n\n-\n**Run tests**:\n\n```\nuv run pytest                       # everything\nuv run pytest tests/test_engine.py  # one file\n```\n\nModel weights and lookup tables are gitignored, so a fresh clone skips the tests that need them and prints what is missing with the command that builds it. To make those skips into failures instead — which is what CI does, after building the tables from the committed corpora:\n\n```\nuv run pytest --require-artifacts\n```\n\n-\n**Test the backends**:\n\n```\n# Local, no API key\nindicate transliterate \"हिंदी\" --engine lookup,model\n\n# LLM (set an API key first)\nexport OPENAI_API_KEY=your-key\nindicate transliterate \"हिंदी\" --engine llm\n```\n\nThe datasets used to train the model:\n\n[Indian Election affidavits](https://affidavit.eci.gov.in/CandidateCustomFilter)[Google Dakshina dataset](https://github.com/google-research-datasets/dakshina)[ESPN Cric Info](https://www.espncricinfo.com/hindi/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard)for hindi version of the[english scorecard](https://www.espncricinfo.com/series/pakistan-tour-of-england-2021-1239529/england-vs-pakistan-1st-odi-1239537/full-scorecard)[IIT Bombay English-Hindi Corpus](https://www.cfilt.iitb.ac.in/iitb_parallel/)\n\nThe v2 models (trained on our data + the public [Aksharantar](https://huggingface.co/datasets/ai4bharat/Aksharantar)\ncorpus) are benchmarked against **AI4Bharat IndicXlit** — the same direction\n(native→Latin), the same test sets, the same metric (Top-1 exact-match,\nmatch-any-reference). Training is leakage-filtered so no eval word appears in it.\n\n| Model | Dakshina (gold) | Held-out-own names¹ |\n|---|---|---|\n| Hindi → English | 74.4% (IndicXlit 73.2%) |\n52.8% (IndicXlit 49.7%) |\n| Punjabi → English | 71.9% (IndicXlit 73.2%) | 56.9% (IndicXlit 53.5%) |\n\n¹ Held-out slice of our own electoral/affidavit names — the cleanest comparison,\nsince IndicXlit never trained on it. **v2 matches or edges IndicXlit on the gold\nbenchmark and beats it on the deployment domain.** Primary metric is Top-1\nexact-match; CER (character error rate) is the soft companion. Reproduce with\n`training/eval.py`\n\nand `training/compare.py`\n\n.\n\nBelow is the edit-distance distribution on the test set (0 = exact match):\n\nRajashekar Chintalapati and Gaurav Sood\n\nThe project welcomes contributions from everyone! In fact, it depends on it. To maintain this welcoming atmosphere, and to collaborate in a fun and productive way, we expect contributors to the project to abide by the [Contributor Code of Conduct](http://contributor-covenant.org/version/1/0/0/).\n\nThe package is released under the [MIT License](https://opensource.org/licenses/MIT).", "url": "https://wpnews.pro/news/show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms", "canonical_source": "https://github.com/in-rolls/indicate", "published_at": "2026-09-03 03:22:09+00:00", "updated_at": "2026-09-03 03:52:23.741870+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "natural-language-processing", "ai-tools"], "entities": ["Indicate", "PyTorch", "Hugging Face", "IIT Bombay"], "alternates": {"html": "https://wpnews.pro/news/show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms", "markdown": "https://wpnews.pro/news/show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms.md", "text": "https://wpnews.pro/news/show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms.txt", "jsonld": "https://wpnews.pro/news/show-hn-indicate-transliterate-indic-languages-with-pytorch-and-llms.jsonld"}}