{"slug": "how-we-built-precise-translation-and-language-identification-for-ai-book", "title": "How We Built Precise Translation and Language Identification for AI Book Translation", "summary": "LectuLibre, an AI book translation service, built a robust two-tier language identification system to solve the problem of misidentified source languages in uploaded books. The team combined fastText, langdetect, and cld3 detectors with a majority-vote algorithm, achieving high accuracy for book-level and per-chunk detection.", "body_md": "*How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation.*\n\nWhen we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation.\n\nUsers upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's `langdetect`\n\nlibrary was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits.\n\nWe needed something far more robust—what we internally call **精准翻译与语言识别** (precise translation and language identification). Here’s how we built it.\n\nOur first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection.\n\nWe realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation.\n\nWe created a `LanguageDetector`\n\nclass that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are:\n\n`lid.176.bin`\n\nmodel (loaded once, not per request)For a given text, we take the top-1 prediction from each detector, and if at least three of them agree, we trust that result. If there's a tie, we fall back to the one with the highest overall confidence (from the model's probability). If the user explicitly set a source language during upload, we honor that, but we still run detection and warn if a mismatch is strong.\n\nHere's a simplified version of our detector:\n\n``` python\nimport fasttext\nimport langdetect\nimport cld3\n\nclass LanguageDetector:\n    def __init__(self):\n        self.ft_model = fasttext.load_model('lid.176.bin')\n\n    def detect(self, text, user_lang=None):\n        # Clean text: replace newlines, keep only first 10000 chars\n        clean = text.strip().replace('\\n', ' ')[:10000]\n        votes = []\n        # fastText\n        pred_ft = self.ft_model.predict(clean, k=1)\n        lang_ft = pred_ft[0][0].replace('__label__', '')\n        conf_ft = pred_ft[1][0]\n        votes.append((lang_ft, conf_ft))\n        # langdetect\n        try:\n            lang_ld = langdetect.detect(clean)\n            votes.append((lang_ld, 1.0))  # langdetect doesn't give confidence\n        except Exception:\n            votes.append(('unknown', 0.0))\n        # cld3\n        pred_cld = cld3.get_language(clean)\n        if pred_cld.is_reliable:\n            votes.append((pred_cld.language, pred_cld.probability))\n        else:\n            votes.append(('unknown', 0.0))\n\n        # Voting: count occurrences, pick max with tie-break on confidence\n        lang_counts = {}\n        for lang, conf in votes:\n            lang_counts[lang] = lang_counts.get(lang, 0) + 1\n        majority_lang = max(lang_counts, key=lambda l: (lang_counts[l], \n                            sum(c for v_l, c in votes if v_l == l)))\n        if majority_lang != 'unknown' and lang_counts[majority_lang] >= 3:\n            # High confidence, possibly override user_lang with warning\n            if user_lang and user_lang != majority_lang:\n                # log warning, but use detection\n                return majority_lang\n            return majority_lang\n        # If no consensus, fallback to user setting or the highest confidence from all\n        if user_lang:\n            return user_lang\n        # Pick the highest confidence individually\n        best = max(votes, key=lambda x: x[1])\n        return best[0]\n```\n\nIn production, we load the fastText model at startup to avoid per-request overhead. We also cache detection results per book (based on a hash of the first 10k characters plus random sampling) to speed up subsequent operations.\n\nFor books that contain multiple languages (e.g., a language textbook with side-by-side translation), we don't require detection at the book level. Instead, we first try to identify the dominant language (the one with >80% of the pages). If the user requests translation, we only translate chunks that match the source language and leave the rest untouched. This requires per-chunk language identification. Our pipeline runs detection on each chunk (around 1000 tokens) before sending it to the LLM, and skips chunks that are clearly not the target language. That prevents the LLM from halting on foreign phrases.\n\nLarge language models are great at translation, but they have token limits and cost per token. Books are long, and sending them whole would bust context windows and API budgets. We needed a chunking strategy that keeps enough context for accurate translation while minimizing token waste.\n\nWe use TikToken (OpenAI's tokenizer) to count tokens (also works for Claude's tokenizer which is similar). We set a maximum chunk size of 3,500 tokens (to leave headroom for the system prompt and response). We also maintain a 200-token overlap with the previous chunk to provide context continuity. This is crucial for sentences that span paragraph boundaries.\n\n``` python\nimport tiktoken\n\ndef chunk_text(text, max_tokens=3500, overlap=200):\n    enc = tiktoken.get_encoding(\"cl100k_base\")\n    tokens = enc.encode(text)\n    chunks = []\n    start = 0\n    while start < len(tokens):\n        end = min(start + max_tokens, len(tokens))\n        chunk = tokens[start:end]\n        # if we're not at the beginning, prepend overlap from previous chunk\n        if start > 0:\n            overlap_start = max(0, start - overlap)\n            prepend = tokens[overlap_start:start]\n            chunk = prepend + chunk\n        chunks.append(enc.decode(chunk))\n        start = end\n    return chunks\n```\n\nWe then pass each chunk to the LLM with a system prompt that includes the source and target languages, and instructions to preserve formatting (markdown, special characters). We use Anthropic's Claude 3 Haiku for cost efficiency, but fall back to Claude Sonnet for complex passages when the model returns a confidence score below a threshold.\n\nTranslating a full book might require 50+ chunks. We run these in parallel using asyncio, but we must respect API rate limits (for Anthropic, it's 5 requests per second on our tier). We implemented a token bucket rate limiter and semaphore:\n\n``` python\nimport asyncio\nfrom aiolimiter import AsyncLimiter\n\nrate_limiter = AsyncLimiter(5, 1)  # 5 requests per 1 second\nsem = asyncio.Semaphore(10)  # max 10 in-flight\n\nasync def translate_chunk(chunk, src_lang, dst_lang):\n    async with sem:\n        async with rate_limiter:\n            # Call Claude API\n            ...\n```\n\nWe also batch large books into groups of chunks and process them with a small number of workers. This keeps the API usage smooth and prevents 429 errors.\n\nEven with good language detection and chunking, LLMs sometimes produce translations that are literal and miss literary nuance. For academic books, that might be fine, but for novels, we needed a way to improve fluency. We added a post-processing step where we run each translated chunk through a second LLM call (a smaller model like Claude Haiku with a higher temperature) to \"polish\" the language only if the user selected the \"literary\" translation mode. This adds cost but significantly improves readability. We found that around 30% of users opt for literary mode.\n\nAfter deploying the multi-model language detector, our language identification accuracy on a test set of 5,000 book excerpts jumped from 85% to 98.2%. Misidentifications dropped to rare edge cases (e.g., very short texts in dialect). In production, false language detection errors fell to near zero, and user complaints about garbled translations ceased.\n\nThe chunking strategy with overlap reduced API retries due to incomplete sentences by 40%, and the literary post-polish improved average user readability ratings (from A/B tests) by 15% on fiction books.\n\nWe also saved about 20% on API costs because the better language detection meant we weren't wasting credits on incorrect translations, and the dynamic chunking allowed us to pack more text per API call.\n\nWe're still exploring: How to handle ancient languages or highly specialized jargon? Is there a way to dynamically adjust chunk size based on sentence boundaries? We'd love to hear from the community—especially anyone who's tackled translation of technical manuals or poetry.\n\n*LectuLibre's translation pipeline continues to evolve, and we're planning to open-source our language detection ensemble and curated dataset. If you're building something similar, reach out!*", "url": "https://wpnews.pro/news/how-we-built-precise-translation-and-language-identification-for-ai-book", "canonical_source": "https://dev.to/jacob_gong/how-we-built-precise-translation-and-language-identification-for-ai-book-translation-1fbd", "published_at": "2026-07-25 03:01:44+00:00", "updated_at": "2026-07-25 03:58:27.123299+00:00", "lang": "en", "topics": ["machine-learning", "natural-language-processing", "ai-products", "developer-tools"], "entities": ["LectuLibre", "fastText", "langdetect", "cld3"], "alternates": {"html": "https://wpnews.pro/news/how-we-built-precise-translation-and-language-identification-for-ai-book", "markdown": "https://wpnews.pro/news/how-we-built-precise-translation-and-language-identification-for-ai-book.md", "text": "https://wpnews.pro/news/how-we-built-precise-translation-and-language-identification-for-ai-book.txt", "jsonld": "https://wpnews.pro/news/how-we-built-precise-translation-and-language-identification-for-ai-book.jsonld"}}