How We Built Precise Translation and Language Identification for AI Book Translation

LectuLibre, an AI book translation service, built a robust two-tier language identification system to solve the problem of misidentified source languages in uploaded books. The team combined fastText, langdetect, and cld3 detectors with a majority-vote algorithm, achieving high accuracy for book-level and per-chunk detection.

How we tackled 精准翻译与语言识别 precise translation and language identification for AI-powered book translation. When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation. Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits. We needed something far more robust—what we internally call 精准翻译与语言识别 precise translation and language identification . Here’s how we built it. Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language e.g., French phrases in an English novel would throw off chunk-level detection. We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation. We created a LanguageDetector class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are: lid.176.bin model loaded once, not per request For a given text, we take the top-1 prediction from each detector, and if at least three of them agree, we trust that result. If there's a tie, we fall back to the one with the highest overall confidence from the model's probability . If the user explicitly set a source language during upload, we honor that, but we still run detection and warn if a mismatch is strong. Here's a simplified version of our detector: python import fasttext import langdetect import cld3 class LanguageDetector: def init self : self.ft model = fasttext.load model 'lid.176.bin' def detect self, text, user lang=None : Clean text: replace newlines, keep only first 10000 chars clean = text.strip .replace '\n', ' ' :10000 votes = fastText pred ft = self.ft model.predict clean, k=1 lang ft = pred ft 0 0 .replace ' label ', '' conf ft = pred ft 1 0 votes.append lang ft, conf ft langdetect try: lang ld = langdetect.detect clean votes.append lang ld, 1.0 langdetect doesn't give confidence except Exception: votes.append 'unknown', 0.0 cld3 pred cld = cld3.get language clean if pred cld.is reliable: votes.append pred cld.language, pred cld.probability else: votes.append 'unknown', 0.0 Voting: count occurrences, pick max with tie-break on confidence lang counts = {} for lang, conf in votes: lang counts lang = lang counts.get lang, 0 + 1 majority lang = max lang counts, key=lambda l: lang counts l , sum c for v l, c in votes if v l == l if majority lang = 'unknown' and lang counts majority lang = 3: High confidence, possibly override user lang with warning if user lang and user lang = majority lang: log warning, but use detection return majority lang return majority lang If no consensus, fallback to user setting or the highest confidence from all if user lang: return user lang Pick the highest confidence individually best = max votes, key=lambda x: x 1 return best 0 In production, we load the fastText model at startup to avoid per-request overhead. We also cache detection results per book based on a hash of the first 10k characters plus random sampling to speed up subsequent operations. For books that contain multiple languages e.g., a language textbook with side-by-side translation , we don't require detection at the book level. Instead, we first try to identify the dominant language the one with 80% of the pages . If the user requests translation, we only translate chunks that match the source language and leave the rest untouched. This requires per-chunk language identification. Our pipeline runs detection on each chunk around 1000 tokens before sending it to the LLM, and skips chunks that are clearly not the target language. That prevents the LLM from halting on foreign phrases. Large language models are great at translation, but they have token limits and cost per token. Books are long, and sending them whole would bust context windows and API budgets. We needed a chunking strategy that keeps enough context for accurate translation while minimizing token waste. We use TikToken OpenAI's tokenizer to count tokens also works for Claude's tokenizer which is similar . We set a maximum chunk size of 3,500 tokens to leave headroom for the system prompt and response . We also maintain a 200-token overlap with the previous chunk to provide context continuity. This is crucial for sentences that span paragraph boundaries. python import tiktoken def chunk text text, max tokens=3500, overlap=200 : enc = tiktoken.get encoding "cl100k base" tokens = enc.encode text chunks = start = 0 while start < len tokens : end = min start + max tokens, len tokens chunk = tokens start:end if we're not at the beginning, prepend overlap from previous chunk if start 0: overlap start = max 0, start - overlap prepend = tokens overlap start:start chunk = prepend + chunk chunks.append enc.decode chunk start = end return chunks We then pass each chunk to the LLM with a system prompt that includes the source and target languages, and instructions to preserve formatting markdown, special characters . We use Anthropic's Claude 3 Haiku for cost efficiency, but fall back to Claude Sonnet for complex passages when the model returns a confidence score below a threshold. Translating a full book might require 50+ chunks. We run these in parallel using asyncio, but we must respect API rate limits for Anthropic, it's 5 requests per second on our tier . We implemented a token bucket rate limiter and semaphore: python import asyncio from aiolimiter import AsyncLimiter rate limiter = AsyncLimiter 5, 1 5 requests per 1 second sem = asyncio.Semaphore 10 max 10 in-flight async def translate chunk chunk, src lang, dst lang : async with sem: async with rate limiter: Call Claude API ... We also batch large books into groups of chunks and process them with a small number of workers. This keeps the API usage smooth and prevents 429 errors. Even with good language detection and chunking, LLMs sometimes produce translations that are literal and miss literary nuance. For academic books, that might be fine, but for novels, we needed a way to improve fluency. We added a post-processing step where we run each translated chunk through a second LLM call a smaller model like Claude Haiku with a higher temperature to "polish" the language only if the user selected the "literary" translation mode. This adds cost but significantly improves readability. We found that around 30% of users opt for literary mode. After deploying the multi-model language detector, our language identification accuracy on a test set of 5,000 book excerpts jumped from 85% to 98.2%. Misidentifications dropped to rare edge cases e.g., very short texts in dialect . In production, false language detection errors fell to near zero, and user complaints about garbled translations ceased. The chunking strategy with overlap reduced API retries due to incomplete sentences by 40%, and the literary post-polish improved average user readability ratings from A/B tests by 15% on fiction books. We also saved about 20% on API costs because the better language detection meant we weren't wasting credits on incorrect translations, and the dynamic chunking allowed us to pack more text per API call. We're still exploring: How to handle ancient languages or highly specialized jargon? Is there a way to dynamically adjust chunk size based on sentence boundaries? We'd love to hear from the community—especially anyone who's tackled translation of technical manuals or poetry. LectuLibre's translation pipeline continues to evolve, and we're planning to open-source our language detection ensemble and curated dataset. If you're building something similar, reach out