How we tackled 精准翻译与语言识别 (precise translation and language identification) for AI-powered book translation.
When we first launched LectuLibre, our AI book translation service, we thought the hardest part would be fine-tuning LLM prompts for literary quality. But we quickly discovered a more fundamental hurdle: if the source language of an uploaded book is misidentified, no amount of prompt engineering can salvage the translation.
Users upload EPUBs and PDFs from all over the world. Some contain metadata specifying the language, but many don't. Others are multilingual books, or have prefaces in a different language. Our initial language detection using Python's langdetect
library was correct only about 85% of the time on real-world uploads. That 15% error rate meant entirely garbled translations, frustrated users, and wasted LLM API credits.
We needed something far more robust—what we internally call 精准翻译与语言识别 (precise translation and language identification). Here’s how we built it.
Our first instinct was to try heavier models like fastText's pre-trained language identification model, which is known for high accuracy. But when we tested it on book excerpts, we hit a new problem: short paragraphs or dialogues in one language embedded in a book of another language (e.g., French phrases in an English novel) would throw off chunk-level detection.
We realized that we needed a two-tier approach: book-level language detection with confidence scoring, and per-chunk verification before translation.
We created a LanguageDetector
class that runs several detectors and picks the majority vote, with a fallback to user-specified language when available. The detectors we use are:
lid.176.bin
model (loaded once, not per request)For a given text, we take the top-1 prediction from each detector, and if at least three of them agree, we trust that result. If there's a tie, we fall back to the one with the highest overall confidence (from the model's probability). If the user explicitly set a source language during upload, we honor that, but we still run detection and warn if a mismatch is strong.
Here's a simplified version of our detector:
import fasttext
import langdetect
import cld3
class LanguageDetector:
def __init__(self):
self.ft_model = fasttext.load_model('lid.176.bin')
def detect(self, text, user_lang=None):
clean = text.strip().replace('\n', ' ')[:10000]
votes = []
pred_ft = self.ft_model.predict(clean, k=1)
lang_ft = pred_ft[0][0].replace('__label__', '')
conf_ft = pred_ft[1][0]
votes.append((lang_ft, conf_ft))
try:
lang_ld = langdetect.detect(clean)
votes.append((lang_ld, 1.0)) # langdetect doesn't give confidence
except Exception:
votes.append(('unknown', 0.0))
pred_cld = cld3.get_language(clean)
if pred_cld.is_reliable:
votes.append((pred_cld.language, pred_cld.probability))
else:
votes.append(('unknown', 0.0))
lang_counts = {}
for lang, conf in votes:
lang_counts[lang] = lang_counts.get(lang, 0) + 1
majority_lang = max(lang_counts, key=lambda l: (lang_counts[l],
sum(c for v_l, c in votes if v_l == l)))
if majority_lang != 'unknown' and lang_counts[majority_lang] >= 3:
if user_lang and user_lang != majority_lang:
return majority_lang
return majority_lang
if user_lang:
return user_lang
best = max(votes, key=lambda x: x[1])
return best[0]
In production, we load the fastText model at startup to avoid per-request overhead. We also cache detection results per book (based on a hash of the first 10k characters plus random sampling) to speed up subsequent operations.
For books that contain multiple languages (e.g., a language textbook with side-by-side translation), we don't require detection at the book level. Instead, we first try to identify the dominant language (the one with >80% of the pages). If the user requests translation, we only translate chunks that match the source language and leave the rest untouched. This requires per-chunk language identification. Our pipeline runs detection on each chunk (around 1000 tokens) before sending it to the LLM, and skips chunks that are clearly not the target language. That prevents the LLM from halting on foreign phrases.
Large language models are great at translation, but they have token limits and cost per token. Books are long, and sending them whole would bust context windows and API budgets. We needed a chunking strategy that keeps enough context for accurate translation while minimizing token waste.
We use TikToken (OpenAI's tokenizer) to count tokens (also works for Claude's tokenizer which is similar). We set a maximum chunk size of 3,500 tokens (to leave headroom for the system prompt and response). We also maintain a 200-token overlap with the previous chunk to provide context continuity. This is crucial for sentences that span paragraph boundaries.
import tiktoken
def chunk_text(text, max_tokens=3500, overlap=200):
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode(text)
chunks = []
start = 0
while start < len(tokens):
end = min(start + max_tokens, len(tokens))
chunk = tokens[start:end]
if start > 0:
overlap_start = max(0, start - overlap)
prepend = tokens[overlap_start:start]
chunk = prepend + chunk
chunks.append(enc.decode(chunk))
start = end
return chunks
We then pass each chunk to the LLM with a system prompt that includes the source and target languages, and instructions to preserve formatting (markdown, special characters). We use Anthropic's Claude 3 Haiku for cost efficiency, but fall back to Claude Sonnet for complex passages when the model returns a confidence score below a threshold.
Translating a full book might require 50+ chunks. We run these in parallel using asyncio, but we must respect API rate limits (for Anthropic, it's 5 requests per second on our tier). We implemented a token bucket rate limiter and semaphore:
import asyncio
from aiolimiter import AsyncLimiter
rate_limiter = AsyncLimiter(5, 1) # 5 requests per 1 second
sem = asyncio.Semaphore(10) # max 10 in-flight
async def translate_chunk(chunk, src_lang, dst_lang):
async with sem:
async with rate_limiter:
...
We also batch large books into groups of chunks and process them with a small number of workers. This keeps the API usage smooth and prevents 429 errors.
Even with good language detection and chunking, LLMs sometimes produce translations that are literal and miss literary nuance. For academic books, that might be fine, but for novels, we needed a way to improve fluency. We added a post-processing step where we run each translated chunk through a second LLM call (a smaller model like Claude Haiku with a higher temperature) to "polish" the language only if the user selected the "literary" translation mode. This adds cost but significantly improves readability. We found that around 30% of users opt for literary mode.
After deploying the multi-model language detector, our language identification accuracy on a test set of 5,000 book excerpts jumped from 85% to 98.2%. Misidentifications dropped to rare edge cases (e.g., very short texts in dialect). In production, false language detection errors fell to near zero, and user complaints about garbled translations ceased.
The chunking strategy with overlap reduced API retries due to incomplete sentences by 40%, and the literary post-polish improved average user readability ratings (from A/B tests) by 15% on fiction books.
We also saved about 20% on API costs because the better language detection meant we weren't wasting credits on incorrect translations, and the dynamic chunking allowed us to pack more text per API call.
We're still exploring: How to handle ancient languages or highly specialized jargon? Is there a way to dynamically adjust chunk size based on sentence boundaries? We'd love to hear from the community—especially anyone who's tackled translation of technical manuals or poetry.
LectuLibre's translation pipeline continues to evolve, and we're planning to open-source our language detection ensemble and curated dataset. If you're building something similar, reach out!