Translating Full Books with LLMs: Our Chunking Strategy for Long-Form Context

The LectuLibre team built a token-based chunking pipeline for translating full-length books with large language models, using Python, FastAPI, and Anthropic's Claude alongside DeepSeek for simpler passages. The system splits text into overlapping token windows with tiktoken, applies paragraph-aware boundary adjustments, and maintains a PostgreSQL-backed glossary and chapter summaries to preserve terminology, character voice, and style across chapters.

How we built a pipeline that preserves context across 100k+ token books using Python, FastAPI, and Claude's context window. When we started building LectuLibre, our AI-powered book translation service, we assumed we could just send an entire book to a large language model and get a translation back. After all, Claude 3.5 Sonnet advertises a 200k token context window. But a typical novel is 80,000–120,000 words, which is roughly 100,000–150,000 tokens. That's technically within limits, but... So we needed a chunking strategy that preserves cross-chapter context: terminology, character voice, consistent style. We split the book into overlapping chunks, translate each with a context buffer containing: The pipeline: We tried LangChain's RecursiveCharacterTextSplitter first, but it works on characters, not tokens, so some chunks exceeded the model's token limit. We switched to tiktoken for exact token counting. python import tiktoken from typing import List, Tuple def chunk text by tokens text: str, target tokens: int = 3000, overlap tokens: int = 500, model: str = "claude-3-5-sonnet-20240620" - List str : Use cl100k base for Claude? Claude uses its own tokenizer but tiktoken's cl100k base approximates well enough for chunk sizing. enc = tiktoken.get encoding "cl100k base" tokens = enc.encode text chunks = start = 0 while start < len tokens : end = min start + target tokens, len tokens chunk tokens = tokens start:end chunks.append enc.decode chunk tokens if end == len tokens : break Move start back by overlap, but not before current start start = max start + target tokens - overlap tokens, start + 1 return chunks But books have paragraphs/chapters; we don't want chunks to split mid-sentence. We later added a paragraph-aware post-processing that adjusts boundaries to the nearest paragraph break within the token window. That improved translation quality. We use Anthropic's Python SDK for Claude, and DeepSeek as a cheaper alternative for simpler passages. Here's the core translation function: python import anthropic from tenacity import retry, stop after attempt, wait exponential client = anthropic.Anthropic api key="..." SYSTEM PROMPT TEMPLATE = """You are a professional literary translator. Translate the following text from {source lang} to {target lang}. Preserve the author's style, tone, and voice. Context from previous chapters: {chapter summary} Glossary of established terms and names use exactly these translations : {glossary} Translate only the provided text. Do not add explanations. """ @retry stop=stop after attempt 3 , wait=wait exponential multiplier=1, min=2, max=10 def translate chunk chunk text: str, source lang: str, target lang: str, glossary: dict, chapter summary: str, model: str = "claude-3-5-sonnet-20240620" - str: glossary str = "\n".join f"{k} - {v}" for k, v in glossary.items prompt = SYSTEM PROMPT TEMPLATE.format source lang=source lang, target lang=target lang, chapter summary=chapter summary, glossary=glossary str response = client.messages.create model=model, max tokens=4096, temperature=0.3, system=prompt, messages= {"role": "user", "content": chunk text} return response.content 0 .text We use tenacity for retries due to occasional API timeouts. The glossary is a simple Python dict stored as JSONB in PostgreSQL via SQLAlchemy. We update it after each chunk: python def update glossary session, glossary, source chunk, translated chunk : Simplified extraction: ask LLM to list new terms extraction prompt = f"Extract proper nouns, technical terms, and repeated phrases from this translated text. Return a JSON list of objects with 'source' and 'translation'." ... call Claude or DeepSeek ... new terms = extract terms translated chunk for term in new terms: glossary term 'source' = term 'translation' session.execute text "UPDATE books SET glossary = :glossary WHERE id = :book id" , {"glossary": json.dumps glossary , "book id": book id} We maintain a chapter summary by asking the LLM to summarize the translated chunk in 2-3 sentences, then concatenate with previous summary but keep it under 500 tokens. For a 120,000-word novel ≈150k tokens , our pipeline creates about 60 chunks of 3000 tokens with 500 overlap. Translation using Claude 3.5 Sonnet costs around $12–15 per book, and takes about 20–30 minutes. Before chunking, a single-shot translation often produced inconsistencies e.g., a character's name changed midway and occasionally timed out. Chunking reduced errors significantly; we measured a 40% drop in user-reported translation errors. But it wasn't perfect. Early on we set overlap to 100 tokens about 3% , and translators noticed context breaks at chunk boundaries: a sentence referencing "the previous chapter's event" would be mistranslated because the LLM didn't have that context. We increased overlap to 500 tokens 16-17% of chunk size , which helped. Another failure: we initially included the full glossary plus all previous summaries in every prompt, which bloated the context and sometimes confused the model. We now cap the glossary to the 50 most recent terms and keep the summary under 500 tokens. tiktoken or the model's tokenizer. We're exploring hierarchical translation: first translate chapter summaries, then use those as context for translating full chapters. That might improve coherence for very long books with complex plots. Takeaway: Translating entire books with LLMs is feasible if you treat it as a context-management problem, not a single-prompt problem. Chunking with overlap, glossary, and summary bridges the gap between the model's context window and a book's length. Open question for the community: How do you handle idiomatic expressions or cultural references that need adaptation rather than literal translation? Any techniques beyond a glossary?