{"slug": "translating-full-books-with-llms-our-chunking-strategy-for-long-form-context", "title": "Translating Full Books with LLMs: Our Chunking Strategy for Long-Form Context", "summary": "The LectuLibre team built a token-based chunking pipeline for translating full-length books with large language models, using Python, FastAPI, and Anthropic's Claude alongside DeepSeek for simpler passages. The system splits text into overlapping token windows with tiktoken, applies paragraph-aware boundary adjustments, and maintains a PostgreSQL-backed glossary and chapter summaries to preserve terminology, character voice, and style across chapters.", "body_md": "*How we built a pipeline that preserves context across 100k+ token books using Python, FastAPI, and Claude's context window.*\n\nWhen we started building LectuLibre, our AI-powered book translation service, we assumed we could just send an entire book to a large language model and get a translation back. After all, Claude 3.5 Sonnet advertises a 200k token context window. But a typical novel is 80,000–120,000 words, which is roughly 100,000–150,000 tokens. That's technically within limits, but...\n\nSo we needed a chunking strategy that preserves cross-chapter context: terminology, character voice, consistent style.\n\nWe split the book into overlapping chunks, translate each with a context buffer containing:\n\nThe pipeline:\n\nWe tried LangChain's `RecursiveCharacterTextSplitter` first, but it works on characters, not tokens, so some chunks exceeded the model's token limit. We switched to `tiktoken` for exact token counting.\n\n``` python\nimport tiktoken\nfrom typing import List, Tuple\n\ndef chunk_text_by_tokens(\n    text: str,\n    target_tokens: int = 3000,\n    overlap_tokens: int = 500,\n    model: str = \"claude-3-5-sonnet-20240620\"\n) -> List[str]:\n    # Use cl100k_base for Claude? Claude uses its own tokenizer but tiktoken's\n    # cl100k_base approximates well enough for chunk sizing.\n    enc = tiktoken.get_encoding(\"cl100k_base\")\n    tokens = enc.encode(text)\n\n    chunks = []\n    start = 0\n    while start < len(tokens):\n        end = min(start + target_tokens, len(tokens))\n        chunk_tokens = tokens[start:end]\n        chunks.append(enc.decode(chunk_tokens))\n\n        if end == len(tokens):\n            break\n        # Move start back by overlap, but not before current start\n        start = max(start + target_tokens - overlap_tokens, start + 1)\n\n    return chunks\n```\n\nBut books have paragraphs/chapters; we don't want chunks to split mid-sentence. We later added a paragraph-aware post-processing that adjusts boundaries to the nearest paragraph break within the token window. That improved translation quality.\n\nWe use Anthropic's Python SDK for Claude, and DeepSeek as a cheaper alternative for simpler passages. Here's the core translation function:\n\n``` python\nimport anthropic\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\nclient = anthropic.Anthropic(api_key=\"...\")\n\nSYSTEM_PROMPT_TEMPLATE = \"\"\"You are a professional literary translator.\nTranslate the following text from {source_lang} to {target_lang}.\nPreserve the author's style, tone, and voice.\n\nContext from previous chapters:\n{chapter_summary}\n\nGlossary of established terms and names (use exactly these translations):\n{glossary}\n\nTranslate only the provided text. Do not add explanations.\n\"\"\"\n\n@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))\ndef translate_chunk(\n    chunk_text: str,\n    source_lang: str,\n    target_lang: str,\n    glossary: dict,\n    chapter_summary: str,\n    model: str = \"claude-3-5-sonnet-20240620\"\n) -> str:\n    glossary_str = \"\\n\".join([f\"{k} -> {v}\" for k, v in glossary.items()])\n    prompt = SYSTEM_PROMPT_TEMPLATE.format(\n        source_lang=source_lang,\n        target_lang=target_lang,\n        chapter_summary=chapter_summary,\n        glossary=glossary_str\n    )\n\n    response = client.messages.create(\n        model=model,\n        max_tokens=4096,\n        temperature=0.3,\n        system=prompt,\n        messages=[{\"role\": \"user\", \"content\": chunk_text}]\n    )\n    return response.content[0].text\n```\n\nWe use `tenacity` for retries due to occasional API timeouts. The glossary is a simple Python dict stored as JSONB in PostgreSQL via SQLAlchemy. We update it after each chunk:\n\n``` python\ndef update_glossary(session, glossary, source_chunk, translated_chunk):\n    # Simplified extraction: ask LLM to list new terms\n    extraction_prompt = f\"Extract proper nouns, technical terms, and repeated phrases from this translated text. Return a JSON list of objects with 'source' and 'translation'.\"\n    # ... call Claude or DeepSeek ...\n    new_terms = extract_terms(translated_chunk)\n    for term in new_terms:\n        glossary[term['source']] = term['translation']\n    session.execute(\n        text(\"UPDATE books SET glossary = :glossary WHERE id = :book_id\"),\n        {\"glossary\": json.dumps(glossary), \"book_id\": book_id}\n    )\n```\n\nWe maintain a chapter summary by asking the LLM to summarize the translated chunk in 2-3 sentences, then concatenate with previous summary but keep it under 500 tokens.\n\nFor a 120,000-word novel (≈150k tokens), our pipeline creates about 60 chunks of 3000 tokens with 500 overlap. Translation using Claude 3.5 Sonnet costs around $12–15 per book, and takes about 20–30 minutes. Before chunking, a single-shot translation often produced inconsistencies (e.g., a character's name changed midway) and occasionally timed out. Chunking reduced errors significantly; we measured a 40% drop in user-reported translation errors.\n\nBut it wasn't perfect. Early on we set overlap to 100 tokens (about 3%), and translators noticed context breaks at chunk boundaries: a sentence referencing \"the previous chapter's event\" would be mistranslated because the LLM didn't have that context. We increased overlap to 500 tokens (16-17% of chunk size), which helped.\n\nAnother failure: we initially included the full glossary plus all previous summaries in every prompt, which bloated the context and sometimes confused the model. We now cap the glossary to the 50 most recent terms and keep the summary under 500 tokens.\n\n`tiktoken` or the model's tokenizer.\nWe're exploring hierarchical translation: first translate chapter summaries, then use those as context for translating full chapters. That might improve coherence for very long books with complex plots.\n\n**Takeaway:** Translating entire books with LLMs is feasible if you treat it as a context-management problem, not a single-prompt problem. Chunking with overlap, glossary, and summary bridges the gap between the model's context window and a book's length.\n\n**Open question for the community:** How do you handle idiomatic expressions or cultural references that need adaptation rather than literal translation? Any techniques beyond a glossary?", "url": "https://wpnews.pro/news/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context", "canonical_source": "https://dev.to/jacob_gong/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context-2o5h", "published_at": "2026-09-19 03:01:25+00:00", "updated_at": "2026-09-19 03:24:15.446653+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-tools", "developer-tools"], "entities": ["LectuLibre", "Anthropic", "Claude 3.5 Sonnet", "DeepSeek", "LangChain", "tiktoken", "FastAPI", "PostgreSQL"], "alternates": {"html": "https://wpnews.pro/news/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context", "markdown": "https://wpnews.pro/news/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context.md", "text": "https://wpnews.pro/news/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context.txt", "jsonld": "https://wpnews.pro/news/translating-full-books-with-llms-our-chunking-strategy-for-long-form-context.jsonld"}}