How we built a pipeline that preserves context across 100k+ token books using Python, FastAPI, and Claude's context window.
When we started building LectuLibre, our AI-powered book translation service, we assumed we could just send an entire book to a large language model and get a translation back. After all, Claude 3.5 Sonnet advertises a 200k token context window. But a typical novel is 80,000–120,000 words, which is roughly 100,000–150,000 tokens. That's technically within limits, but...
So we needed a chunking strategy that preserves cross-chapter context: terminology, character voice, consistent style.
We split the book into overlapping chunks, translate each with a context buffer containing:
The pipeline:
We tried LangChain's RecursiveCharacterTextSplitter first, but it works on characters, not tokens, so some chunks exceeded the model's token limit. We switched to tiktoken for exact token counting.
import tiktoken
from typing import List, Tuple
def chunk_text_by_tokens(
text: str,
target_tokens: int = 3000,
overlap_tokens: int = 500,
model: str = "claude-3-5-sonnet-20240620"
) -> List[str]:
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode(text)
chunks = []
start = 0
while start < len(tokens):
end = min(start + target_tokens, len(tokens))
chunk_tokens = tokens[start:end]
chunks.append(enc.decode(chunk_tokens))
if end == len(tokens):
break
start = max(start + target_tokens - overlap_tokens, start + 1)
return chunks
But books have paragraphs/chapters; we don't want chunks to split mid-sentence. We later added a paragraph-aware post-processing that adjusts boundaries to the nearest paragraph break within the token window. That improved translation quality.
We use Anthropic's Python SDK for Claude, and DeepSeek as a cheaper alternative for simpler passages. Here's the core translation function:
import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential
client = anthropic.Anthropic(api_key="...")
SYSTEM_PROMPT_TEMPLATE = """You are a professional literary translator.
Translate the following text from {source_lang} to {target_lang}.
Preserve the author's style, tone, and voice.
Context from previous chapters:
{chapter_summary}
Glossary of established terms and names (use exactly these translations):
{glossary}
Translate only the provided text. Do not add explanations.
"""
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(
chunk_text: str,
source_lang: str,
target_lang: str,
glossary: dict,
chapter_summary: str,
model: str = "claude-3-5-sonnet-20240620"
) -> str:
glossary_str = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
prompt = SYSTEM_PROMPT_TEMPLATE.format(
source_lang=source_lang,
target_lang=target_lang,
chapter_summary=chapter_summary,
glossary=glossary_str
)
response = client.messages.create(
model=model,
max_tokens=4096,
temperature=0.3,
system=prompt,
messages=[{"role": "user", "content": chunk_text}]
)
return response.content[0].text
We use tenacity for retries due to occasional API timeouts. The glossary is a simple Python dict stored as JSONB in PostgreSQL via SQLAlchemy. We update it after each chunk:
def update_glossary(session, glossary, source_chunk, translated_chunk):
extraction_prompt = f"Extract proper nouns, technical terms, and repeated phrases from this translated text. Return a JSON list of objects with 'source' and 'translation'."
new_terms = extract_terms(translated_chunk)
for term in new_terms:
glossary[term['source']] = term['translation']
session.execute(
text("UPDATE books SET glossary = :glossary WHERE id = :book_id"),
{"glossary": json.dumps(glossary), "book_id": book_id}
)
We maintain a chapter summary by asking the LLM to summarize the translated chunk in 2-3 sentences, then concatenate with previous summary but keep it under 500 tokens.
For a 120,000-word novel (≈150k tokens), our pipeline creates about 60 chunks of 3000 tokens with 500 overlap. Translation using Claude 3.5 Sonnet costs around $12–15 per book, and takes about 20–30 minutes. Before chunking, a single-shot translation often produced inconsistencies (e.g., a character's name changed midway) and occasionally timed out. Chunking reduced errors significantly; we measured a 40% drop in user-reported translation errors.
But it wasn't perfect. Early on we set overlap to 100 tokens (about 3%), and translators noticed context breaks at chunk boundaries: a sentence referencing "the previous chapter's event" would be mistranslated because the LLM didn't have that context. We increased overlap to 500 tokens (16-17% of chunk size), which helped.
Another failure: we initially included the full glossary plus all previous summaries in every prompt, which bloated the context and sometimes confused the model. We now cap the glossary to the 50 most recent terms and keep the summary under 500 tokens.
tiktoken or the model's tokenizer.
We're exploring hierarchical translation: first translate chapter summaries, then use those as context for translating full chapters. That might improve coherence for very long books with complex plots.
Takeaway: Translating entire books with LLMs is feasible if you treat it as a context-management problem, not a single-prompt problem. Chunking with overlap, glossary, and summary bridges the gap between the model's context window and a book's length.
Open question for the community: How do you handle idiomatic expressions or cultural references that need adaptation rather than literal translation? Any techniques beyond a glossary?