How we added an on-demand translation help feature to our book translation platform, streaming LLM suggestions for tricky passages.
At LectuLibre, our AI-powered book translation service allows users to upload EPUB or PDF files and get translations generated by large language models like Claude and DeepSeek. But we quickly noticed a pain point: automated translations, while fast, sometimes produced awkward or ambiguous results for culturally specific phrases, idioms, or technical jargon. Users wanted a way to get instant, contextual help for these tricky passages without leaving the platform. That’s when we set out to build the 翻译与转录求助 (Translation Assistance) feature — an interactive side panel where users can select any sentence or paragraph and receive alternative translations, explanations, and stylistic suggestions from an LLM in real time.
In this article, I’ll walk you through the engineering challenge, the architecture we chose, and the specific code and trade-offs that made it work smoothly under production constraints.
The core requirement was simple: a user highlights a piece of text in the translated book and clicks “Get Assistance”. Immediately, the system should stream back multiple translation options, a brief explanation of differences, and stylistic notes — all aware of the surrounding context, the author’s style, and the target language.
Under the hood, this meant:
We run a Python/FastAPI backend on a VPS, with the existing translation pipeline making batch calls to Claude’s API for full-book translation. For the assistance feature, we built a separate endpoint that accepts a book ID, a start/end offset, and streams back assistance via Server-Sent Events (SSE). The flow:
messages.create(stream=True)
) and forward each chunk as an SSE event.When a book is uploaded, we parse the EPUB/PDF, extract text, and store it as chunks (paragraphs) with their position and metadata. In PostgreSQL, we have a table book_chunks
with columns book_id
, chunk_index
, text
, and lang
. This allows us to quickly fetch the chunk containing the selected offset plus a few neighbors with a simple query:
async def get_context_chunks(book_id: str, chunk_index: int, context_size: int = 2):
query = """
SELECT chunk_index, text
FROM book_chunks
WHERE book_id = :book_id
AND chunk_index BETWEEN :start_idx AND :end_idx
ORDER BY chunk_index
"""
values = {
"book_id": book_id,
"start_idx": max(0, chunk_index - context_size),
"end_idx": chunk_index + context_size,
}
return await database.fetch_all(query, values)
This gives us the surrounding text without pulling the entire book into memory.
We used FastAPI’s StreamingResponse
with a generator that yields SSE-formatted messages. The trick is that the Claude SDK’s stream=True
returns an async iterator, so we can await
each chunk and yield it immediately:
from fastapi import APIRouter, Request
from fastapi.responses import StreamingResponse
import anthropic
router = APIRouter()
client = anthropic.AsyncAnthropic()
@router.post("/assist/stream")
async def stream_assistance(request: Request):
data = await request.json()
book_id = data["book_id"]
chunk_index = data["chunk_index"]
selected_text = data["selected_text"]
context_chunks = await get_context_chunks(book_id, chunk_index)
context = "\n".join([c["text"] for c in context_chunks])
prompt = build_assistance_prompt(context, selected_text)
async def event_generator():
try:
async with client.messages.stream(
model="claude-3-5-sonnet-20240620",
max_tokens=1024,
temperature=0.7,
messages=[{"role": "user", "content": prompt}],
) as stream:
async for text in stream.text_stream:
yield f"data: {text}\n\n"
yield "data: [DONE]\n\n"
except anthropic.APIStatusError as e:
yield f"data: ERROR: {e.message}\n\n"
return StreamingResponse(event_generator(), media_type="text/event-stream")
This gives the frontend a real-time, word-by-word streaming experience. The event_generator
is an async generator thanks to stream.text_stream
, which is a core feature of the Anthropic SDK.
To get useful output, we crafted a prompt that asks for three distinct translation options, each with a brief note. We also instruct the model to preserve context and provide a style guide. A simplified version:
def build_assistance_prompt(context: str, selected: str) -> str:
return f"""You are a professional literary translator. Given the following context from a book:
{context}
The user has selected this passage for translation help:
"{selected}"
Provide:
1. Three alternative translations into the target language, labeled Option A, B, C.
2. For each option, a one-sentence explanation of the stylistic or semantic choice.
3. A final recommendation with justification.
Respond in JSON format with keys: options (list of dicts with translation, explanation), recommendation.
"""
We then parse the JSON on the frontend to display the alternatives cleanly. The streaming text is accumulated and displayed as it arrives, giving the impression of immediate response.
Streaming helps perceived performance, but the time to first token is still around 1-2 seconds for a 150-token context. We considered pre-fetching context embeddings to speed up retrieval, but the overhead of the LLM call dominates. So we optimized by:
async
SDK to avoid blocking the event loop.max_tokens
(1024) because the response is short.Each assistance call burns tokens. With Claude 3.5 Sonnet, input tokens are $3/MTok, output $15/MTok. We track usage per user and implemented a daily limit of 50 assistance requests. If a user exceeds it, they get a polite message. This was done using a simple rate limiter with Redis:
import aioredis
import time
redis = aioredis.from_url("redis://localhost")
async def check_rate_limit(user_id: str, limit: int = 50, window: int = 86400) -> bool:
key = f"assist_rate:{user_id}"
current = await redis.incr(key)
if current == 1:
await redis.expire(key, window)
return current <= limit
We initially used DeepSeek because it’s cheaper, but the translation quality for literary nuance was noticeably lower, and the streaming API was less stable. Claude’s output was richer, so we settled on that despite the cost. For non-literary works, we might offer a toggle to DeepSeek.
We limited context to 3 paragraphs (roughly 500 tokens) to stay well within the model’s 200K context window while keeping prompt costs low. That’s usually enough for the model to understand tone and style.
After deploying, the “翻译与转录求助” feature quickly became one of the most used parts of LectuLibre. Users loved the immediacy and the ability to see multiple translation options side by side. Average response time from click to first token is 1.4 seconds, and a complete response (3 options) takes about 5-7 seconds, which users found acceptable.
Key takeaways:
asyncio
and async SDKs prevents thread exhaustion under load.We’re considering adding user feedback to fine-tune the assistance model, and possibly exposing the feature as a standalone API for translators. We’d love to hear how other teams handle real-time LLM streaming at scale — especially around batching requests and reducing token costs. Drop your thoughts in the comments!