Building Real-Time AI Translation Assistance with FastAPI, Claude, and Server-Sent Events

LectuLibre, an AI-powered book translation platform, has introduced an interactive Translation Assistance feature that streams real-time suggestions from Claude for tricky passages. The feature, built on a Python/FastAPI backend with Server-Sent Events, allows users to highlight text and receive alternative translations, explanations, and stylistic notes with surrounding context. The engineering team detailed the architecture, including chunked text storage in PostgreSQL and streaming responses from Claude's API.

How we added an on-demand translation help feature to our book translation platform, streaming LLM suggestions for tricky passages. At LectuLibre, our AI-powered book translation service allows users to upload EPUB or PDF files and get translations generated by large language models like Claude and DeepSeek. But we quickly noticed a pain point: automated translations, while fast, sometimes produced awkward or ambiguous results for culturally specific phrases, idioms, or technical jargon. Users wanted a way to get instant, contextual help for these tricky passages without leaving the platform. That’s when we set out to build the 翻译与转录求助 Translation Assistance feature — an interactive side panel where users can select any sentence or paragraph and receive alternative translations, explanations, and stylistic suggestions from an LLM in real time. In this article, I’ll walk you through the engineering challenge, the architecture we chose, and the specific code and trade-offs that made it work smoothly under production constraints. The core requirement was simple: a user highlights a piece of text in the translated book and clicks “Get Assistance”. Immediately, the system should stream back multiple translation options, a brief explanation of differences, and stylistic notes — all aware of the surrounding context, the author’s style, and the target language. Under the hood, this meant: We run a Python/FastAPI backend on a VPS, with the existing translation pipeline making batch calls to Claude’s API for full-book translation. For the assistance feature, we built a separate endpoint that accepts a book ID, a start/end offset, and streams back assistance via Server-Sent Events SSE . The flow: messages.create stream=True and forward each chunk as an SSE event.When a book is uploaded, we parse the EPUB/PDF, extract text, and store it as chunks paragraphs with their position and metadata. In PostgreSQL, we have a table book chunks with columns book id , chunk index , text , and lang . This allows us to quickly fetch the chunk containing the selected offset plus a few neighbors with a simple query: python async def get context chunks book id: str, chunk index: int, context size: int = 2 : query = """ SELECT chunk index, text FROM book chunks WHERE book id = :book id AND chunk index BETWEEN :start idx AND :end idx ORDER BY chunk index """ values = { "book id": book id, "start idx": max 0, chunk index - context size , "end idx": chunk index + context size, } return await database.fetch all query, values This gives us the surrounding text without pulling the entire book into memory. We used FastAPI’s StreamingResponse with a generator that yields SSE-formatted messages. The trick is that the Claude SDK’s stream=True returns an async iterator, so we can await each chunk and yield it immediately: python from fastapi import APIRouter, Request from fastapi.responses import StreamingResponse import anthropic router = APIRouter client = anthropic.AsyncAnthropic @router.post "/assist/stream" async def stream assistance request: Request : data = await request.json book id = data "book id" chunk index = data "chunk index" selected text = data "selected text" context chunks = await get context chunks book id, chunk index context = "\n".join c "text" for c in context chunks prompt = build assistance prompt context, selected text async def event generator : try: async with client.messages.stream model="claude-3-5-sonnet-20240620", max tokens=1024, temperature=0.7, messages= {"role": "user", "content": prompt} , as stream: async for text in stream.text stream: yield f"data: {text}\n\n" yield "data: DONE \n\n" except anthropic.APIStatusError as e: yield f"data: ERROR: {e.message}\n\n" return StreamingResponse event generator , media type="text/event-stream" This gives the frontend a real-time, word-by-word streaming experience. The event generator is an async generator thanks to stream.text stream , which is a core feature of the Anthropic SDK. To get useful output, we crafted a prompt that asks for three distinct translation options, each with a brief note. We also instruct the model to preserve context and provide a style guide. A simplified version: php def build assistance prompt context: str, selected: str - str: return f"""You are a professional literary translator. Given the following context from a book: {context} The user has selected this passage for translation help: "{selected}" Provide: 1. Three alternative translations into the target language, labeled Option A, B, C. 2. For each option, a one-sentence explanation of the stylistic or semantic choice. 3. A final recommendation with justification. Respond in JSON format with keys: options list of dicts with translation, explanation , recommendation. """ We then parse the JSON on the frontend to display the alternatives cleanly. The streaming text is accumulated and displayed as it arrives, giving the impression of immediate response. Streaming helps perceived performance, but the time to first token is still around 1-2 seconds for a 150-token context. We considered pre-fetching context embeddings to speed up retrieval, but the overhead of the LLM call dominates. So we optimized by: async SDK to avoid blocking the event loop. max tokens 1024 because the response is short.Each assistance call burns tokens. With Claude 3.5 Sonnet, input tokens are $3/MTok, output $15/MTok. We track usage per user and implemented a daily limit of 50 assistance requests. If a user exceeds it, they get a polite message. This was done using a simple rate limiter with Redis: python import aioredis import time redis = aioredis.from url "redis://localhost" async def check rate limit user id: str, limit: int = 50, window: int = 86400 - bool: key = f"assist rate:{user id}" current = await redis.incr key if current == 1: await redis.expire key, window return current <= limit We initially used DeepSeek because it’s cheaper, but the translation quality for literary nuance was noticeably lower, and the streaming API was less stable. Claude’s output was richer, so we settled on that despite the cost. For non-literary works, we might offer a toggle to DeepSeek. We limited context to 3 paragraphs roughly 500 tokens to stay well within the model’s 200K context window while keeping prompt costs low. That’s usually enough for the model to understand tone and style. After deploying, the “翻译与转录求助” feature quickly became one of the most used parts of LectuLibre. Users loved the immediacy and the ability to see multiple translation options side by side. Average response time from click to first token is 1.4 seconds, and a complete response 3 options takes about 5-7 seconds, which users found acceptable. Key takeaways: asyncio and async SDKs prevents thread exhaustion under load.We’re considering adding user feedback to fine-tune the assistance model, and possibly exposing the feature as a standalone API for translators. We’d love to hear how other teams handle real-time LLM streaming at scale — especially around batching requests and reducing token costs. Drop your thoughts in the comments