{"slug": "building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent", "title": "Building Real-Time AI Translation Assistance with FastAPI, Claude, and Server-Sent Events", "summary": "LectuLibre, an AI-powered book translation platform, has introduced an interactive Translation Assistance feature that streams real-time suggestions from Claude for tricky passages. The feature, built on a Python/FastAPI backend with Server-Sent Events, allows users to highlight text and receive alternative translations, explanations, and stylistic notes with surrounding context. The engineering team detailed the architecture, including chunked text storage in PostgreSQL and streaming responses from Claude's API.", "body_md": "*How we added an on-demand translation help feature to our book translation platform, streaming LLM suggestions for tricky passages.*\n\nAt LectuLibre, our AI-powered book translation service allows users to upload EPUB or PDF files and get translations generated by large language models like Claude and DeepSeek. But we quickly noticed a pain point: automated translations, while fast, sometimes produced awkward or ambiguous results for culturally specific phrases, idioms, or technical jargon. Users wanted a way to get instant, contextual help for these tricky passages without leaving the platform. That’s when we set out to build the **翻译与转录求助** (Translation Assistance) feature — an interactive side panel where users can select any sentence or paragraph and receive alternative translations, explanations, and stylistic suggestions from an LLM in real time.\n\nIn this article, I’ll walk you through the engineering challenge, the architecture we chose, and the specific code and trade-offs that made it work smoothly under production constraints.\n\nThe core requirement was simple: a user highlights a piece of text in the translated book and clicks “Get Assistance”. Immediately, the system should stream back multiple translation options, a brief explanation of differences, and stylistic notes — all aware of the surrounding context, the author’s style, and the target language.\n\nUnder the hood, this meant:\n\nWe run a Python/FastAPI backend on a VPS, with the existing translation pipeline making batch calls to Claude’s API for full-book translation. For the assistance feature, we built a separate endpoint that accepts a book ID, a start/end offset, and streams back assistance via Server-Sent Events (SSE). The flow:\n\n`messages.create(stream=True)`\n\n) and forward each chunk as an SSE event.When a book is uploaded, we parse the EPUB/PDF, extract text, and store it as chunks (paragraphs) with their position and metadata. In PostgreSQL, we have a table `book_chunks`\n\nwith columns `book_id`\n\n, `chunk_index`\n\n, `text`\n\n, and `lang`\n\n. This allows us to quickly fetch the chunk containing the selected offset plus a few neighbors with a simple query:\n\n``` python\nasync def get_context_chunks(book_id: str, chunk_index: int, context_size: int = 2):\n    query = \"\"\"\n        SELECT chunk_index, text\n        FROM book_chunks\n        WHERE book_id = :book_id\n          AND chunk_index BETWEEN :start_idx AND :end_idx\n        ORDER BY chunk_index\n    \"\"\"\n    values = {\n        \"book_id\": book_id,\n        \"start_idx\": max(0, chunk_index - context_size),\n        \"end_idx\": chunk_index + context_size,\n    }\n    return await database.fetch_all(query, values)\n```\n\nThis gives us the surrounding text without pulling the entire book into memory.\n\nWe used FastAPI’s `StreamingResponse`\n\nwith a generator that yields SSE-formatted messages. The trick is that the Claude SDK’s `stream=True`\n\nreturns an async iterator, so we can `await`\n\neach chunk and yield it immediately:\n\n``` python\nfrom fastapi import APIRouter, Request\nfrom fastapi.responses import StreamingResponse\nimport anthropic\n\nrouter = APIRouter()\nclient = anthropic.AsyncAnthropic()\n\n@router.post(\"/assist/stream\")\nasync def stream_assistance(request: Request):\n    data = await request.json()\n    book_id = data[\"book_id\"]\n    chunk_index = data[\"chunk_index\"]\n    selected_text = data[\"selected_text\"]\n\n    context_chunks = await get_context_chunks(book_id, chunk_index)\n    context = \"\\n\".join([c[\"text\"] for c in context_chunks])\n\n    prompt = build_assistance_prompt(context, selected_text)\n\n    async def event_generator():\n        try:\n            async with client.messages.stream(\n                model=\"claude-3-5-sonnet-20240620\",\n                max_tokens=1024,\n                temperature=0.7,\n                messages=[{\"role\": \"user\", \"content\": prompt}],\n            ) as stream:\n                async for text in stream.text_stream:\n                    yield f\"data: {text}\\n\\n\"\n                yield \"data: [DONE]\\n\\n\"\n        except anthropic.APIStatusError as e:\n            yield f\"data: ERROR: {e.message}\\n\\n\"\n\n    return StreamingResponse(event_generator(), media_type=\"text/event-stream\")\n```\n\nThis gives the frontend a real-time, word-by-word streaming experience. The `event_generator`\n\nis an async generator thanks to `stream.text_stream`\n\n, which is a core feature of the Anthropic SDK.\n\nTo get useful output, we crafted a prompt that asks for three distinct translation options, each with a brief note. We also instruct the model to preserve context and provide a style guide. A simplified version:\n\n``` php\ndef build_assistance_prompt(context: str, selected: str) -> str:\n    return f\"\"\"You are a professional literary translator. Given the following context from a book:\n\n{context}\n\nThe user has selected this passage for translation help:\n\"{selected}\"\n\nProvide:\n1. Three alternative translations into the target language, labeled Option A, B, C.\n2. For each option, a one-sentence explanation of the stylistic or semantic choice.\n3. A final recommendation with justification.\n\nRespond in JSON format with keys: options (list of dicts with translation, explanation), recommendation.\n\"\"\"\n```\n\nWe then parse the JSON on the frontend to display the alternatives cleanly. The streaming text is accumulated and displayed as it arrives, giving the impression of immediate response.\n\nStreaming helps perceived performance, but the time to first token is still around 1-2 seconds for a 150-token context. We considered pre-fetching context embeddings to speed up retrieval, but the overhead of the LLM call dominates. So we optimized by:\n\n`async`\n\nSDK to avoid blocking the event loop.`max_tokens`\n\n(1024) because the response is short.Each assistance call burns tokens. With Claude 3.5 Sonnet, input tokens are $3/MTok, output $15/MTok. We track usage per user and implemented a daily limit of 50 assistance requests. If a user exceeds it, they get a polite message. This was done using a simple rate limiter with Redis:\n\n``` python\nimport aioredis\nimport time\n\nredis = aioredis.from_url(\"redis://localhost\")\n\nasync def check_rate_limit(user_id: str, limit: int = 50, window: int = 86400) -> bool:\n    key = f\"assist_rate:{user_id}\"\n    current = await redis.incr(key)\n    if current == 1:\n        await redis.expire(key, window)\n    return current <= limit\n```\n\nWe initially used DeepSeek because it’s cheaper, but the translation quality for literary nuance was noticeably lower, and the streaming API was less stable. Claude’s output was richer, so we settled on that despite the cost. For non-literary works, we might offer a toggle to DeepSeek.\n\nWe limited context to 3 paragraphs (roughly 500 tokens) to stay well within the model’s 200K context window while keeping prompt costs low. That’s usually enough for the model to understand tone and style.\n\nAfter deploying, the “翻译与转录求助” feature quickly became one of the most used parts of LectuLibre. Users loved the immediacy and the ability to see multiple translation options side by side. Average response time from click to first token is 1.4 seconds, and a complete response (3 options) takes about 5-7 seconds, which users found acceptable.\n\nKey takeaways:\n\n`asyncio`\n\nand async SDKs prevents thread exhaustion under load.We’re considering adding user feedback to fine-tune the assistance model, and possibly exposing the feature as a standalone API for translators. We’d love to hear how other teams handle real-time LLM streaming at scale — especially around batching requests and reducing token costs. Drop your thoughts in the comments!", "url": "https://wpnews.pro/news/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent", "canonical_source": "https://dev.to/jacob_gong/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent-events-5al", "published_at": "2026-08-01 03:02:37+00:00", "updated_at": "2026-08-01 03:38:23.421128+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "developer-tools", "ai-products"], "entities": ["LectuLibre", "Claude", "FastAPI", "DeepSeek", "Anthropic", "PostgreSQL", "Server-Sent Events"], "alternates": {"html": "https://wpnews.pro/news/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent", "markdown": "https://wpnews.pro/news/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent.md", "text": "https://wpnews.pro/news/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent.txt", "jsonld": "https://wpnews.pro/news/building-real-time-ai-translation-assistance-with-fastapi-claude-and-server-sent.jsonld"}}