{"slug": "streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep", "title": "Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s", "summary": "A developer benchmarked a FastAPI SSE endpoint that streams retrieved RAG sources as the first event before LLM tokens, measuring a 52 ms first byte versus 1,664 ms for a non-streaming JSON response, with total latency unchanged at roughly 1,670 ms. The tests, run on a 2 vCPU GitHub runner with simulated retrieval and token timing, found that a blocking time.sleep call inside the async generator pushed median first byte to 3,280 ms with 5 concurrent clients, while GZipMiddleware left text/event-stream untouched at 52 ms.", "body_md": "A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?\n\n**Short answer:** send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an `async` generator: with 5 clients the median first byte rose to 3,280 ms.\n\nTested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.\n\nThe endpoint is a plain [`StreamingResponse`](https://fastapi.tiangolo.com/advanced/custom-response/) with `media_type=\"text/event-stream\"` and hand-formatted events. Recent FastAPI versions also ship a dedicated `EventSourceResponse` in [`fastapi.sse`](https://fastapi.tiangolo.com/reference/sse/); I did not test it, and the manual version works the same way on older releases.\n\nThe stream has three kinds of event:\n\n`sources`: one event with the retrieved chunks (sent right after retrieval).`token`: one event per token.` done`: a final marker so the client knows the stream ended cleanly.\nThe comparison endpoint does the same work and returns one JSON body at the end. A client script (httpx) times the first byte, the `sources` event, the first `token` event and the end. Each case ran in a real uvicorn process.\n\nAll times in milliseconds (median over runs; last column is the worst first byte).\n\n| Case | First byte | Sources | First token | Total | Worst first byte | \n|---|---|---|---|---|---|\n| Non-streaming JSON, 1 client | 1664 | - | - | 1664 | 1666 | \n| SSE sources-first, 1 client | 52 | 52 | 453 | 1673 | 53 | \n| SSE sources-first, 20 concurrent | 71 | 71 | 472 | 1715 | 86 | \n| SSE, 5 concurrent (control) | 54 | 54 | 455 | 1685 | 56 | \n| SSE + blocking `time.sleep` , 5 concurrent | 3280 | 3280 | 3681 | 4894 | 6510 | \n| SSE + `GZipMiddleware` , 1 client | 52 | 52 | 453 | 1673 | 53 | \n\n`time.sleep` variant does the same simulated work with a blocking call. With 5 clients the median first byte was 3,280 ms, about 61 times the control (54 ms), and the worst was 6,510 ms. One client's stream stalls everyone else's. In real code the culprit is a sync SDK call, a CPU-heavy rerank or a blocking DB driver inside `async def`. Use the async client, or run sync work in a thread (` run_in_threadpool`, `asyncio.to_thread`).` GZipMiddleware` leaves `text/event-stream` alone, and all 62 events arrived as separate chunks, with first byte unchanged at 52 ms. Other middleware, or a proxy, may behave differently.`X-Accel-Buffering: no` header below asks it not to, but I did not test it.\n\n``` python\nimport json\nfrom fastapi import FastAPI\nfrom fastapi.responses import StreamingResponse\n\napp = FastAPI()\n\ndef sse(event: str, data) -> bytes:\n    return f\"event: {event}\\ndata: {json.dumps(data)}\\n\\n\".encode()\n\n@app.get(\"/ask\")\nasync def ask(q: str):\n    async def events():\n        chunks = await retrieve(q)             # async vector search\n        yield sse(\"sources\", chunks)           # evidence first\n        async for token in llm_stream(q, chunks):  # async client, never blocking\n            yield sse(\"token\", token)\n        yield sse(\"done\", {})\n    return StreamingResponse(\n        events(),\n        media_type=\"text/event-stream\",\n        headers={\"Cache-Control\": \"no-cache\", \"X-Accel-Buffering\": \"no\"},\n    )\n```\n\n`retrieve` and `llm_stream` are your own functions. The full test server and benchmark script are on the [original post](https://recallrun.dev/posts/fastapi-sse-streaming-llm/).\n\n`await`-able or pushed to a thread.\nIf you want to check the sources the model cites afterwards, I wrote up [validating LLM citations in FastAPI](https://recallrun.dev/posts/validate-llm-citations-fastapi/).", "url": "https://wpnews.pro/news/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep", "canonical_source": "https://dev.to/recallrundev/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-timesleep-that-pushed-it-to-528n", "published_at": "2026-11-06 15:49:36+00:00", "updated_at": "2026-11-06 15:59:34.819665+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-agents"], "entities": ["FastAPI", "Starlette", "uvicorn", "httpx", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep", "markdown": "https://wpnews.pro/news/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep.md", "text": "https://wpnews.pro/news/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep.txt", "jsonld": "https://wpnews.pro/news/streaming-llm-answers-from-fastapi-with-sse-sources-at-52-ms-and-one-time-sleep.jsonld"}}