Streaming LLM answers from FastAPI with SSE: sources at 52 ms, and one time.sleep that pushed it to 3.3 s

A developer benchmarked a FastAPI SSE endpoint that streams retrieved RAG sources as the first event before LLM tokens, measuring a 52 ms first byte versus 1,664 ms for a non-streaming JSON response, with total latency unchanged at roughly 1,670 ms. The tests, run on a 2 vCPU GitHub runner with simulated retrieval and token timing, found that a blocking time.sleep call inside the async generator pushed median first byte to 3,280 ms with 5 concurrent clients, while GZipMiddleware left text/event-stream untouched at 52 ms.

A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it? Short answer: send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same 1,673 ms vs 1,664 ms . The most common way to break it is a blocking call inside an async generator: with 5 clients the median first byte rose to 3,280 ms. Tested on a 2 vCPU GitHub runner CPU only , Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval 50 ms , LLM first token 400 ms and token rate 60 tokens x 20 ms are simulated with asyncio.sleep, so absolute numbers are not what a real model gives. The endpoint is a plain StreamingResponse https://fastapi.tiangolo.com/advanced/custom-response/ with media type="text/event-stream" and hand-formatted events. Recent FastAPI versions also ship a dedicated EventSourceResponse in fastapi.sse https://fastapi.tiangolo.com/reference/sse/ ; I did not test it, and the manual version works the same way on older releases. The stream has three kinds of event: sources : one event with the retrieved chunks sent right after retrieval . token : one event per token. done : a final marker so the client knows the stream ended cleanly. The comparison endpoint does the same work and returns one JSON body at the end. A client script httpx times the first byte, the sources event, the first token event and the end. Each case ran in a real uvicorn process. All times in milliseconds median over runs; last column is the worst first byte . | Case | First byte | Sources | First token | Total | Worst first byte | |---|---|---|---|---|---| | Non-streaming JSON, 1 client | 1664 | - | - | 1664 | 1666 | | SSE sources-first, 1 client | 52 | 52 | 453 | 1673 | 53 | | SSE sources-first, 20 concurrent | 71 | 71 | 472 | 1715 | 86 | | SSE, 5 concurrent control | 54 | 54 | 455 | 1685 | 56 | | SSE + blocking time.sleep , 5 concurrent | 3280 | 3280 | 3681 | 4894 | 6510 | | SSE + GZipMiddleware , 1 client | 52 | 52 | 453 | 1673 | 53 | time.sleep variant does the same simulated work with a blocking call. With 5 clients the median first byte was 3,280 ms, about 61 times the control 54 ms , and the worst was 6,510 ms. One client's stream stalls everyone else's. In real code the culprit is a sync SDK call, a CPU-heavy rerank or a blocking DB driver inside async def . Use the async client, or run sync work in a thread run in threadpool , asyncio.to thread . GZipMiddleware leaves text/event-stream alone, and all 62 events arrived as separate chunks, with first byte unchanged at 52 ms. Other middleware, or a proxy, may behave differently. X-Accel-Buffering: no header below asks it not to, but I did not test it. python import json from fastapi import FastAPI from fastapi.responses import StreamingResponse app = FastAPI def sse event: str, data - bytes: return f"event: {event}\ndata: {json.dumps data }\n\n".encode @app.get "/ask" async def ask q: str : async def events : chunks = await retrieve q async vector search yield sse "sources", chunks evidence first async for token in llm stream q, chunks : async client, never blocking yield sse "token", token yield sse "done", {} return StreamingResponse events , media type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, retrieve and llm stream are your own functions. The full test server and benchmark script are on the original post https://recallrun.dev/posts/fastapi-sse-streaming-llm/ . await -able or pushed to a thread. If you want to check the sources the model cites afterwards, I wrote up validating LLM citations in FastAPI https://recallrun.dev/posts/validate-llm-citations-fastapi/ .