A RAG answer takes seconds to finish. Users judge the wait by when something appears. So: what does streaming change in a FastAPI endpoint, and what breaks it?
Short answer: send the retrieved sources as the first SSE event, then the tokens. In my test the first byte arrived after 52 ms instead of 1,664 ms, and the first token after 453 ms. The total time stayed the same (1,673 ms vs 1,664 ms). The most common way to break it is a blocking call inside an async generator: with 5 clients the median first byte rose to 3,280 ms.
Tested on a 2 vCPU GitHub runner (CPU only), Python 3.12, fastapi 0.161.1, starlette 1.9.0, uvicorn 0.54.0, httpx 0.28.1, real uvicorn server on localhost. Retrieval (50 ms), LLM first token (400 ms) and token rate (60 tokens x 20 ms) are simulated with asyncio.sleep, so absolute numbers are not what a real model gives.
The endpoint is a plain StreamingResponse with media_type="text/event-stream" and hand-formatted events. Recent FastAPI versions also ship a dedicated EventSourceResponse in fastapi.sse; I did not test it, and the manual version works the same way on older releases.
The stream has three kinds of event:
sources: one event with the retrieved chunks (sent right after retrieval).token: one event per token. done: a final marker so the client knows the stream ended cleanly.
The comparison endpoint does the same work and returns one JSON body at the end. A client script (httpx) times the first byte, the sources event, the first token event and the end. Each case ran in a real uvicorn process.
All times in milliseconds (median over runs; last column is the worst first byte).
| Case | First byte | Sources | First token | Total | Worst first byte |
|---|---|---|---|---|---|
| Non-streaming JSON, 1 client | 1664 | - | - | 1664 | 1666 |
| SSE sources-first, 1 client | 52 | 52 | 453 | 1673 | 53 |
| SSE sources-first, 20 concurrent | 71 | 71 | 472 | 1715 | 86 |
| SSE, 5 concurrent (control) | 54 | 54 | 455 | 1685 | 56 |
SSE + blocking time.sleep , 5 concurrent |
3280 | 3280 | 3681 | 4894 | 6510 |
SSE + GZipMiddleware , 1 client |
52 | 52 | 453 | 1673 | 53 |
time.sleep variant does the same simulated work with a blocking call. With 5 clients the median first byte was 3,280 ms, about 61 times the control (54 ms), and the worst was 6,510 ms. One client's stream stalls everyone else's. In real code the culprit is a sync SDK call, a CPU-heavy rerank or a blocking DB driver inside async def. Use the async client, or run sync work in a thread ( run_in_threadpool, asyncio.to_thread). GZipMiddleware leaves text/event-stream alone, and all 62 events arrived as separate chunks, with first byte unchanged at 52 ms. Other middleware, or a proxy, may behave differently.X-Accel-Buffering: no header below asks it not to, but I did not test it.
import json
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
app = FastAPI()
def sse(event: str, data) -> bytes:
return f"event: {event}\ndata: {json.dumps(data)}\n\n".encode()
@app.get("/ask")
async def ask(q: str):
async def events():
chunks = await retrieve(q) # async vector search
yield sse("sources", chunks) # evidence first
async for token in llm_stream(q, chunks): # async client, never blocking
yield sse("token", token)
yield sse("done", {})
return StreamingResponse(
events(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
retrieve and llm_stream are your own functions. The full test server and benchmark script are on the original post.
await-able or pushed to a thread.
If you want to check the sources the model cites afterwards, I wrote up validating LLM citations in FastAPI.