Just built some FastAPI SSE backend streaming LLM responses token-by-token,
after i got some free time.
Two things broke in production that worked fine locally 🙃:
nginx buffers SSE by default. The stream was delivering locally, dead
silent on Railway.proxy_buffering
off, two hours later, fixed.The LLM API returns an
empty_retrieval
event when no docs match. I wasn't
handling it, so the frontend sat on a spinner indefinitely. Added
the handler, resolved.
source & further reading
dev.to — original article
Let another AI write your teacher data
Semantic Caching in Enterprise RAG: Production Architectures for Faster, Lower-Cost LLM Systems
The AI that was smart on the bench suddenly started making mistakes in production