13:29
2026-09-26
sference.com
ai-infrastructure
Prefill Concurrency in SGLang: Consistent TTFT Under Multi-Tenant Load
Sference implemented prefill concurrency in SGLang to improve time to first token (TTFT) in multi-tenant deployments, where stock SGLang allows only one request to be mid-chunk at a time and that requ…