AugServe

AugServe, a new framework for optimizing large language model inference, achieves a 4.7x increase in effective throughput over vLLM and reduces time-to-first-token by 96.3% through adaptive scheduling…