RouteWise, a multi-provider LLM routing system presented by Haoran Ni of Nanjing University, reduces cost by 26.6%, mean time to first token by 58.4%, and SLO violations by 98.2% compared with OpenRou…
FleetSieve, a new profiling method for LLM serving fleets, reaches the oracle aggregate decision using 22,200 GPU-seconds, 6.9% less than uniform random profiling on a fixed H100 grid for a 31B-parame…
A benchmark study by an independent researcher found that burstier request arrivals speed up LLM inference, contradicting standard intuition. The analysis of vLLM serving shows that higher burstiness …
MorphServe, a system developed by researchers and detailed in a paper on arXiv, enables elastic model precision for bursty LLM serving by temporarily swapping low-impact FP16 layers to INT4 quantizati…
Reinforcement learning agents using REINFORCE and PPO algorithms achieved a 3.5x improvement over Round-Robin and 48% over Shortest-Queue in multi-GPU routing optimization, boosting throughput by 60% …