Abstract #
The same open-weight large language model is increasingly available from multiple providers with very different pricing models, including on-demand APIs, fixed quotas, and limited concurrency. This creates an opportunity to reduce inference cost and latency without changing model quality—but only if routing accounts for both heterogeneous pricing and provider latencies that vary over time. Existing routers typically optimize cost or latency alone, trade cost for model quality, or consider only on-demand API pricing.
In this talk, I’ll present RouteWise, a multi-provider routing system that unifies heterogeneous pricing models and jointly optimizes cost and latency. RouteWise intelligently allocates limited subscription capacity, adapts routing to recent provider performance, and exposes a single control knob for navigating the cost–latency trade-off. Evaluated with BurstGPT and a production agentic-workload trace, RouteWise reduces cost by 26.6%, mean time to first token by 58.4%, and SLO violations by 98.2% compared with OpenRouter’s automatic routing.
Bio #
Haoran Ni is a senior undergraduate student majoring in Computer Science at Nanjing University, currently a student research intern working with Professor Juncheng Yang. He works on ML systems, including LLM routing, ML artifact storage and agent runtime systems.