TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

A research paper titled "TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving" proposes a joint scheduling approach for agentic LLM services that alternate between GPU model inference and distributed CPU tool execution. The authors argue existing systems overlook tool-return readiness in service evaluation, wasting limited resources, and rely on heuristic strategies to decide admission and placement. The work targets scheduling efficiency for LLM-based agent serving across clustered GPUs and external tools.

Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems 1 overlook tool-return readiness in service evaluation, wasting limited resources, and 2 use heuristic strategies to decide admission and placement on G