SteerDistill: Quantizing LLM Agents via Closed-Loop Teacher Steering

Researchers propose SteerDistill, a method that quantizes LLM agents by using a full-precision teacher to steer a low-bit student toward environment transitions that match successful full-precision rollouts, addressing task failures caused by quantized agents overlooking task requirements or environmental conditions. The work targets the trade-off between lower inference cost from low-bit quantization and degraded task success in LLM agents.

Low-bit quantization reduces the inference cost of LLM agents but can compromise task success. Examining failed rollouts, we find that quantized agents overlook task requirements or environmental conditions, producing environment transitions that diverge from successful full-precision references. Co