AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines

Researchers introduced AutoTuneBench, a benchmark and measurement protocol that makes trust architectural for LLM agents auto-tuning GPU kernels and serving engines, after characterizing four failure modes from a four-day pilot corpus of 619 model calls. The protocol, frozen as code with test-enforced provenance, a database-level validator, anti-cheat checks outside the agent's modification surface, pre-registered readouts, and paired-seed statistics with a 5% cross-run coefficient-of-variation cap, showed the best kernel reads 10.6x against a naive baseline but 2.03x against an honest one, one configuration delivers 1.174x on one machine and 1.0049x on another, and the KernelBench Level-1 suite admits 51% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail were released as open artifacts.

arXiv:2609.18123v1 Announce Type: new Abstract: Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5\% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall 2.4840 vs 2.4957\,ms ; and the KernelBench Level-1 suite admits 51\% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus vLLM and SGLang , and its audit trail are released as open artifacts.