{"slug": "autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving", "title": "AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines", "summary": "Researchers introduced AutoTuneBench, a benchmark and measurement protocol that makes trust architectural for LLM agents auto-tuning GPU kernels and serving engines, after characterizing four failure modes from a four-day pilot corpus of 619 model calls. The protocol, frozen as code with test-enforced provenance, a database-level validator, anti-cheat checks outside the agent's modification surface, pre-registered readouts, and paired-seed statistics with a 5% cross-run coefficient-of-variation cap, showed the best kernel reads 10.6x against a naive baseline but 2.03x against an honest one, one configuration delivers 1.174x on one machine and 1.0049x on another, and the KernelBench Level-1 suite admits 51% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail were released as open artifacts.", "body_md": "arXiv:2609.18123v1 Announce Type: new \nAbstract: Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5\\% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall (2.4840 vs 2.4957\\,ms); and the KernelBench Level-1 suite admits 51\\% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail are released as open artifacts.", "url": "https://wpnews.pro/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving", "canonical_source": "https://www.machinebrief.com/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-7ba3", "published_at": "2026-09-17 04:00:00+00:00", "updated_at": "2026-09-17 04:55:29.085188+00:00", "lang": "en", "topics": ["ai-research", "large-language-models", "ai-infrastructure", "mlops", "ai-agents"], "entities": ["AutoTuneBench", "vLLM", "SGLang", "KernelBench", "PyTorch"], "alternates": {"html": "https://wpnews.pro/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving", "markdown": "https://wpnews.pro/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving.md", "text": "https://wpnews.pro/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving.txt", "jsonld": "https://wpnews.pro/news/autotunebench-trustworthy-measurement-for-agent-auto-tuning-of-llm-serving.jsonld"}}