{"slug": "are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and", "title": "Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent", "summary": "Researchers from Atrex introduced Atrex-Bench, a benchmark of 30 GPU operators and 440 shapes sampled from production inference traces, and found that even the best frontier coding agents achieve only ~10% of hardware roofline on these operators. To address this gap, they released Atrex-Kernel-Agent (AKA), a profile-driven optimization agent that converts fallback kernels into production-quality code matching hand-tuned baselines.", "body_md": "arXiv:2607.14541v1 Announce Type: new\nAbstract: Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads. We present Atrex-Bench, a benchmark whose 30 operators and 440 shapes are sampled directly from full-cluster production inference traces of compute-limited, memory-rich GPUs. Each problem carries an importance weight derived from its share of observed GPU time, weighted by application card-hours and computed separately for the serving phases in which it runs, together with a per-problem roofline ceiling, so the aggregate score emphasizes the kernels that consume the most serving time. Evaluating six frontier coding agents on Atrex-Bench shows that even the best vanilla model reaches only ${\\sim}10\\%$ of the hardware roofline on production operators; and correctness alone overstates capability, since much of the apparent pass rate comes from PyTorch fallbacks rather than kernels the model wrote. To close this gap, we co-release Atrex-Kernel-Agent (AKA), a profile-driven kernel-optimization agent that combines iterative measure-revise search, optimization dropout for escaping stalled search contexts, and a layered GPU-optimization knowledge base (298 reference-kernel files and 244 optimization-knowledge documents, plus external upstream reference projects for API/ISA lookup). In a controlled case study, the agent converts zero-FlyDSL fallbacks into real kernels that match or exceed hand-tuned production baselines.", "url": "https://wpnews.pro/news/are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and", "canonical_source": "https://www.machinebrief.com/news/are-llm-generated-gpu-kernels-production-ready-a-trace-drive-wdh6", "published_at": "2026-07-17 04:00:00+00:00", "updated_at": "2026-07-17 05:02:54.090188+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-agents", "ai-research"], "entities": ["Atrex", "Atrex-Bench", "Atrex-Kernel-Agent", "AKA", "PyTorch", "GPU"], "alternates": {"html": "https://wpnews.pro/news/are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and", "markdown": "https://wpnews.pro/news/are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and.md", "text": "https://wpnews.pro/news/are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and.txt", "jsonld": "https://wpnews.pro/news/are-llm-generated-gpu-kernels-production-ready-a-trace-driven-benchmark-and.jsonld"}}