Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles

A new paper argues that GPU-kernel benchmarks used to grade LLM-generated code rely on a few random inputs and a loose floating-point tolerance, making their correctness verdicts unreliable even as those verdicts feed leaderboards and reinforcement-learning rewards. The work proposes mutation analysis to measure the strength of these benchmark oracles, noting that prior efforts have patched weak checkers by hand with extra input distributions and fuzzing.

Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed leaderboards and reinforcement-learning rewards. Recent work agrees these checkers are weak and patches them by hand---extra input distributions, fuz