PTXBench: What about just CUDA-PTX?

PTXBench, a benchmark released August 17, 2026 with code on GitHub, evaluates how well frontier LLMs can write architecture-specific CUDA-PTX for H100 and B200 GPUs, finding GEMM nearly solved while attention lags. Claude Opus 4.8 reached 1.012x cuBLAS performance on Blackwell and Gemini 3.1 Pro reached 0.892x, measured against cuBLAS 13.1, cuDNN 9.20.0 and FlashInfer 0.6.14. Adapting Qwen3.6-27B with a 158-record Fixit dataset solved all five primary evaluation problems, though the model still fails on head-dimension-96 backward attention.

PTXBench: What about just CUDA-PTX? August 17, 2026 The code for PTXBench is available here https://github.com/zhang677/PTXBench . Why not directly generate CUDA-PTX? PTX is the lowest-level GPU interface that CUDA programmers can explicitly control, so generating CUDA with inline, architecture-specific PTX offers the shortest path from a new hardware feature to a working kernel. This approach has traditionally looked unattractive: PTX is difficult to program and validate, while abstractions such as Triton and CuTeDSL provide productivity and portability. Yet those abstractions must continually absorb new instructions, layouts, and synchronization mechanisms through compiler engineering. As GPU architectures evolve faster and LLMs become better at code generation and iterative repair, directly generating CUDA-PTX becomes appealing as a way to use new hardware capabilities before the higher-level software stack fully catches up. What does PTXBench ask? PTXBench asks how well current LLMs can reason about architecture-specific PTX on H100 and B200 GPUs, not merely whether they can emit a fast CUDA kernel. A model receives an architecture-specific knowledge pack, writes CUDA-PTX, and revises it over multiple turns using compilation, sanitization, correctness, and performance feedback. The evaluation framework separately checks whether the kernel is functionally correct, whether the requested instruction family actually executes at runtime, and whether the kernel is competitive with frontier libraries. This separation also points to the techniques that will matter next: execution-grounded repair, targeted post-training, runtime instruction verification, and much stronger testing infrastructure. Takeaway 1: GEMM is close; attention is not Frontier LLMs are beginning to make architecture-specific PTX work, but capability falls sharply as the workload becomes more complex. GEMM is closest to being solved: Claude Opus 4.8 reaches 1.012x cuBLAS performance on Blackwell, while Gemini 3.1 Pro reaches 0.892x. Attention remains substantially harder, especially on Blackwell, and backward attention is harder still. PTXBench measures performance against frontier libraries: cuBLAS 13.1 for GEMM, cuDNN 9.20.0 for the primary attention workloads, and FlashInfer 0.6.14 for GQA. Speedup is the reference-library latency divided by the generated kernel’s latency, so 1.0x means matching the corresponding performance baseline and values above 1.0x mean surpassing it. Figure 1 uses two different kinds of ratios. The x-axis p is a speedup threshold relative to the library baseline; the y-axis, Fast<sub p</sub , is the metric proposed by KernelBench https://github.com/ScalingIntelligence/KernelBench : the percentage of evaluated turns that produced a correct kernel with speedup greater than p . For example, Fast<sub 0.8</sub = 40% means that 40% of turns produced a correct kernel exceeding 0.8x baseline performance. The “Target inst.” rows additionally require the kernel to execute a selected target instruction at runtime. Takeaway 2: A small, well-constructed SFT dataset can pay off Specializing a model for CUDA-PTX may not require an enormous corpus. PTXBench adapts Qwen3.6-27B with Fixit examples built from the base model’s failed kernels, execution feedback, teacher repairs, and synthesized repair rationales. The smallest balanced recipe that solves all five primary evaluation problems contains only 158 records, whereas the base model produces no correct kernel on any Hopper workload in the main model comparison. The improvement is meaningful but not universal: the adapted model transfers to GEMM, all four head-dimension-64 attention tasks, both head-dimension-96 forward tasks, and—at a much lower success rate—GQA, yet still fails on head-dimension-96 backward attention. Coverage, balance, and teacher quality therefore matter at least as much as raw dataset size. Cross-language transfer is more mixed Figure 3 . On the same five Hopper workloads, using the same checkpoint to generate Triton lowers turn-level correctness relative to the base model on every workload, yet raises the best correct speedup on the causal variants from 0.238x to 0.632x for MHA-Fwd-Causal and from 0.037x to 0.331x for MHA-Bwd-Causal. The SFT recipe can therefore improve peak performance substantially even while making correct Triton kernels less likely. Takeaway 3: Abstractions still matter, but cracks are appearing Higher-level abstractions still provide a major advantage, especially on newer hardware, but that advantage is no longer universal. Under the same eight-turn refinement protocol with execution feedback, Triton reaches a higher best correct speedup than CUDA-PTX in 18 of 20 model–architecture–workload comparisons across Gemini 3.1 Pro and GPT-5.6 Sol, including all 10 Blackwell comparisons. The two exceptions are GPT-5.6 Sol on H100 causal MHA forward, where CUDA-PTX reaches 0.865x versus Triton’s 0.852x, and H100 causal MHA backward, where CUDA-PTX reaches 0.746x versus Triton’s 0.734x. The largest gaps appear in Blackwell attention: on B200 causal MHA backward, Gemini produces no correct CUDA-PTX kernel but reaches 0.437x with Triton, while GPT-5.6 Sol improves from 0.339x to 0.494x. Triton’s compiler-encoded optimizations and configuration tuning remain especially valuable on Blackwell, while the two H100 exceptions show that recent LLMs can already make direct CUDA-PTX outperform Triton in selected settings. Takeaway 4: Testing will become the bottleneck Finding kernels that are both correct and safe requires repeatedly running expensive performance measurements. PTXBench reduces this cost by leveraging the temporary locality of kernel evaluation requests for parallel agent loops. Specifically, PTXBench runs a separate host process that caches workload state such as input tensors, reference outputs, and reference latencies in GPU memory. Reusing that state improves kernel evaluation throughput by 2.24x, as Figure 5 shows. Even with reuse, however, the cached path still takes 2.72x as long as kernel execution alone. What’s Next? In the future, two priorities follow. First, optimize the checkers. NVIDIA Compute Sanitizer’s racecheck , for example, can take more than 1000x as long as native execution, while LLM serving has benefited from far greater investment. Faster incremental checks, cache-aware sanitization, and better overlap between generation and profiling would let agents learn from more execution feedback in the same time. Second, robustify the checkers. Output comparison alone cannot catch memory-safety bugs, races, asynchronous-lifetime violations, or every runtime failure.<sup 1 fn:cuda-free-async </sup Future checkers must make these properties, runtime error tracing, and an explicit unknown state first-class signals. As kernel agents explore increasingly adversarial corners of CUDA semantics, checker speed will determine how quickly they improve, and checker robustness will determine whether their apparent wins are real. Acknowledgments PTXBench is a collaborative effort by Genghan Zhang, Yixin Dong, Chengze Fan, Zhichen Zeng, Yueming Yuan, Shaowei Zhu, and Kunle Olukotun. We are grateful to members of RadixArk, the SGLang community, and the Stanford Pervasive Parallelism Lab for their technical support, insightful discussions, and help. The project received generous support from the Gemini Academic Program and the Tinker Research Grant. 1. Undefined behavior is an important caveat. One numerically correct PTXBench kernel called cudaFreeAsync on temporary storage and then enqueued another consumer of that storage on the same stream. This stream-ordered use-after-free is undefined under the CUDA runtime https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/stream-ordered-memory-allocation.html : it may crash, silently corrupt results, or appear to work. In this case, the correctness checks passed, but Nsight Compute profiling failed. We retain the raw functional-correctness result for audit but classify the kernel as a runtime error in analysis; it contributes to neither the unrestricted nor target-instruction correct-kernel count. ↩ fnref:cuda-free-async