{"slug": "agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on", "title": "AgentKernelArena: Benchmarking AI Coding Agents for GPU Kernel Optimization on AMD Instinct GPUs", "summary": "AMD released AgentKernelArena, an open-source benchmarking framework for AI coding agents on GPU kernel optimization tasks using AMD Instinct GPUs. In tests on a 44-task subset, AMD's GEAKv3 agent achieved average speedups of 9.04× on HIP2HIP tasks, outperforming general-purpose agents like Claude Code and Cursor. The framework provides standardized evaluation across 214 tasks to enable fair comparison and reproducible results.", "body_md": "# AgentKernelArena: Benchmarking AI Coding Agents for GPU Kernel Optimization on AMD Instinct GPUs[#](#agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on-amd-instinct-gpus)\n\nAI coding agents such as [Cursor Agent](https://cursor.com/), [Claude Code](https://www.anthropic.com/claude-code), and [OpenAI Codex](https://openai.com/codex/) are improving fast, and people increasingly trust them with specialized, high-stakes work, including GPU kernel optimization. But most of the public evidence is still a cherry-picked demo on a single kernel, not a controlled, head-to-head comparison on the same tasks, the same hardware, and the same scoring rules. On [AMD Instinct™ GPUs](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html), where every percentage point of kernel performance translates directly into training and inference cost, that gap matters.\n\n**AgentKernelArena (AKA)** closes that gap. It is an open-source benchmarking arena, built by AMD, that measures how well AI coding agents perform on real GPU kernel optimization tasks. You drop an agent into AKA — Cursor Agent, Claude Code, Codex, [GEAK](https://github.com/AMD-AGI/GEAK), a single-LLM call, or your own — and the framework runs it in an isolated workspace, on the same task definitions every other agent sees, and scores it through one shared automated pipeline: compile, correctness, performance.\n\nIn this blog we walk through how AKA works, how to run it yourself, and what we found when we benchmarked six agent/model configurations on a 44-task subset on AMD Instinct™ MI300X. The lineup includes **GEAKv3**, AMD’s in-house GPU kernel optimization agent. If you are new to the GPU side of this problem, the [AMD Instinct™ MI300X](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html) product page and the [AMD ROCm™ documentation](https://rocm.docs.amd.com/) are good starting points, and our [GEAK family blog](https://rocm.blogs.amd.com/artificial-intelligence/geak-agents-family/README.html) covers the agent side in depth.\n\n## Key Takeaways[#](#key-takeaways)\n\n**AgentKernelArena is an open-source arena** for benchmarking AI coding agents on GPU kernel optimization on AMD Instinct™ GPUs. It ships with**214 tasks** across four categories:`triton2triton`\n\n,`hip2hip`\n\n,`torch2hip`\n\n, and repository-scale tasks.**One unified evaluator scores every agent the same way**: 20 points for compilation, 100 for correctness, and 100× the measured speedup over baseline. Per-test-case metrics, medians, and P25/P75/P90 percentiles come out of the box.**We evaluated 6 agent/model configurations on a 44-task subset on MI300X.** On this subset, GEAKv3 recorded average speedups of**9.04×** on 20 HIP2HIP tasks,**2.75×** on 20 Triton2Triton tasks, and**1.20×** on 4 repository-scale rocPRIM tasks — ahead of the other five configurations in every category.**Among the frontier general-purpose agents,** Claude Code (Opus 4.6) and Cursor (Opus 4.6) reached 6.08× and 5.03× average speedup on HIP2HIP. Triton2Triton was the most challenging category for every agent we tested.**AKA is built for fair, reproducible comparison — and for A/B testing agent-side changes** such as new tools, MCP servers, or prompt tweaks. Timestamped runs and resumable evaluation let you attribute any score change to a real change in the agent.\n\nFigure 1, below, summarizes these results at a glance — the average speedup each agent recorded in each task category.\n\n*Figure 1. AgentKernelArena results at a glance: average speedup over baseline by agent and task category on AMD Instinct™ MI300X, evaluated on a representative subset of 44 tasks.*\n\n## Why a Standardized Arena for Kernel Agents?[#](#why-a-standardized-arena-for-kernel-agents)\n\nGPU kernel optimization is hard for an AI agent. A good kernel agent has to reason about tiling and block sizes, memory coalescing, occupancy, [LDS](https://rocm.docs.amd.com/projects/HIP/en/latest/) usage, asynchronous copies, MFMA/WMMA matrix operations, numerical stability, and correctness on edge-case shapes — often all at once. A flashy result on one kernel tells you very little about how the same agent will handle the next one.\n\nMost public kernel benchmarks today evaluate single LLM calls or fixed-shape correctness, not full agent workflows. They typically do not measure:\n\n**kernel-to-kernel optimization**, where the agent starts from a working baseline and has to make it faster without breaking correctness;** repository-level optimization**, where the agent has to navigate and edit a real upstream codebase;** real GPU performance on AMD Instinct™ hardware**, measured under controlled, reproducible conditions;** the effect of agent-side machinery**— prompts, tools, MCP servers, planners, profilers — on the final score.\n\nAKA closes exactly these gaps. We built it as an *arena*, not a single benchmark, so the evaluation logic, task definitions, scoring, and reporting all live *outside* the agent. New agents and new tasks plug into the same evaluator and produce comparable scores by construction.\n\n## What Is AgentKernelArena?[#](#what-is-agentkernelarena)\n\nAgentKernelArena runs AI coding agents on GPU kernel optimization tasks inside isolated, reproducible workspaces, and scores them with a single, centralized evaluator. Figure 2 shows the end-to-end pipeline.\n\n*Figure 2. AgentKernelArena end-to-end pipeline. Task definitions, cheatsheets, and agent configuration feed into a workspace-setup and baseline-measurement stage. The agent then gets a configurable number of iterations (read → edit → compile → test → profile). Once it finishes, a centralized evaluator — identical for every agent — runs compilation, correctness, and performance measurement, and computes the final score.*\n\nThe per-task loop is straightforward:\n\n**Pick an agent, a model, and a task.** AKA spins up an isolated workspace, copies in the source kernel and task scripts, and measures the baseline performance.**The agent reads the task and iterates** on the kernel inside its workspace.**The unified evaluator compiles the optimized kernel**, runs the correctness suite across every configured test shape, measures performance with warmup and benchmark iterations, and computes the final score.**AKA writes per-task results**, per-test-case speedup plots, and aggregate reports (CSV, JSON, text) into a timestamped run directory you can compare or resume later.\n\nA key design choice is the separation of *optimization* from *evaluation*: the agent only optimizes the kernel, while the framework owns all measurement. This removes a real fairness problem — otherwise every agent would time execution slightly differently and scores would stop being comparable.\n\n### Supported Agents and Models[#](#supported-agents-and-models)\n\nAKA ships with first-class integrations for a range of frontier and AMD agents:\n\n**Cursor Agent**(via the`cursor-agent`\n\nCLI)**Claude Code**(via the`claude`\n\nCLI)**OpenAI Codex**(via the`codex`\n\nCLI)**SWE-agent****GEAKv3**, AMD’s in-house GPU kernel optimization agent** Single-LLM-call agents**(for clean single-shot baselines)** Your own agent**, registered via a single launcher function\n\nYou select the model through the agent layer. The runs in this blog use Anthropic Claude Opus 4.6 and Sonnet 4.6, OpenAI GPT-5.3 Codex, and Cursor Composer 2. You can also point AKA at other models through OpenRouter or a [vLLM](https://github.com/vllm-project/vllm) endpoint.\n\n### Task Categories[#](#task-categories)\n\nAKA currently includes 214 task configurations across four categories, listed in Table 1. Together they cover the kinds of GPU kernel work an agent faces in practice — from optimizing existing [Triton](https://triton-lang.org/) and [HIP](https://rocm.docs.amd.com/projects/HIP/) kernels, to translating PyTorch operators, to editing real upstream repositories.\n\nCategory |\n# Tasks |\nDescription |\n|---|---|---|\n|\n148 |\nOptimize existing Triton kernels (e.g., vLLM, ROCmBench, TritonBench-modified). |\n|\n36 |\nOptimize HIP kernels (gpumode-style ops, MMCV-style ops). |\n|\n26 |\nConvert PyTorch operators into optimized HIP kernels. |\n|\n4 |\nOptimize kernels inside full upstream repositories such as rocPRIM. |\n\n*Table 1. AgentKernelArena task categories and counts.*\n\n### What the Agent Sees: Prompts, Cheatsheets, and Instructions[#](#what-the-agent-sees-prompts-cheatsheets-and-instructions)\n\nAKA assembles a structured prompt for every task from the same set of sections, so every agent — frontier or custom — gets the same information in the same shape. As Figure 3 illustrates, the sections cover the task-type role, the source files and target functions, a GPU architecture pre-check, the task-specific optimization instructions, a completion directive, the cheatsheet block, and the workspace path.\n\n*Figure 3. The shared prompt pipeline. Cursor Agent, Claude Code, and Codex all receive the identical prompt for a given task; only the CLI binary, its flags, and the backend model differ.*\n\nTwo pieces of this prompt are worth calling out, because they meaningfully change agent behavior:\n\n**Domain cheatsheets.** Each task type and GPU architecture can attach a verbatim cheatsheet to the prompt: an MI300X architecture guide (CDNA3 topology, memory hierarchy, MFMA instructions), a HIP best-practices document, and a Triton best-practices document. The cheatsheets live in`src/prompts/cheatsheet/`\n\nand are selected automatically from`target_gpu_model`\n\n. Adding a new GPU architecture is a small change: add an arch entry, drop in a markdown guide, and the prompt picks it up.**Task-specific instructions.** Each task’s`config.yaml`\n\ncan override the auto-generated instructions with a`prompt.instructions`\n\nfield. This is how we tell the agent exactly what to optimize — for example,*“Optimize the Triton*When omitted, AKA generates instructions automatically from the compile, correctness, and performance commands.`fused_moe_kernel`\n\nfor maximum GPU throughput; keep the`fused_moe`\n\nsignature; output must match the reference within`atol=5e-2, rtol=5e-2`\n\nfor float16.”\n\nOne more deliberate choice: `max_iterations`\n\nis delivered as a natural-language directive (“you must iterate up to 3 versions”) appended to the prompt, not as a hard cap on tool calls. Agents are free to make as many internal tool calls per version as they need. This keeps the task prompt and iteration directive consistent across different agent runtimes, while allowing each agent to use its native tool-calling workflow.\n\n### Unified Evaluator and Scoring[#](#unified-evaluator-and-scoring)\n\nThe unified evaluator does five things:\n\n**Compiles** the optimized code with the task’s declared compile command (`hipcc`\n\nfor HIP, Triton for Triton, and so on).**Runs correctness checks** on every configured test shape, not just one.**Measures performance** with configurable warmup and benchmark iterations, computes per-test-case speedup, and aggregates into median, standard deviation, and P25/P75/P90 percentiles.**Produces per-test-case speedup plots** and a structured`task_result.yaml`\n\nper task.**Writes run-level reports**(`overall_report.txt`\n\n,`overall_summary.csv`\n\n,`task_type_breakdown.json`\n\n) inside a timestamped`run_{YYYYMMDD_HHMMSS}/`\n\nworkspace.\n\nEach task contributes to a per-task score, summarized in Table 2:\n\nMetric |\nPoints |\nDescription |\n|---|---|---|\nCompilation |\n20 |\nCode compiles successfully without errors. |\nCorrectness |\n100 |\nCode produces correct output across all configured test shapes. |\nSpeedup |\nratio × 100 |\nMeasured performance improvement over the baseline kernel. |\n\n*Table 2. AgentKernelArena scoring system (default).*\n\nFigure 4, below, visualizes how these three dimensions stack into a single cumulative score.\n\n*Figure 4. The cumulative scoring system rewards each dimension of kernel quality.*\n\nA submission that compiles, passes correctness, and lands a 1.5× speedup scores **270 points** on that task. Correctness is weighted heavily on purpose: a fast but incorrect kernel is useless in practice, while a correct kernel that merely matches the baseline is still a valid replacement. If you prefer a different policy, swap the scoring function — the evaluator stays the same.\n\n## Walk Through AgentKernelArena Yourself[#](#walk-through-agentkernelarena-yourself)\n\nThe best way to understand AKA is to run it. In this section we take one real task — the vLLM [Triton](https://triton-lang.org/) RMS-norm kernel — all the way from its baseline to an agent-optimized version, and read the score the evaluator produces. Everything here runs on a single AMD Instinct™ MI300X inside the official ROCm PyTorch container, and every command is copy-paste ready. Follow along in your own terminal as you read.\n\n### Step 0 — Set up the environment[#](#step-0-set-up-the-environment)\n\nStart the official ROCm PyTorch container, clone the repo, and install the lightweight Python dependencies (PyTorch and Triton already ship in the container):\n\n```\n# Start the official ROCm PyTorch container on your MI300X host\ndocker run -it --rm \\\n  --device /dev/kfd --device /dev/dri \\\n  --group-add video --ipc host --shm-size 16G \\\n  rocm/pytorch:rocm7.1.1_ubuntu24.04_py3.12_pytorch_release_2.10.0\n\n# Inside the container\ngit clone https://github.com/AMD-AGI/AgentKernelArena.git\ncd AgentKernelArena\npip install -r requirements.txt\n```\n\nAKA launches agents through their official CLIs, so install the one you want to drive and export its API key. We will use [Claude Code](https://www.anthropic.com/claude-code):\n\n```\nnpm install -g @anthropic-ai/claude-code\nexport ANTHROPIC_API_KEY=\"your_anthropic_key\"\n```\n\n### Step 1 — Look at a single task[#](#step-1-look-at-a-single-task)\n\nEvery task is a self-contained folder under `tasks/`\n\n. Open the RMS-norm task and you will find three files that fully define it:\n\n```\ntasks/triton2triton/vllm/triton_rms_norm/\n├── source/triton_rms_norm.py     # the baseline kernel the agent will optimize\n├── scripts/task_runner.py        # how the kernel is compiled, checked, and timed\n└── config.yaml                   # what the agent is told, and which commands to run\n```\n\nThe **source file** holds the starting kernel — a straightforward, correct, two-pass Triton RMS norm. This is the agent’s “before”:\n\n``` python\n@triton.jit\ndef _rms_norm_kernel(input_ptr, weight_ptr, output_ptr, ...):\n    # Pass 1: accumulate sum of squares in float32\n    # Pass 2: normalize and apply weight\n    ...\n```\n\nThe **task runner** is the contract for “is this kernel correct and how fast is it.” It defines the test shapes and the timing protocol, so every agent is measured the same way — note the five input shapes, the float32 reference, and the warmup/benchmark iteration counts:\n\n```\nTEST_SHAPES = [(32, 128), (64, 512), (128, 1024), (256, 2048), (512, 4096)]\nWARMUP_ITERATIONS = 10\nBENCHMARK_ITERATIONS = 100\n# correctness checks torch.allclose(result, ref, atol=1e-2, rtol=1e-2)\n```\n\nThe **task config** is what ties it together — the kernel symbol to optimize, the compile/correctness/performance commands, and the natural-language `instructions`\n\nthe agent actually reads:\n\n```\ntarget_kernel_functions:\n  - _rms_norm_kernel\ncompile_command:\n  - python3 scripts/task_runner.py compile\ncorrectness_command:\n  - python3 scripts/task_runner.py correctness\nperformance_command:\n  - python3 scripts/task_runner.py performance\ntask_type: triton2triton\nprompt:\n  instructions: |\n    Optimize the Triton RMS normalization kernel `_rms_norm_kernel` for maximum\n    GPU throughput while maintaining numerical correctness.\n    ...\n    Constraints:\n    - Must maintain the same function signature for `rms_norm`\n    - Output must match reference within atol=1e-2, rtol=1e-2 for float16\n```\n\nBefore any agent touches the kernel, you can run the exact same measurement the framework uses to establish the baseline. From the task directory:\n\n```\ncd tasks/triton2triton/vllm/triton_rms_norm\npython3 scripts/task_runner.py compile\npython3 scripts/task_runner.py correctness\npython3 scripts/task_runner.py performance\n```\n\nYou will see the baseline compile, pass correctness on all five shapes, and report a per-shape execution time. That time is the denominator of every speedup AKA will report. Now let’s hand the kernel to an agent.\n\n### Step 2 — Run an agent on the task[#](#step-2-run-an-agent-on-the-task)\n\nDrive the whole framework from a single `config.yaml`\n\nat the repository root. A minimal config that points one agent at our one task looks like this:\n\n```\nagent:\n  template: claude_code        # or: cursor, codex, geak_optimagentv2, single_llm_call, ...\ntasks:\n  - triton2triton/vllm/triton_rms_norm\n  # - triton2triton/vllm/*     # run every Triton kernel under vllm\n  # - all                      # run every task in the suite\ntarget_gpu_model: MI300        # maps to the gfx942 architecture\nlog_directory: logs\nworkspace_directory_prefix: workspace\n```\n\nPer-agent runtime settings — the backing model, the iteration budget, and the timeout — live in `agents/<agent>/agent_config.yaml`\n\n. For Claude Code, `agents/claude_code/agent_config.yaml`\n\nsets `model: claude-opus-4-6`\n\n, `max_iterations: 3`\n\n, and `timeout_seconds: 3600`\n\n. Change those to swap the model or the budget without touching anything else.\n\nWith the config and your key in place, a full run is one command:\n\n```\npython main.py\n```\n\nHere is what AKA does for you: it spins up an isolated, timestamped workspace, copies the task into it, measures the baseline, hands the kernel to Claude Code with the shared prompt, lets the agent iterate, and then runs the centralized evaluator on whatever the agent produced. You will see the agent’s output stream live in your terminal.\n\n### Step 3 — Read the results and the score[#](#step-3-read-the-results-and-the-score)\n\nEach run lands in its own timestamped directory, so no run ever overwrites another:\n\n```\nworkspace_MI300_claude_code/run_<timestamp>/\n├── triton2triton_vllm_triton_rms_norm_<timestamp>/\n│   ├── source/triton_rms_norm.py   # the agent's optimized kernel\n│   └── task_result.yaml            # per-task verdict and timings\n└── reports/\n    ├── overall_report.txt          # human-readable summary\n    ├── overall_summary.csv         # per-task scores and speedups\n    └── task_type_breakdown.json    # aggregate stats by task type\n```\n\nOpen `task_result.yaml`\n\nto see exactly how the task scored:\n\n```\ntask_name: triton2triton/vllm/triton_rms_norm\npass_compilation: true\npass_correctness: true\nbase_execution_time: ...        # baseline ms (averaged across shapes)\nbest_optimized_execution_time: ...   # optimized ms\nspeedup_ratio: ...              # base / optimized\n```\n\nPlug those into the scoring rule from Table 2 — 20 points for compiling, 100 for correctness, and `speedup × 100`\n\nfor performance — and you have the task’s score, the same number that lands in `overall_summary.csv`\n\n. `diff`\n\nthe agent’s `source/triton_rms_norm.py`\n\nagainst the original to see exactly what it changed (block-size tuning, fused passes, vectorized loads). That is the full loop: one task, one agent, one reproducible score.\n\nTo pause and continue a long evaluation, pass `--resume-latest`\n\nor `--resume-run run_YYYYMMDD_HHMMSS`\n\n. Because the evaluator lives outside the agent, any score you produce is reproducible by anyone running the same config on the same hardware.\n\n### Bring Your Own Task or Agent[#](#bring-your-own-task-or-agent)\n\nAKA is designed so that adding to the arena is cheap. **A new task** is just a folder with a `config.yaml`\n\n(source files, the kernel symbol to optimize, and compile/correctness/performance commands) plus a `scripts/task_runner.py`\n\n. Every new task must first pass our **task validator agent**, which runs 10 automatic sanity checks — schema, kernel symbol resolution, compile/correctness/performance, GPU hang detection, and report-template compatibility — before it can be merged:\n\n```\n# config.yaml at the repo root\nagent:\n  template: task_validator\ntasks:\n  - triton2triton/vllm/my_new_task\npython main.py   # writes validation_report.yaml into the workspace\n```\n\n**A new agent** plugs in through one launcher function and a registry entry. Implement `launch_agent`\n\n, decorate it with `@register_agent`\n\n, and add it to the `AgentType`\n\nenum so AKA can find it:\n\n``` python\n# agents/your_agent/launch_agent.py\nfrom agents import register_agent\n\n@register_agent(\"your_agent\")\ndef launch_agent(eval_config: dict, task_config_dir: str, workspace: str) -> str:\n    \"\"\"Build the prompt, run your agent inside `workspace`, and return its output.\n    The framework owns all measurement — your agent only edits the kernel.\"\"\"\n    ...\n    return agent_output\n```\n\nAdd a matching entry to the `AgentType`\n\nenum and its import in `src/module_registration.py`\n\n, point `config.yaml`\n\nat `template: your_agent`\n\n, and your agent is scored by the exact same evaluator as every other agent in the arena. Figure 5 contrasts the two agent shapes the arena hosts today: the single-agent loop used by the frontier CLIs, and GEAKv3’s parallel sub-agents with a patch-selection step.\n\n*Figure 5. Agent architecture comparison: single-agent designs (Claude Code, Cursor, Codex) versus GEAKv3’s parallel sub-agents with patch selection.*\n\n## Benchmarking Setup[#](#benchmarking-setup)\n\nFor this initial release, we picked **44 representative tasks** that span the major task types and a range of difficulty levels. We ran every configuration on AMD Instinct™ MI300X with identical task definitions, prompts, and the unified evaluator. Every agent ran with the same compute budget: a 3600-second per-task timeout and `max_iterations`\n\nset to 3. We evaluated six configurations:\n\nGEAKv3 with Claude Opus 4.6\n\nCursor Agent with Claude Opus 4.6\n\nCursor Agent with GPT-5.3 Codex\n\nCursor Agent with Composer 2\n\nClaude Code with Claude Opus 4.6\n\nClaude Code with Claude Sonnet 4.6\n\nOne caveat matters for reading these results fairly. The general-purpose coding agents — Cursor Agent, Claude Code, and Codex — all run through the same harness, receive the identical task prompt, and share the same iteration budget, so the differences between them come down to each agent’s own orchestration and its backing model. GEAKv3 is not a general coding agent: it is a dedicated GPU kernel-optimization agent with a more involved architecture — parallel sub-agents that explore different optimizations, plus a patch-selection step — so it brings domain-specific machinery the others do not. We report all of them side by side because that is the comparison AKA is built to make, but GEAKv3’s design is meaningfully different from the single-agent loop the frontier agents use. Note too that the frontier agents are moving targets: their CLIs may change their internal tool use, orchestration, or default model over time, so the same configuration can produce different numbers on a later run.\n\nThe 44 tasks break down as: **20 HIP2HIP** tasks (gpumode-style attention, MLP, normalization, activation, and loss kernels), **20 Triton2Triton** tasks (drawn from vLLM and ROCmBench), and **4 repository-scale rocPRIM** tasks (L3 difficulty).\n\n### System Configuration[#](#system-configuration)\n\nWe ran every experiment inside the official ROCm PyTorch container on AMD Instinct™ MI300X. Table 3 lists the exact hardware and software versions.\n\nComponent |\nSpecification |\n|---|---|\nGPU |\nAMD Instinct™ MI300X (192 GB HBM3) |\nROCm |\n7.1.1 |\nDocker image |\n|\nHost OS |\nUbuntu 24.04 |\nPython |\n3.12 |\nPyTorch |\n2.10.0 (ROCm) |\nTriton |\n3.6.0 |\nPer-task timeout |\n3600 s |\n|\n3 |\n\n*Table 3. Hardware and software configuration used for all benchmarks.*\n\n## Results[#](#results)\n\nScope.These results are acontrolled comparison, not a peak-performance benchmark. Every configuration ran the same 44 tasks under one fixed recipe — identical task definitions and prompts, the same 3600 s per-task timeout,`max_iterations=3`\n\n, and one shared evaluator — with no per-agent or per-model tuning. Read therelativedifferences between configurations rather than the absolute speedup magnitudes: the numbers are a floor with tuning headroom, not a ceiling, and they reflect this 44-task subset on AMD Instinct™ MI300X. They are also a snapshot in time — the frontier agent CLIs and their backing models change frequently, and an agent may alter its internal orchestration, tool calls, or default model between releases, so re-running the same configuration later can produce different numbers.\n\n### HIP2HIP — 20 Selected Kernels[#](#hip2hip-20-selected-kernels)\n\nHIP2HIP tasks ask each agent to take an existing, correct HIP kernel and meaningfully optimize it on MI300X, subject to correctness checks on multiple input shapes.\n\n**The 20 kernels — from GPU Mode:** CrossEntropyLossLabelSmoothing, Feedforward, FusedLeakyReLU, GateGRUSelectionLayer, Gather, GELU, InnerProd, ItemQueryAttention, KDLoss, layer_normalization, MaskedLanguageModel, MLP_model, MultiHeadAttention, NormalAttention_dot, NormalAttention_embedded_gaussian, PositionEmbedder, Sigmoid, and SiLU. **From other HIP sources:** assign_score_withk and ball_query.\n\nTable 4 reports the per-agent results on these 20 kernels.\n\nAgent (Model) |\nCompile % |\nCorrect % |\nSpeedup>1 % |\nAvg Speedup |\nMedian |\nP25 / P75 / P90 |\nAvg Score |\n|---|---|---|---|---|---|---|---|\nCursor (Composer 2) |\n100% |\n100% |\n65% |\n1.34× |\n1.02× |\n1.00 / 1.75 / 2.17× |\n254.34 |\nCursor (Claude Opus 4.6) |\n100% |\n100% |\n100% |\n5.03× |\n1.99× |\n1.21 / 4.20 / 20.57× |\n623.05 |\nCursor (GPT-5.3 Codex) |\n100% |\n100% |\n90% |\n3.06× |\n1.33× |\n1.02 / 2.24 / 10.37× |\n426.02 |\nClaude Code (Sonnet 4.6) |\n100% |\n100% |\n95% |\n5.27× |\n2.85× |\n1.04 / 6.46 / 17.99× |\n647.12 |\nClaude Code (Opus 4.6) |\n100% |\n100% |\n95% |\n6.08× |\n3.05× |\n1.38 / 8.44 / 18.53× |\n727.99 |\n|\n100% |\n100% |\n95% |\n|\n3.03× |\n1.24 / 10.15 / 38.81× |\n|\n\n*Table 4. HIP2HIP results on 20 selected kernels (AMD Instinct™ MI300X).*\n\nEvery agent/model we tested compiles and passes correctness on every HIP2HIP task. On this selected subset, frontier general-purpose agents were very reliable at producing code that passed AKA’s configured correctness checks. Where they differ is performance. On this subset, GEAKv3 with Claude Opus 4.6 recorded a 9.04× average speedup and a long upper tail (P90 of 38.81×). Among the general-purpose agents, Claude Code (Opus 4.6) recorded 6.08×, with Claude Code (Sonnet 4.6) at 5.27× and Cursor (Opus 4.6) at 5.03×. The lighter-weight models recorded smaller gains: Cursor (Composer 2) reached 1.34× and improved on the baseline for 65% of tasks.\n\n### Triton2Triton — 20 Selected Kernels[#](#triton2triton-20-selected-kernels)\n\nTriton-to-Triton optimization is harder than HIP-to-HIP. The Triton search space is narrower — block sizes, `num_warps`\n\n, `num_stages`\n\n, autotune configs, masking strategies, fused-vs-split passes — and many reference kernels are already reasonably well-tuned.\n\n**The 20 kernels — from vLLM:** triton_awq_dequantize, triton_awq_gemm, triton_batched_moe, triton_flash_prefill_attention, triton_fused_moe, triton_layernorm_gated, triton_matmul_persistent, triton_rms_norm, triton_scaled_mm, and triton_topk_log_softmax. **From ROCmBench:** test_add_kernel, test_batched_vecmat, test_block_copy, test_kernel_dot, layernorm, naive_softmax, rmsnorm_fwd, softmax, gemm, and moe_gemm.\n\nThe results for these kernels are shown in Table 5.\n\nAgent (Model) |\nCompile % |\nCorrect % |\nSpeedup>1 % |\nAvg Speedup |\nMedian |\nP25 / P75 / P90 |\nAvg Score |\n|---|---|---|---|---|---|---|---|\nCursor (Composer 2) |\n100% |\n100% |\n40% |\n0.98× |\n0.99× |\n0.91 / 1.03 / 1.10× |\n217.76 |\nCursor (Claude Opus 4.6) |\n100% |\n95% |\n90% |\n1.96× |\n1.05× |\n1.03 / 1.20 / 1.31× |\n300.97 |\nCursor (GPT-5.3 Codex) |\n100% |\n100% |\n65% |\n0.99× |\n1.00× |\n0.95 / 1.03 / 1.09× |\n219.19 |\nClaude Code (Sonnet 4.6) |\n100% |\n100% |\n80% |\n1.71× |\n1.03× |\n1.00 / 1.15 / 2.81× |\n290.74 |\nClaude Code (Opus 4.6) |\n100% |\n100% |\n70% |\n1.95× |\n1.04× |\n0.99 / 1.24 / 2.71× |\n314.63 |\n|\n100% |\n100% |\n80% |\n|\n|\n1.09 / 2.04 / 6.81× |\n|\n\n*Table 5. Triton2Triton results on 20 selected kernels (AMD Instinct™ MI300X).*\n\nOn Triton2Triton the gap between agents narrows. GEAKv3 recorded a 2.75× average and a 1.31× median in this subset. Because GEAKv3’s *median* is well above 1, these gains look broad rather than driven by a single outlier kernel. Claude Code (Opus 4.6) and Cursor (Opus 4.6) recorded very similar averages (1.95× and 1.96×) but lower medians (~1.04×), which points to a heavier-tailed distribution — most of their gains come from a few kernels. Composer 2 and GPT-5.3 Codex through Cursor stayed near 1.0× on average on most Triton kernels in this subset.\n\n### Repository-Scale — 4 L3 rocPRIM Tasks[#](#repository-scale-4-l3-rocprim-tasks)\n\nRepository-scale tasks are the most realistic and the most demanding. The agent works inside a full upstream repository ([rocPRIM](https://github.com/ROCm/rocPRIM)) and has to locate, edit, and verify the relevant kernels without breaking the build or the correctness suite.\n\n**The 4 rocPRIM kernels:** block_radix_rank, device_binary_search, device_merge_sort, and device_search_n.\n\nTable 6 summarizes how each agent fared on these repository-scale tasks.\n\nAgent (Model) |\nCompile % |\nCorrect % |\nSpeedup>1 % |\nAvg Speedup |\nMedian |\nP25 / P75 / P90 |\nAvg Score |\n|---|---|---|---|---|---|---|---|\nCursor (Composer 2) |\n100% |\n100% |\n50% |\n0.95× |\n1.04× |\n0.68 / 1.14 / 1.20× |\n215.24 |\nCursor (Claude Opus 4.6) |\n100% |\n100% |\n75% |\n1.08× |\n1.06× |\n1.01 / 1.16 / 1.25× |\n227.91 |\nCursor (GPT-5.3 Codex) |\n100% |\n100% |\n50% |\n1.06× |\n1.04× |\n1.00 / 1.13 / 1.19× |\n225.58 |\nClaude Code (Opus 4.6) |\n100% |\n100% |\n100% |\n1.15× |\n1.11× |\n1.04 / 1.29 / 1.45× |\n234.62 |\n|\n100% |\n100% |\n100% |\n|\n|\n1.10 / 1.33 / 1.43× |\n|\n\n*Table 6. Repository-scale results on 4 L3 rocPRIM tasks (AMD Instinct™ MI300X).*\n\nEvery agent compiles and passes correctness on rocPRIM, but the speedups are smaller than on HIP2HIP. rocPRIM is already a heavily tuned production library, so the headroom for further optimization is naturally limited. GEAKv3 recorded a 1.20× average with a 100% Speedup>1 rate, and Claude Code (Opus 4.6) recorded 1.15× (also 100% Speedup>1). The takeaway: on real upstream code, both model capability and kernel-specific specialization matter — but neither one buys you double-digit speedups.\n\n### The Overall Picture[#](#the-overall-picture)\n\nTable 7 brings the three categories together, listing each configuration’s average speedup side by side.\n\nAgent (Model) |\nHIP2HIP (20) |\nTriton2Triton (20) |\nRepo rocPRIM (4 L3) |\n|---|---|---|---|\nGEAKv3 (Opus 4.6) |\n|\n|\n|\nClaude Code (Opus 4.6) |\n6.08× |\n1.95× |\n1.15× |\nCursor (Opus 4.6) |\n5.03× |\n1.96× |\n1.08× |\nCursor (GPT-5.3 Codex) |\n3.06× |\n0.99× |\n1.06× |\nCursor (Composer 2) |\n1.34× |\n0.98× |\n0.95× |\n\n*Table 7. Average speedup by category on the selected 44-task subset (AMD Instinct™ MI300X).*\n\nThree patterns stand out across the three categories:\n\n**GEAKv3 recorded higher average speedups than the other configurations in every category** on this subset, by the widest margin on HIP2HIP.**HIP2HIP showed the largest speedups of the three categories**, suggesting that iterative agent workflows can uncover substantial kernel-level optimizations on this subset.** Model choice has a consistent effect.**Switching from Opus 4.6 to Composer 2 inside the same Cursor agent moved the HIP2HIP average from 5.03× to 1.34× and pushed Triton2Triton and rocPRIM below 1×.\n\n## Discussion[#](#discussion)\n\nThese initial results match what we see when we work with these agents day to day. Frontier general-purpose agents are now very reliable at producing *correct* GPU code — compilation and correctness rates are at or near 100% across almost every cell of our tables. What separates them is whether they can find non-trivial *performance* improvements: restructuring the kernel (grid layout, tiling, fused-vs-split passes), tuning launch parameters for the target architecture, and avoiding common anti-patterns.\n\nGEAKv3 is AMD’s in-house agent, designed specifically for GPU kernel optimization. On this subset it recorded higher average speedups than the other configurations on HIP2HIP and Triton2Triton. We read this as one data point — on a small, fixed task set — that a domain-specific agent can be competitive with frontier general-purpose agents on kernel tasks, not as a general ranking. A larger task set and more runs per configuration would be needed to draw stronger conclusions, and that is exactly the kind of comparison AKA is built to support.\n\nOne thing to be clear about: **AKA is not a final leaderboard, and these results are not a verdict on any single agent.** AKA is an arena. Anyone running the same config on the same hardware can reproduce these scores. Any change in score should map back to a concrete change in the agent — a different model, prompt, tool, or orchestration pattern. That is exactly what makes AKA useful as an A/B testing harness for agent improvements.\n\n## Summary[#](#summary)\n\nIn this blog you learned what AgentKernelArena is and why it matters, how its unified pipeline separates optimization from evaluation to keep every agent’s score comparable (Figure 2), and how its cumulative scoring rewards compilation, correctness, and measured speedup (Table 2, Figure 4). You walked through a complete run on the vLLM Triton RMS-norm kernel end to end — from baseline measurement to reading the final score — and saw how to bring your own task or agent into the arena. And you saw what we found when we benchmarked six agent/model configurations across HIP2HIP, Triton2Triton, and repository-scale rocPRIM tasks on AMD Instinct™ MI300X (Tables 4–7, Figure 1).\n\nTo recap the headline numbers: AgentKernelArena puts AI coding agents for GPU kernel optimization on a common, reproducible footing. It decouples the agent from the evaluator, supports heterogeneous launchers and models, and reports unified scores with per-test-case detail — enabling fair comparisons that were previously hard to make. On the 44-task subset we evaluated here, GEAKv3 recorded higher average speedups than the other configurations on HIP2HIP (9.04×), Triton2Triton (2.75×), and repository-scale rocPRIM (1.20×); among the general-purpose coding agents we tested, Claude Code (Opus 4.6) and Cursor (Opus 4.6) recorded the largest speedups.\n\nTry AKA. Run your own agent against the same tasks, contribute new tasks, or propose a new scoring policy. The arena is built to grow, and this release is just the starting line: in upcoming work we plan to expand to other task types, add more agents and models, broaden coverage to additional AMD Instinct™ hardware targets, and publish deeper A/B studies on how agent-side changes — new tools, MCP servers, planners, and prompt strategies — move the score. Check the GitHub repository for the latest tasks and results. The framework is open-source and actively maintained — we believe rigorous, domain-specific benchmarks like AgentKernelArena are essential for driving genuine progress in AI-assisted GPU programming.\n\n## Additional Resources[#](#additional-resources)\n\nAgentKernelArena GitHub repository:\n\n[AMD-AGI/AgentKernelArena](https://github.com/AMD-AGI/AgentKernelArena)GEAK GitHub repository:\n\n[AMD-AGI/GEAK](https://github.com/AMD-AGI/GEAK)GEAK family blog:\n\n[https://rocm.blogs.amd.com/artificial-intelligence/geak-agents-family/README.html](https://rocm.blogs.amd.com/artificial-intelligence/geak-agents-family/README.html)AMD Instinct™ MI300X:\n\n[https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html](https://www.amd.com/en/products/accelerators/instinct/mi300/mi300x.html)AMD ROCm™ documentation:\n\n[https://rocm.docs.amd.com/](https://rocm.docs.amd.com/)HIP programming guide:\n\n[https://rocm.docs.amd.com/projects/HIP/en/latest/](https://rocm.docs.amd.com/projects/HIP/en/latest/)ROCm workload tuning guide:\n\n[https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)rocprof-compute (profiler):\n\n[ROCm/rocm-systems](https://github.com/ROCm/rocm-systems/tree/develop/projects/rocprofiler-compute)Triton language:\n\n[https://triton-lang.org/](https://triton-lang.org/)rocPRIM:\n\n[ROCm/rocPRIM](https://github.com/ROCm/rocPRIM)ROCm Blogs:\n\n[https://rocm.blogs.amd.com/](https://rocm.blogs.amd.com/)\n\n## Bias, Risks, and Limitations[#](#bias-risks-and-limitations)\n\nAI coding agents may generate kernels that are incorrect, unsafe, or that contain subtle numerical issues. AgentKernelArena’s correctness gate reduces but does not eliminate this risk; users must conduct their own evaluation and validation before deploying any generated kernel.\n\nThe results in this blog are based on a 44-task subset and a fixed compute budget. They illustrate the framework and should not be read as a definitive ranking of any agent or model.\n\n## License[#](#license)\n\nThe AgentKernelArena code is released under the Apache 2.0 license.\n\n## Acknowledgements[#](#acknowledgements)\n\nThis work was made possible thanks to the contributions of Parsa Fashi, Ji Liu, Yue Liu, Yuchen Yang, Hao Li, Ziqiong Liu, Saptarshi Majumder, Subrahmanya Pavankumar Dubagunta, Pratik Prabhanjan Brahma, Dong Li, Arsalan Farooq.\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, AMD CDNA, AMD Instinct, AMD ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. PyTorch is a registered trademark of Meta Platforms, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved", "url": "https://wpnews.pro/news/agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/agent-kernel-arena/README.html", "published_at": "2026-07-03 00:00:00+00:00", "updated_at": "2026-07-07 01:03:27.407402+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-research", "ai-products", "developer-tools"], "entities": ["AMD", "AgentKernelArena", "GEAKv3", "Claude Code", "Cursor Agent", "OpenAI Codex", "AMD Instinct MI300X", "ROCm"], "alternates": {"html": "https://wpnews.pro/news/agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on", "markdown": "https://wpnews.pro/news/agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on.md", "text": "https://wpnews.pro/news/agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on.txt", "jsonld": "https://wpnews.pro/news/agentkernelarena-benchmarking-ai-coding-agents-for-gpu-kernel-optimization-on.jsonld"}}