{"slug": "measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published", "title": "Measured attention kernels vs. AMD's own on MI350X, with every loss published", "summary": "Riptide published measured benchmarks of its attention kernels against AMD's own on the AMD Instinct MI350X, reporting a 3.32× speedup [3.28, 3.38] over AMD's best PyTorch backend for d128 causal forward-plus-backward at sequence length 512, while losing 0.82–0.84× at 16k tokens for d128 bf16. All runs used bf16 with 32 heads, 16k tokens per batch, ROCm 10.2 and PyTorch 2.15, with 95% confidence intervals and float64 correctness checks, and Riptide published every loss alongside the wins. The kernels are not open source and are offered to partners for evaluation and licensing, and against AMD's AITER library Riptide's deterministic backward ran 1.51–2.88× faster while AITER's head-dim-128 forward stayed faster.", "body_md": "Measured results for [Riptide](https://riptidecompute.com)'s GPU kernels on AMD Instinct MI350X, and the benchmark\nharness that produced them. Every number is measured in the same session as AMD's own kernels, with 95% confidence\nintervals and a float64 correctness check.\n\n**This repository holds the results, the method and the harness. The kernels themselves are not open source.** They\nare available to partners for evaluation and licensing: [sam@s7vc.com](mailto:sam@s7vc.com).\n\n**All reports: [RESULTS.md](https://github.com/s7ventures/riptide-amd/blob/main/RESULTS.md)** · each run has a verdict, a chart, \"where we lose\", how it was measured,\nthe raw data and a machine-readable `results.json` ([format](https://github.com/s7ventures/riptide-amd/blob/main/docs/results-format.md)).\n\n`F.scaled_dot_product_attention`, bf16, 32 heads, 16k tokens per batch. TFLOP/s at the median of 30 samples\n[95% CI]. \"AMD best\" is the faster of PyTorch's FLASH_ATTENTION (AOTriton) and EFFICIENT_ATTENTION backends in the\nsame session. ROCm 10.2, PyTorch 2.15. fwd+bwd is one training step of the attention layer.\n\n| shape | Riptide | AMD best | Riptide / AMD best | \n|---|---|---|---|\n| d128 causal fwd+bwd, S = 512 | 249 [247, 252] | 75 | **3.32×** [3.28, 3.38] | \n| d64 causal fwd+bwd, S = 512 | 213 [211, 215] | 69 | **3.10×** [3.07, 3.14] | \n| d128 fwd+bwd, S = 1024 | 477 [472, 481] | 417 | **1.14×** [1.12, 1.17] | \n| d128 causal fwd+bwd, S = 2048 | 444 [440, 447] | 389 | **1.14×** [1.12, 1.16] | \n| d64 forward, S = 16384 | 717 [714, 718] | 594 | **1.21×** [1.19, 1.22] | \n| d64 causal fwd+bwd, S = 16384 | 514 [513, 515] | 464 | **1.11×** [1.10, 1.11] | \n| d128 forward, S = 16384 | 821 [819, 823] | 982 | 0.84× [0.83, 0.84] | \n| d128 fwd+bwd, S = 16384 | 635 [635, 635] | 756 | 0.84× [0.84, 0.84] | \n| d128 causal fwd+bwd, S = 16384 | 597 [596, 597] | 732 | 0.82× [0.81, 0.82] | \n\n- **Ahead** : head dim 64 at every length (forward 1.14–1.39×, training 1.03–1.14× from 1k tokens, 1.9–3.1× at 512);\nhead dim 128 training up to 1k (full) and 2k (causal) tokens, 3.32× at 512 causal; head dim 128 causal forward up\nto 2k tokens and fp16 forward at every length.\n- **Behind** : head dim 128 bf16 from 2k (full) / 4k (causal) tokens: 0.82–0.84× at 16k, up from 0.61× / 0.48× that\nmorning. fp16 head dim 128 forward is even or ahead at every length (1.0–1.21×).\n\nFull report: [results/attention/2026-11-19f/REPORT.md](https://github.com/s7ventures/riptide-amd/blob/main/results/attention/2026-11-19f/REPORT.md)\n(the day's earlier runs: 2026-11-19, -19b, -19c, -19d).\n\nRun d kernels (the AITER run came before run f's forward changes). Same pod, AITER `amd-aiter` 0.1.7 `flash_attn_func`, bf16, 32 heads, 16k tokens per batch, S = 1k–16k, median of\n30 calls; outputs agree to bf16 rounding. Raw rows: [results/attention/2026-11-19-aiter/vs_aiter.jsonl](https://github.com/s7ventures/riptide-amd/blob/main/results/attention/2026-11-19-aiter/vs_aiter.jsonl).\n\n|  | forward | fwd+bwd, vs AITER's deterministic backward | \n|---|---|---|\n| d64 full / causal | 1.00–1.12× / 1.03–1.12× | 1.51–1.77× / 1.85–2.88× | \n| d128 full / causal | 0.78–0.90× / 0.78–0.82× | 1.05–1.20× / 1.61–1.72× | \n\nAITER ran its deterministic backward (Riptide's backward is deterministic too); its non-deterministic backward has not been timed yet, so these rows are not a comparison with AITER's fastest backward. AITER's head-dim-128 forward is faster than Riptide's.\n\n| case | Riptide | baseline | ratio | \n|---|---|---|---|\n| Qwen2.5-0.5B decode, batch 1, prompt 128 | 399 tok/s | 106 tok/s (Hugging Face eager) | **3.75×** [3.73, 3.75] | \n| same, batch 8 |  |  | **2.36×** | \n| same vs torch.compile, batch 1 |  |  | **1.37×** [1.36, 1.38] | \n| MNIST MLP training step | 0.127 ms | 0.225 ms (torch.cuda graphs) | **1.76×** [1.59, 1.90] | \n| empty kernel launch | 1.53 µs | 6.83 µs (Triton) | **4.47×** | \n\nLevel or behind: graph replay per launch equals torch.cuda graphs; batch 8 decode vs torch.compile 0.96× / 0.79×.\nAll rows: [results/launch/2026-11-19/headline.md](https://github.com/s7ventures/riptide-amd/blob/main/results/launch/2026-11-19/headline.md).\n\n`bench/` measures AMD's own stack on any MI300/MI350 machine: GEMM, attention, elementwise ops, convolution,\ntraining, LLM inference, the library CLIs and MIGraphX. Statistics, correctness checks and the FLOP accounting are\nin [docs/methodology.md](https://github.com/s7ventures/riptide-amd/blob/main/docs/methodology.md); how to run it is in [docs/reproducing.md](https://github.com/s7ventures/riptide-amd/blob/main/docs/reproducing.md). The\nRiptide rows need the riptide-amd package (partners); everything else runs as is.\n\nEvaluation access and licensing: **[sam@s7vc.com](mailto:sam@s7vc.com)** · LinkedIn: [Samario Torres](https://www.linkedin.com/in/samariotorres/) · [riptidecompute.com](https://riptidecompute.com)\n\nThe harness and documentation in this repository are licensed under Apache-2.0 (see LICENSE). The results are published for reference; please cite them as in CITATION.cff.", "url": "https://wpnews.pro/news/measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published", "canonical_source": "https://github.com/s7ventures/riptide-amd", "published_at": "2026-11-19 20:50:25+00:00", "updated_at": "2026-11-19 21:15:18.708003+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-chips", "machine-learning"], "entities": ["Riptide", "AMD", "AMD Instinct MI350X", "PyTorch", "ROCm", "AITER", "Qwen2.5-0.5B", "Triton"], "also_reported_by": [], "takeaway": "Riptide's attention kernels beat AMD's best PyTorch backend by 3.32× at d128 causal S=512 but trail at 0.82–0.84× for d128 bf16 at 16k tokens, so kernel choice should be gated on head dimension and sequence length.", "recommendation": {"advice": "Benchmark attention kernels per head dimension and sequence length before adopting them: Riptide's kernels win 3.32× at d128 causal S=512 but lose 0.82–0.84× at 16k tokens.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published", "markdown": "https://wpnews.pro/news/measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published.md", "text": "https://wpnews.pro/news/measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published.txt", "jsonld": "https://wpnews.pro/news/measured-attention-kernels-vs-amd-s-own-on-mi350x-with-every-loss-published.jsonld"}}