# Measured attention kernels vs. AMD's own on MI350X, with every loss published

> Source: <https://github.com/s7ventures/riptide-amd>
> Published: 2026-11-19 20:50:25+00:00

**Recommendation:** Benchmark attention kernels per head dimension and sequence length before adopting them: Riptide's kernels win 3.32× at d128 causal S=512 but lose 0.82–0.84× at 16k tokens.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

Measured results for [Riptide](https://riptidecompute.com)'s GPU kernels on AMD Instinct MI350X, and the benchmark
harness that produced them. Every number is measured in the same session as AMD's own kernels, with 95% confidence
intervals and a float64 correctness check.

**This repository holds the results, the method and the harness. The kernels themselves are not open source.** They
are available to partners for evaluation and licensing: [sam@s7vc.com](mailto:sam@s7vc.com).

**All reports: [RESULTS.md](https://github.com/s7ventures/riptide-amd/blob/main/RESULTS.md)** · each run has a verdict, a chart, "where we lose", how it was measured,
the raw data and a machine-readable `results.json` ([format](https://github.com/s7ventures/riptide-amd/blob/main/docs/results-format.md)).

`F.scaled_dot_product_attention`, bf16, 32 heads, 16k tokens per batch. TFLOP/s at the median of 30 samples
[95% CI]. "AMD best" is the faster of PyTorch's FLASH_ATTENTION (AOTriton) and EFFICIENT_ATTENTION backends in the
same session. ROCm 10.2, PyTorch 2.15. fwd+bwd is one training step of the attention layer.

| shape | Riptide | AMD best | Riptide / AMD best | 
|---|---|---|---|
| d128 causal fwd+bwd, S = 512 | 249 [247, 252] | 75 | **3.32×** [3.28, 3.38] | 
| d64 causal fwd+bwd, S = 512 | 213 [211, 215] | 69 | **3.10×** [3.07, 3.14] | 
| d128 fwd+bwd, S = 1024 | 477 [472, 481] | 417 | **1.14×** [1.12, 1.17] | 
| d128 causal fwd+bwd, S = 2048 | 444 [440, 447] | 389 | **1.14×** [1.12, 1.16] | 
| d64 forward, S = 16384 | 717 [714, 718] | 594 | **1.21×** [1.19, 1.22] | 
| d64 causal fwd+bwd, S = 16384 | 514 [513, 515] | 464 | **1.11×** [1.10, 1.11] | 
| d128 forward, S = 16384 | 821 [819, 823] | 982 | 0.84× [0.83, 0.84] | 
| d128 fwd+bwd, S = 16384 | 635 [635, 635] | 756 | 0.84× [0.84, 0.84] | 
| d128 causal fwd+bwd, S = 16384 | 597 [596, 597] | 732 | 0.82× [0.81, 0.82] | 

- **Ahead** : head dim 64 at every length (forward 1.14–1.39×, training 1.03–1.14× from 1k tokens, 1.9–3.1× at 512);
head dim 128 training up to 1k (full) and 2k (causal) tokens, 3.32× at 512 causal; head dim 128 causal forward up
to 2k tokens and fp16 forward at every length.
- **Behind** : head dim 128 bf16 from 2k (full) / 4k (causal) tokens: 0.82–0.84× at 16k, up from 0.61× / 0.48× that
morning. fp16 head dim 128 forward is even or ahead at every length (1.0–1.21×).

Full report: [results/attention/2026-11-19f/REPORT.md](https://github.com/s7ventures/riptide-amd/blob/main/results/attention/2026-11-19f/REPORT.md)
(the day's earlier runs: 2026-11-19, -19b, -19c, -19d).

Run d kernels (the AITER run came before run f's forward changes). Same pod, AITER `amd-aiter` 0.1.7 `flash_attn_func`, bf16, 32 heads, 16k tokens per batch, S = 1k–16k, median of
30 calls; outputs agree to bf16 rounding. Raw rows: [results/attention/2026-11-19-aiter/vs_aiter.jsonl](https://github.com/s7ventures/riptide-amd/blob/main/results/attention/2026-11-19-aiter/vs_aiter.jsonl).

|  | forward | fwd+bwd, vs AITER's deterministic backward | 
|---|---|---|
| d64 full / causal | 1.00–1.12× / 1.03–1.12× | 1.51–1.77× / 1.85–2.88× | 
| d128 full / causal | 0.78–0.90× / 0.78–0.82× | 1.05–1.20× / 1.61–1.72× | 

AITER ran its deterministic backward (Riptide's backward is deterministic too); its non-deterministic backward has not been timed yet, so these rows are not a comparison with AITER's fastest backward. AITER's head-dim-128 forward is faster than Riptide's.

| case | Riptide | baseline | ratio | 
|---|---|---|---|
| Qwen2.5-0.5B decode, batch 1, prompt 128 | 399 tok/s | 106 tok/s (Hugging Face eager) | **3.75×** [3.73, 3.75] | 
| same, batch 8 |  |  | **2.36×** | 
| same vs torch.compile, batch 1 |  |  | **1.37×** [1.36, 1.38] | 
| MNIST MLP training step | 0.127 ms | 0.225 ms (torch.cuda graphs) | **1.76×** [1.59, 1.90] | 
| empty kernel launch | 1.53 µs | 6.83 µs (Triton) | **4.47×** | 

Level or behind: graph replay per launch equals torch.cuda graphs; batch 8 decode vs torch.compile 0.96× / 0.79×.
All rows: [results/launch/2026-11-19/headline.md](https://github.com/s7ventures/riptide-amd/blob/main/results/launch/2026-11-19/headline.md).

`bench/` measures AMD's own stack on any MI300/MI350 machine: GEMM, attention, elementwise ops, convolution,
training, LLM inference, the library CLIs and MIGraphX. Statistics, correctness checks and the FLOP accounting are
in [docs/methodology.md](https://github.com/s7ventures/riptide-amd/blob/main/docs/methodology.md); how to run it is in [docs/reproducing.md](https://github.com/s7ventures/riptide-amd/blob/main/docs/reproducing.md). The
Riptide rows need the riptide-amd package (partners); everything else runs as is.

Evaluation access and licensing: **[sam@s7vc.com](mailto:sam@s7vc.com)** · LinkedIn: [Samario Torres](https://www.linkedin.com/in/samariotorres/) · [riptidecompute.com](https://riptidecompute.com)

The harness and documentation in this repository are licensed under Apache-2.0 (see LICENSE). The results are published for reference; please cite them as in CITATION.cff.
