{"slug": "an-educational-gemm-ladder-for-helios-gpus", "title": "An Educational GEMM Ladder for Helios GPUs", "summary": "AMD's Helios GPU, the Instinct MI455X, packs 432 GB of HBM4, 23 TB/s of HBM bandwidth per GPU, and 40 PFLOPs of FP4 compute, according to AMD's product brochure. A new educational blog post builds a ladder of BF16 general matrix multiplication (GEMM) kernels in HipKittens to show kernel developers how Helios's architecture — 256 workgroup processors across eight Accelerator Complex Dies, 320 KB of LDS per WGP, and a 72-GPU rack scale-up domain at 3.6 TB/s — affects kernel design. The ladder is inspired by Simon Boehm's CUDA GEMM worklog and targets large frontier models and long-context agentic workloads.", "body_md": "# An Educational GEMM Ladder for Helios GPUs[#](#an-educational-gemm-ladder-for-helios-gpus)\n\nAMD Helios will be an important platform for AI. Helios offers 432 GB of HBM4,\n23 TB/s of HBM bandwidth per GPU, and 40 PFLOPs of FP4 compute\n[AMD Instinct™ MI455X GPU](https://www.amd.com/content/dam/amd/en/documents/products/accelerators/instinct/amd-instinct-mi455x_brochure.pdf).\nThese capabilities will be especially valuable for large frontier models and long-context\nagentic workloads.\n\nIn this blog post, we highlight several features of the Helios architecture and build an\neducational ladder of BF16 general matrix multiplication (GEMM) kernels that progressively\ntakes advantage of them. The ladder is inspired by Simon Boehm’s CUDA GEMM worklog\n[How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog](https://siboehm.com/articles/22/CUDA-MMM)\nand is intended to help kernel developers understand how Helios’s new hardware features\naffect kernel design.\n\n## A HipKittens Refresher[#](#a-hipkittens-refresher)\n\nBoth the kernel implementations and optimization ladder in this post use HipKittens, so we\ncollect the main references here before diving in. The framework is introduced in\n[HipKittens: Fast and Furious AMD Kernels](https://arxiv.org/abs/2511.08083), and the\n[HipKittens repository](https://github.com/HazyResearch/HipKittens) contains its source and\nkernel examples.\n\n## Helios Feature Overview[#](#helios-feature-overview)\n\nA Helios GPU contains 256 workgroup processors (WGPs), organized into eight Accelerator Complex Dies (XCDs). Each WGP has 320 KB of local data share (LDS) and 1024 32-bit registers per wave. The GPU has 432 GB of HBM4 with 23 TB/s of peak bandwidth. The Helios scale-up domain includes 72 GPUs per rack with 3.6 TB/s of bandwidth and a unified virtual-memory abstraction that simplifies intra-node memory access.\n\n| Hardware unit | Description | \n|---|---|\n| Single Instruction Multiple Data processor (SIMD) | A group of 32 lanes with its own set of vector general-purpose registers (VGPRs). | \n| Workgroup processor (WGP) | One of the GPU’s 256 processors. Previously referred to as a compute unit (CU) on earlier AMD GPU generations. A WGP contains two SIMD pairs, or four SIMDs in total. | \n| Shader Engine (SE) | A collection of 16 physically co-located WGPs. | \n| Accelerator Complex Die (XCD) | A collection of 32 physically co-located WGPs on a chiplet. | \n| I/O Die (IOD) | A base die with four XCDs stacked on top. Each IOD contains 96 MiB of coherent L2 cache. | \n| GPU | One AMD Instinct™ MI455X GPU consists of two IODs. | \n\nTable 1. Physical compute hierarchy of a CDNA™ 5 Helios GPU.\n\n| Execution unit | Description | \n|---|---|\n| Thread | The smallest unit of execution on the GPU. | \n| Wave | A collection of 32 threads that executes in lockstep. Earlier AMD GPUs used 64 threads per wave. | \n| Workgroup | A collection of waves co-scheduled on a WGP. | \n| Workgroup cluster | A collection of workgroups running concurrently on a Shader Engine. | \n| Grid | The complete collection of workgroups, or workgroup clusters, launched by one kernel. | \n\nTable 2. Logical HIP execution hierarchy on CDNA™ 5.\n\n| Memory | Description | \n|---|---|\n| VGPR | The SIMD-scoped vector register file: 1024 registers, each with 32 lanes of 32-bit values. | \n| LDS/L1 | Each WGP has six 64 KB hardware partitions. Up to five (320 KB) can be allocated to LDS, with at least one retained for L1. | \n| L2 | Two coherent 96 MB halves, one per IOD, totaling 192 MB per device. | \n| High-Bandwidth Memory (HBM) | Eight 54 GB HBM4 stacks, totaling 432 GB. | \n\nTable 3. Physical memory hierarchy of a CDNA™ 5 Helios GPU.\n\nThe key changes at each level of the memory hierarchy include:\n\n- **Partitioned LDS.** Each WGP has five 64 KB LDS partitions. LDS remains banked,\nso layouts must avoid bank conflicts. Two 256-byte-per-cycle paths, one per SIMD pair,\nserve LDS. Concurrent accesses to the same partition can cause partition conflicts, so\nhigh-bandwidth kernels must consider both bank placement within a partition and placement\nacross partitions. A single\n256-byte-per-cycle path is already sufficient to saturate the matrix core units.\n- **Cache structure, NUMA effects, and memory prefetching.** Earlier AMD GPUs used both a\nper-XCD L2 cache and a global last-level cache (LLC). Helios simplifies this hierarchy to\na single L2 cache, physically implemented as two coherent 96 MB halves per GPU. The half\ncloser to a given processor provides substantially higher bandwidth than the remote half\n(more than 40 TB/s for near L2 versus approximately 20 TB/s for remote L2). Across the\nGPU’s eight XCDs, four XCDs reside in each local L2 NUMA domain. Cache hints let kernel\ndevelopers manage L2 behavior, including prefetching global memory into L2 from either\nthe device or the host.\n- **Tensor Data Movement (TDM) for global HBM.** TDM provides a DMA-style path between\nHBM and LDS. It supports scatter-gather access patterns and exposes its descriptor\narchitecture in the ISA. Unlike a hardware-swizzled load, TDM does not rearrange LDS data\non the fly, so padding or layout design is still required to avoid bank conflicts.\n\nThe key changes to the execution model include:\n\n- **Wave size.** Helios uses 32 threads per wave, compared with 64 threads per wave on\nprevious AMD GPUs. On earlier generations, a 64-thread wave executed across 16 physical\nSIMD lanes, creating less regular lane ownership and memory-access patterns that kernel\nprogrammers had to account for when optimizing memory layouts[AMD GPUs go brrr](https://hazyresearch.stanford.edu/blog/2025-11-09-amd-brr) .[\\[1\\]](#id2) Helios pairs 32-thread waves with 32 physical SIMD lanes, enabling more regular lane\nownership and simplifying memory-layout optimization.\n- **Workgroup-cluster launch and multicast.** Helios can guarantee that groups of up to\n16 workgroups are co-located across nearby WGPs, enabling data sharing and synchronization\nacross the cluster. Instead of having every workgroup independently request the same data,\none load can be multicast to multiple WGPs, increasing effective bandwidth through cache\nreuse.\n\nNow let’s put these features into action.\n\n## Educational GEMM Ladder[#](#educational-gemm-ladder)\n\nInspired by Simon Boehm’s GEMM worklog, we present an educational GEMM ladder for Helios GPUs. Figure 1 introduces the MI455X hardware hierarchy and shows how GEMM tiles map onto it:\n\nFigure 1: The MI455X hierarchy narrows from the GPU to XCDs, WGPs, SIMDs, waves, and threads (top). GEMM maps A and B tiles to WGPs that accumulate C output tiles (bottom).\n\nFor a large GEMM, the output matrix is divided into tiles that can be computed independently. Each workgroup, a collection of waves co-scheduled on a WGP, computes one output tile. Every WGP has its own register file and LDS, as well as circuitry for matrix multiplication, exponentials, and other arithmetic in data types including BF16, FP8, FP6, and FP4. All WGPs can also access the GPU’s shared cache hierarchy and HBM. Figure 2 summarizes the measured performance across the optimization ladder:\n\nEven a kernel midway through the ladder outperforms the well-optimized MI355X GEMM kernel, and the final Helios kernels approach twice its performance. These tests were run on early-access GPUs, which continue to receive substantial firmware and software improvements.\n\nEach rung computes $C=AB$, where $A \\in \\mathbb{R}^{M \\times K}$,\n$B \\in \\mathbb{R}^{K \\times N}$, and $C \\in \\mathbb{R}^{M \\times N}$. The inputs and\noutput use BF16 precision. The kernels are written with\n[HipKittens: Fast and Furious AMD Kernels](https://github.com/HazyResearch/HipKittens).\nFor each kernel, we report the\nPFLOP/s attained for $M=N=K=8192$, using 500 warm-up iterations and 100 measured iterations\nwith the L2 cache cleared. The exact benchmarking scripts are available in the HipKittens\nrepository.\n\nFor each rung, we also show the kernel’s hot loop—that is, its iteration over the GEMM\nK dimension—captured with AMD Advanced Thread Trace (ATT) using the profiling tools in the\n[ROCm Systems repository](https://github.com/ROCm/rocm-systems). In these visualizations,\neach row depicts one wave’s instruction execution over time, and a group of rows shows\nexecution on one or more of the WGP’s four SIMDs.\n\n### Level 0: Naive Baseline ([gemm_naive.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/00_gemm_naive.cpp#L58-L82))[#](#level-0-naive-baseline-gemm-naive-cpp)\n\nEach workgroup computes a $64 \\times 64$ output tile using four waves. The waves are arranged in a $2 \\times 2$ grid; each wave computes a $32 \\times 32$ region of the output tile and maintains a corresponding register tile for accumulation. The kernel iterates over the K dimension in chunks of 32. During each iteration, all threads cooperatively load $64 \\times 32$ tiles of A and B from global memory into LDS. After synchronizing, each wave loads its A and B subtiles from LDS into registers, performs the matrix multiplication, and accumulates the result in its output tile.\n\nThis baseline uses one LDS buffer for A and B and does not overlap data movement with compute. Every K iteration therefore proceeds serially: load A and B from global memory into LDS, synchronize, load from LDS into registers and compute, synchronize again, and only then begin loading the next K tile. The second synchronization is required because the same LDS buffer is reused in every iteration. As a result, the matrix units are idle during memory movement, and the memory pipeline is underutilized during computation. The figure below shows the resulting serialized schedule:\n\n#### Level 0 APIs[#](#level-0-apis)\n\n| API | Purpose | \n|---|---|\n| [load(A_LDS, A_global)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/global_to_register.cuh#L29-L98) | Uses vector lanes to copy a global tile through registers into LDS. | \n| [load(A_reg, A_LDS)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/shared_to_register.cuh#L845-L914) | Loads one wave’s LDS fragment into registers. | \n| [sync::fence()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L180-L216) | Drains memory traffic before LDS is published or reused. | \n| [sync::sync()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L156-L170) | Waits for every wave at the workgroup barrier. | \n| [mma_ABt(C, A_reg, B_reg)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/register/tile/mma.cuh#L280-L308) | Accumulates a BF16 $AB^T$ product into FP32 registers. | \n| [store(C_global, C_acc)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/global_to_register.cuh#L127-L196) | Converts and writes the FP32 accumulator directly to the global C tile. | \n\n#### Level 0 Pseudocode[#](#level-0-pseudocode)\n\n``` php\nfor each K tile:\n    load(A_LDS, A_global);       // A: global -> staging registers -> LDS\n    load(B_LDS, B_global);       // B: global -> staging registers -> LDS\n    sync::fence();               // Wait for global-to-LDS traffic\n    sync::sync();                // Wait for peer waves to publish LDS\n    load(A_reg, A_LDS);          // A: LDS -> registers\n    load(B_reg, B_LDS);          // B: LDS -> registers\n    mma_ABt(C, A_reg, B_reg);    // Accumulate C += A * B^T\n    sync::fence();               // Wait for LDS reads\n    sync::sync();                // Wait before reusing LDS\n```\n\nThe trace in Figure 4 shows one SIMD with 12 resident wave tracks from different workgroups; the scheduler switches among them automatically to maximize resource utilization:\n\nOn SIMD0 wave slot 0, early green VALU instructions come from register-mediated A/B fills\nand address calculations. Four purple WMMA instructions at cycles 1,218–1,620 map to\n`mma_ABt`. The long yellow intervals are consistent with publish and reuse synchronization,\nalthough the color alone does not identify a specific barrier.\n\n### Level 1: Double-Buffered in LDS ([gemm_double_buf.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/01_gemm_double_buf.cpp#L65-L89))[#](#level-1-double-buffered-in-lds-gemm-double-buf-cpp)\n\n- **Performance:** Less than 1% faster than Level 0 (25.3% to 25.4% of the MI355X baseline in Figure 2).\n\nThe previous kernel severely underutilizes the 320 KB of LDS available per WGP. At a\n$64 \\times 64$ output tile and `BLOCK_K=32`, Level 0 uses one 8.5 KB stage—only 2.7% of\nthe budget. Two stages require 17 KB, or 5.3%, so double buffering is a natural next step.\n\nThis kernel allocates two LDS buffer sets for A and B and turns the K loop into a two-stage software pipeline. Initially, a prologue loads and publishes the first A/B tiles; then, each iteration issues HBM loads into the inactive buffer while WMMA consumes the current one. A workgroup barrier at the end ensures the next buffer is ready to read and the current buffer is safe to overwrite before swapping.\n\n**Why it helps:** Double buffering overlaps memory loads with computation, increasing\ninstruction-level parallelism. The measured performance step stays small here because fills are still\nregister-mediated and each K block drains fully before handoff. Level 2 keeps the same\nstaging; with async direct-to-LDS copies, that is when the benefits of this buffering are\nrealized. The figure below illustrates the double-buffered schedule:\n\n#### Level 1 APIs[#](#level-1-apis)\n\n| API | Purpose | \n|---|---|\n| [allocate_in<segment<0>, Tile, 2>()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/common/util.cuh#L478-L501) | Reserves two tightly packed LDS slots for the current and next operand tiles. | \n| [sync::wait_ds<0>()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L206-L216) | Drains the final LDS reads before the kernel exits. | \n\n#### Level 1 Pseudocode[#](#level-1-pseudocode)\n\n```\nA_LDS[2];\nB_LDS[2];\n\nload(A_LDS[current], A_global);\nload(B_LDS[current], B_global);\nsync::fence();\nsync::arrive();\nsync::wait();                    // Publish the first LDS stage\n\nfor each K tile:\n    load(A_LDS[next], A_global_clamped);\n    load(B_LDS[next], B_global_clamped);\n\n    load(A_reg, A_LDS[current]);\n    load(B_reg, B_LDS[current]);\n    mma_ABt(C, A_reg, B_reg);\n\n    sync::fence();               // Wait for fills and reads\n    sync::arrive();\n    sync::wait();                // Hand off one workgroup barrier\n    swap(current, next);\n```\n\nThe trace in Figure 6 shows how the scheduler interleaves the resident waves at this level:\n\nLike the naive kernel, the scheduler switches among 12 waves from different workgroups. Much of the SIMD’s time is still spent on green VALU work because vector lanes both load data from global memory into registers and write those registers to LDS. Matrix work remains sparse.\n\n### Level 2: Asynchronous HBM Loads ([gemm_async.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/02_gemm_async.cpp#L59-L83))[#](#level-2-asynchronous-hbm-loads-gemm-async-cpp)\n\n- **Performance:** 48% faster than Level 1.\n\nThe AMD Instinct MI455X GPU can copy global memory directly into LDS without passing the data\nthrough the register file. An asynchronous copy is fire-and-forget and retires on `asynccnt`.\nThe wave checks that counter only when the data becomes a dependency, so the fill no longer\nhas to lead the iteration or end in a blanket drain.\n\n| Rung | Fill path | Per K block | \n|---|---|---|\n| Naive | Register-mediated | Two full barriers, two LDS drains, and two global-load drains | \n| Double-buffered | Register-mediated | One full barrier, one LDS drain, and one global-load drain | \n| Asynchronous | Direct to LDS | One split barrier, one LDS drain, and one asynchronous-copy drain | \n\nDirect-to-LDS loads avoid staging through VGPRs, reducing register pressure and freeing register space for the larger output tiles introduced in later levels. They also eliminate a register store-and-writeback path to LDS.\n\n**Why it helps:** Asynchronous loads move data from global memory directly into LDS, avoiding\na round trip through the vector register file. The figure below illustrates how the direct-to-LDS\ncopy overlaps the rest of the pipeline:\n\n#### Level 2 APIs[#](#level-2-apis)\n\n| API | Purpose | \n|---|---|\n| [load_async(A_LDS\\[next\\], A_global)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/global_to_shared.cuh#L375-L441) | Starts a direct global-to-LDS copy without using VGPRs. | \n| [sync::wait_async<0>()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L229-L239) | Drains unordered asynchronous copies before stage handoff. | \n| [sync::arrive() / sync::wait()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L136-L155) | Separates workgroup-barrier signaling from waiting. | \n| [sched::compiler_fence()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sched/sched.cuh#L199-L214) | Prevents the compiler from moving work across a handoff. | \n\n#### Level 2 Pseudocode[#](#level-2-pseudocode)\n\n```\nload_async(A_LDS[current], A_global);\nload_async(B_LDS[current], B_global);\nsync::wait_async<0>();\nsched::compiler_fence();\nsync::arrive();\nsync::wait();                    // Publish the first LDS stage\nsched::compiler_fence();\n\nfor each K tile:\n    load(A_reg, A_LDS[current]);\n    load(B_reg, B_LDS[current]);\n\n    load_async(A_LDS[next], A_global_clamped);\n    load_async(B_LDS[next], B_global_clamped);\n\n    sync::wait_ds<0>();          // Wait for current LDS reads\n    mma_ABt(C, A_reg, B_reg);\n    sync::wait_async<0>();       // Wait for next global-to-LDS fills\n    sched::compiler_fence();\n    sync::arrive();\n    sync::wait();\n    sched::compiler_fence();\n    swap(current, next);\n```\n\nThe trace in Figure 8 shows nine resident wave tracks. There is significantly less time waiting for vector work than in earlier levels because the vector lane only issues direct-to-LDS loads before consuming larger chunks from LDS.\n\n### Level 3: Increasing Output Tile Size to 128 x 128 ([gemm_128x128.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/03_gemm_128x128.cpp#L58-L82))[#](#level-3-increasing-output-tile-size-to-128-x-128-gemm-128x128-cpp)\n\n- **Performance:** 80% faster than Level 2.\n\nFor one output tile, GEMM’s arithmetic intensity is\n\n```\n\\[\n\\frac{2MNK}{MK + KN + MN}.\n\\]\n```\n\nWhen $M=N=K$, this simplifies to $2N/3$. Compute therefore grows cubically with tile size while the required memory movement grows quadratically. A central GEMM design principle is to maximize the output tile handled by each WGP while remaining within the available register and LDS budgets.\n\nLarger tiles also increase data reuse. Four WGPs independently computing adjacent $64 \\times 64$ output tiles must reload shared A and B panels. One WGP computing the same $128 \\times 128$ output region loads each panel once and reuses it across the larger tile, reducing traffic through the memory hierarchy. The trade-off is that larger tiles can reduce WGP occupancy for small problems.\n\nLevel 3 computes a $128 \\times 128$ output tile per WGP. Each wave still owns a $32 \\times 32$ output tile, so the workgroup launches 16 waves.\n\n**Why it helps:** Increasing the output tile size raises arithmetic intensity and reduces\nmemory traffic through greater per-WGP data reuse. Figure 9 shows the schedule for the\nlarger WGP output tile:\n\nThe trace in Figure 10 contains more matrix instructions per wave because each WGP is responsible for a larger output tile:\n\n### Level 4: Increasing Output Tile Size to 256 x 256 ([gemm_256x256.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/04_gemm_256x256.cpp#L62-L86))[#](#level-4-increasing-output-tile-size-to-256-x-256-gemm-256x256-cpp)\n\n- **Performance:** 25% faster than Level 3.\n\nThis kernel computes a $256 \\times 256$ output tile per WGP. Each wave owns a $64 \\times 32$ output tile, and the workgroup launches 16 waves—four per SIMD. This further increases per-WGP reuse through LDS.\n\n#### Level 4 Configuration[#](#level-4-configuration)\n\n```\nBLOCK_M = BLOCK_N = 256;\nWARPS_M = WARPS_N = 4;\nrt_fl<64, 64> C_acc;\n\n// The asynchronous double-buffered K loop is otherwise unchanged.\n```\n\nFigure 11 illustrates the schedule with a $256 \\times 256$ WGP output tile:\n\nThe trace in Figure 12 begins with asynchronous-load issue, followed by large LDS-read blocks and then dense groups of purple WMMA instructions. These groups reflect the increased matrix work per wave:\n\n### Level 5: Deepening the K Stride for WMMA Instructions ([gemm_deepk.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/05_gemm_deepk.cpp#L62-L86))[#](#level-5-deepening-the-k-stride-for-wmma-instructions-gemm-deepk-cpp)\n\n- **Performance:** 19% faster than Level 4.\n\nLevel 1 introduced double buffering between HBM and LDS, but a GEMM kernel can also stall while moving data from LDS to registers. Level 5 adds a two-stage register buffer for that path. HBM-to-LDS loads now bring in $256 \\times 128$ tiles of A and $128 \\times 256$ tiles of B, rather than the $256 \\times 32$ and $32 \\times 256$ tiles used by Level 4.\n\nWithin the outer K loop, an inner loop runs four K=32 substeps. In each substep, a wave loads a $64 \\times 32$ A tile and a $32 \\times 64$ B tile into one register-buffer slot while performing matrix multiplication on the other slot. This overlaps LDS-to-register movement with computation, in addition to the existing overlap between HBM and LDS.\n\n**Why it helps:** A deeper K loop creates finer-grained pipeline stages and more opportunities\nto overlap LDS reads with matrix computation. The figure below illustrates the four-substep\nregister pipeline:\n\n#### Level 5 Pseudocode[#](#level-5-pseudocode)\n\n```\nload_async(A_LDS[current], A_global);\nload_async(B_LDS[current], B_global);\nsync::wait_async<0>();\nsched::compiler_fence();\nsync::arrive();\nsync::wait();\nsched::compiler_fence();\n\nfor each K stage:\n    load(A_reg[0], A_LDS[current][0]);\n    load(B_reg[0], B_LDS[current][0]);\n\n    load_async(A_LDS[next], A_global_clamped);\n    load_async(B_LDS[next], B_global_clamped);\n\n    for substep = 0 .. 2:\n        load(A_reg[next_reg], A_LDS[current][substep + 1]);\n        load(B_reg[next_reg], B_LDS[current][substep + 1]);\n        sync::wait_ds<DS_SUB>();\n        mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n        swap(current_reg, next_reg);\n\n    sync::wait_ds<0>();\n    mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n    sync::wait_async<0>();\n    sched::compiler_fence();\n    sync::arrive();\n    sync::wait();\n    sched::compiler_fence();\n    swap(current, next);\n```\n\nThe trace in Figure 14 shows four resident wave tracks. Instead of large, sequential blocks of LDS reads and compute, each substep interleaves non-matrix work for the next substep with matrix work for the current one:\n\n### Level 6: Accounting for Partitioned LDS ([gemm_segment.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/06_gemm_segment.cpp#L64-L88))[#](#level-6-accounting-for-partitioned-lds-gemm-segment-cpp)\n\n- **Performance:** No measurable change for this benchmark shape.\n\nBank-conflict-free LDS accesses can still serialize through partition conflicts when warps on different SIMD pairs target the same 64 KB LDS partition. The WGP’s five LDS partitions are served by two 256-byte-per-cycle paths, one per SIMD pair. Level 6 places the A and B rings in different partitions so simultaneous operand reads avoid partition conflicts and can use both paths.\n\nFor a detailed treatment of the partitioned LDS organization and its conflict behavior, see\n[A Deep Dive into LDS Optimizations on AMD Instinct MI450 GPUs](https://rocm.blogs.amd.com/software-tools-optimization/mi450-lds-optimization/README.html).\n\nOnly allocation changes: all A subtiles are placed in one blocked array, followed by all B subtiles in a different partition. The K loop and its one split barrier are unchanged from Level 5. Figure 15 compares the Level 5 and Level 6 LDS allocation orders:\n\nFigure 16 shows that the execution order remains unchanged:\n\n**Why it helps:** Although this rung does not improve the measured $8192^3$ BF16 GEMM,\npartition-aware placement reduces serialization for other shapes, workloads, and\nlower-precision data types.\n\n#### Level 6 Pseudocode[#](#level-6-pseudocode)\n\n```\n// A and B stages are allocated in separate LDS partitions.\nload_async(A_LDS[current], A_global);\nload_async(B_LDS[current], B_global);\nsync::wait_async<0>();\nsched::compiler_fence();\nsync::arrive();\nsync::wait();\nsched::compiler_fence();\n\nfor each K stage:\n    load(A_reg[0], A_LDS[current][0]);\n    load(B_reg[0], B_LDS[current][0]);\n\n    load_async(A_LDS[next], A_global_clamped);\n    load_async(B_LDS[next], B_global_clamped);\n\n    for substep = 0 .. 2:\n        load(A_reg[next_reg], A_LDS[current][substep + 1]);\n        load(B_reg[next_reg], B_LDS[current][substep + 1]);\n        sync::wait_ds<DS_SUB>();\n        mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n        swap(current_reg, next_reg);\n\n    sync::wait_ds<0>();\n    mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n    sync::wait_async<0>();\n    sched::compiler_fence();\n    sync::arrive();\n    sync::wait();\n    sched::compiler_fence();\n    swap(current, next);\n```\n\nThe trace in Figure 17 shows the unchanged four-group WMMA order:\n\nOn SIMD0 wave slot 3, the four purple WMMA groups at cycles 2,531–2,941, 3,051–3,642,\n3,746–4,293, and 4,385–4,771 match the four `mma_ABt` substeps from Level 5. Partition-aware\nplacement changes LDS addresses but not the matrix-operation order. The trace cannot reveal\nwhich 64 KB partition a particular LDS access used.\n\n### Level 7: Using TDM Loads ([gemm_tdm.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/07_gemm_tdm.cpp#L61-L85))[#](#level-7-using-tdm-loads-gemm-tdm-cpp)\n\n- **Performance:** 38% faster than Level 6.\n\nThe Tensor Data Mover is an asynchronous data engine available to each WGP. Device-side TDM descriptors describe affine patterns with up to five dimensions and direct the engine to load data into LDS or store it to global memory. This offloads address generation and load instruction issue from the vector lanes.\n\nTDM moves an entire panel from a device-built descriptor. Only two issuer waves post A and B\nwhile the remaining waves continue computing. Wave 0 posts the A descriptor and wave 1 posts\nthe B descriptor so that the transfers use different engine parities. The register ring is\nunchanged, but `tensorcnt` replaces the asynchronous-copy drain, and one deep panel replaces\nfour separately filled subtiles. With two LDS stages, `wait_tdm<S-2>` becomes\n`wait_tdm<0>`, a full drain.\n\nThe kernel also uses padded LDS layouts to produce bank-conflict-free accesses instead of spending vector instructions rearranging data during the fill. Most per-lane load, address-generation, and layout work disappears, leaving more issue bandwidth available for matrix instructions while TDM independently fills the next stage.\n\n**Why it helps:**\n\n1. Only two waves issue tensor loads; the others continue until the data becomes a dependency.\n2. Each wave can request one large two-dimensional transfer instead of many 128-bit global-to-LDS loads.\n3. The engine is launched with only two issue instructions, one from each issuer wave.\n4. Address generation, padding, transposition when needed, and zero filling are offloaded to a dedicated functional unit.\n5. The simpler hazard structure is easier for the compiler to optimize and reduces register pressure.\n\nThe figure below illustrates the descriptor-driven TDM pipeline:\n\n#### Level 7 APIs[#](#level-7-apis)\n\n| API | Purpose | \n|---|---|\n| [tdm::load_async(…)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/tdm.cuh#L201-L257) | Posts one descriptor-driven global-to-LDS panel transfer. | \n| [sync::wait_tdm<0>()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sync/barrier.cuh#L241-L252) | Drains both TDM transfers before the LDS stage is published or reused. | \n\n#### Level 7 Pseudocode[#](#level-7-pseudocode)\n\n```\nif (wave_id == 0)\n    tdm::load_async(A_LDS[current], A_global);\nif (wave_id == 1)\n    tdm::load_async(B_LDS[current], B_global);\nsync::wait_tdm<0>();\nsched::compiler_fence();\nsync::arrive();\nsync::wait();\nsched::compiler_fence();\n\nfor each K stage:\n    load(A_reg[0], A_LDS[current][0]);\n    load(B_reg[0], B_LDS[current][0]);\n\n    if (wave_id == 0)\n        tdm::load_async(A_LDS[next], A_global, count_or_zero);\n    if (wave_id == 1)\n        tdm::load_async(B_LDS[next], B_global, count_or_zero);\n\n    for substep = 0 .. 2:\n        load(A_reg[next_reg], A_LDS[current][substep + 1]);\n        load(B_reg[next_reg], B_LDS[current][substep + 1]);\n        sync::wait_ds<DS_SUB>();\n        mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n        swap(current_reg, next_reg);\n\n    sync::wait_ds<0>();\n    mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n    sync::wait_tdm<0>();\n    sched::compiler_fence();\n    sync::arrive();\n    sync::wait();\n    sched::compiler_fence();\n    swap(current, next);\n```\n\nThe trace in Figure 19 shows the resulting reduction in lane-issued fill work:\n\nOn SIMD0 wave slot 0, decoded WMMA groups at cycles 289–533, 638–1,145, and 1,249–1,732\nmap to the inner loop’s three `mma_ABt` calls; cycles 1,826–2,192 map to the final call.\nThe small green prefix is ordinary VALU work used to build TDM descriptors and offsets.\nDescriptor-driven panel movement removes broad lane-issued fill work, leaving matrix groups\nas the dominant instruction color. Physical slot labels do not identify source `wave_id`.\n\n### Level 8: Using Split Barriers ([gemm_split_bar.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/08_gemm_split_bar.cpp#L71-L95))[#](#level-8-using-split-barriers-gemm-split-bar-cpp)\n\n- **Performance:** 4% faster than Level 7.\n\nA workgroup barrier often prevents one wave from overwriting an LDS buffer while another wave is still reading it. With a conventional barrier, each wave signals completion and immediately waits, leaving early waves idle until the slowest wave arrives.\n\nA split barrier separates the signal from the wait. After a wave completes its final LDS read, its operands are safely held in registers, so it signals that the LDS buffer can be released. The wave then performs its final register-only WMMA before waiting for its peers. Compiler fences keep the WMMA inside this interval; moving it outside the signal and wait would preserve numerical correctness but lose the intended overlap.\n\n**Why it helps:** Split barriers overlap the final K substep with peer-wave arrival, hiding\nsome synchronization latency with useful computation. The figure below illustrates the\nmatrix work placed between barrier arrival and wait:\n\n#### Level 8 Core Scheduling Change[#](#level-8-core-scheduling-change)\n\n```\nsync::wait_ds<0>();\nsync::wait_tdm<0>();\nsync::arrive();                  // Release the LDS stage\nmma_ABt(C, A_reg[final], B_reg[final]);\nsync::wait();                    // Wait for peer waves\n```\n\nThe trace in Figure 21 shows that scheduling interval:\n\nOn SIMD0 wave slot 0, the barrier signal issues at cycle 1,649, followed by 16 WMMA instructions at cycles 1,653–1,776 and the barrier wait at cycle 1,784. Slot 1 repeats the same signal, WMMA, and wait sequence at cycles 1,923–2,058. The final purple block is matrix work deliberately placed inside the split-barrier window.\n\n### Level 9: Using Workgroup Clusters and Multicast ([gemm_wgc_multicast.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/09_gemm_wgc_multicast.cpp#L80-L109))[#](#level-9-using-workgroup-clusters-and-multicast-gemm-wgc-multicast-cpp)\n\n- **Performance:** 7% faster than Level 8.\n\nWorkgroup clusters can contain up to 16 workgroups launched concurrently. Workgroups in a cluster can declare that they will share selected data with other WGPs in that cluster. Repeated L2 requests are then deduplicated through multicast.\n\nThis kernel arranges the workgroups in a $4 \\times 4$ cluster. Each A panel is shared down a cluster column, and each B panel is shared across a cluster row. Four workgroups can therefore consume one L2 return instead of issuing four independent requests. A square cluster deduplicates traffic for both operands. The multicast mask must include the requester and may contain no more than five destinations; an incorrect row or column mask is a correctness error. Figure 22 illustrates how the cluster shares A and B panels:\n\n| One panel consumed by a cluster row | Panels multicast across rows and columns | \n|---|---|\n\nFigure 22: A $4 \\times 4$ cluster reuses A panels down columns and B panels across rows, reducing repeated L2 requests.\n\nThe stage handoff now uses both workgroup and cluster barriers. The final WMMA remains inside both split-barrier windows: wave 0 signals cluster arrival, and then every wave waits.\n\n**Why it helps:** Workgroup clusters and multicast broadcast shared panels from L2, increasing\neffective L2 bandwidth.\n\nThe figure below shows the cluster-scoped synchronization in the pipeline:\n\n#### Level 9 APIs[#](#level-9-apis)\n\n| API | Purpose | \n|---|---|\n| [__cluster_dims__(4, 4, 1)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/09_gemm_wgc_multicast.cpp#L74-L83) | Declares a $4 \\times 4$ workgroup cluster on the kernel. | \n| [cluster::sync() / arrive() / wait()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/cluster/cluster.cuh#L56-L87) | Publishes the prologue and protects later stage handoffs across the cluster. | \n\n#### Level 9 Pseudocode[#](#level-9-pseudocode)\n\n```\nmaskA = cluster::mask(0x1111 << cluster_x);\nmaskB = cluster::mask(0x000F << (4 * cluster_y));\n\nif (wave_id == 0)\n    tdm::load_async(A_LDS[current], A_global, maskA);\nif (wave_id == 1)\n    tdm::load_async(B_LDS[current], B_global, maskB);\nsync::wait_tdm<0>();\ncluster::sync();\n\nfor each K stage:\n    load(A_reg[0], A_LDS[current][0]);\n    load(B_reg[0], B_LDS[current][0]);\n\n    if (wave_id == 0)\n        tdm::load_async(A_LDS[next], A_global, maskA, count_or_zero);\n    if (wave_id == 1)\n        tdm::load_async(B_LDS[next], B_global, maskB, count_or_zero);\n\n    for substep = 0 .. 2:\n        load(A_reg[next_reg], A_LDS[current][substep + 1]);\n        load(B_reg[next_reg], B_LDS[current][substep + 1]);\n        sync::wait_ds<DS_SUB>();\n        mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n        swap(current_reg, next_reg);\n\n    sync::wait_ds<0>();\n    sync::wait_tdm<0>();\n    sync::arrive();              // Signal the workgroup barrier\n    if (wave_id == 0)\n        cluster::arrive();       // Signal the cluster barrier\n    mma_ABt(C, A_reg[current_reg], B_reg[current_reg]);\n    sync::wait();\n    cluster::wait();\n    swap(current, next);\n```\n\nThe trace in Figure 24 shows the final WMMA group inside both barrier windows:\n\nSIMD0 wave slot 0 has a royal-blue `TDM_WAIT` interval around cycles 1,600–2,050. After the\ndrain, the workgroup signal issues at cycle 2,069, the cluster signal at 2,082, and 16 WMMA\ninstructions from the final `mma_ABt` at cycles 2,101–2,221. The purple work after the blue\ninterval is therefore inside both barrier windows. The physical slot label does not identify\nthe A issuer; source `wave_id`, not slot number, selects descriptor posters.\n\n### Level 10: Efficient GEMM Epilogues ([gemm_epilogue.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/10_gemm_epilogue.cpp#L132-L156))[#](#level-10-efficient-gemm-epilogues-gemm-epilogue-cpp)\n\n- **Performance:** 8% faster than Level 9.\n\nLevel 10 stages the C tile through LDS before writing it to global memory. LDS transforms the wave-local, column-major accumulator layout into a row-major tile and enables wider, coalesced stores.\n\n**Why it helps:** Packing the C tile in LDS produces more efficient global-memory store\npatterns at the end of the kernel.\n\nThe figure below illustrates the LDS-staged output epilogue:\n\n#### Level 10 APIs[#](#level-10-apis)\n\n| API | Purpose | \n|---|---|\n| [sched::lock_simd()](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/sched/sched.cuh#L100-L124) | Keeps a wave issuing back-to-back WMMAs on one SIMD. | \n| [store(C_LDS, C_acc)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/shared_to_register.cuh#L595-L635) | Stages scattered accumulator values into LDS. | \n| [store(C_global, C_LDS)](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/include/cdna5/ops/warp/memory/tile/global_to_shared.cuh#L218-L274) | Writes the assembled C tile as wider, coalesced runs. | \n\n#### Level 10 Pseudocode[#](#level-10-pseudocode)\n\n```\nsched::lock_simd();\n\nfor each K stage:\n    // Same TDM, multicast, and split-barrier pipeline as Level 9.\n\nsync::wait_ds<0>();\nsync::wait_tdm<0>();\nsync::arrive();\nsync::wait();\n\nstore(C_LDS, C_acc);             // C: registers -> LDS\nsync::wait_ds<0>();\nsync::arrive();\nsync::wait();\nstore(C_global, C_LDS);          // Coalesced C: LDS -> global\n```\n\nFigure 26 compares the two epilogues at the same time scale:\n\n| Level 9 direct epilogue | Level 10 LDS-staged epilogue | \n|---|---|\n| Narrow stores remain as scattered, per-column transactions after the final matrix work. | Green and orange activity remains interleaved while the waves assemble and stream wider, coalesced stores. | \n\nFigure 26: Direct and LDS-staged GEMM epilogues at the same time scale.\n\nThe Level 10 epilogue replaces direct per-wave stores with an explicit register-to-LDS-to-global gather-and-stream path. LDS reorganizes wave-local accumulator fragments before the global write, creating a more regular and wider store stream.\n\n### Level 11: One Wave per SIMD ([gemm_one_wave.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/11_gemm_one_wave.cpp#L90-L114))[#](#level-11-one-wave-per-simd-gemm-one-wave-cpp)\n\n**Performance:** 6% faster than Level 10.\n\nLevel 11 keeps the $256 \\times 256$ workgroup tile but replaces the $4 \\times 4$ wave grid with a $2 \\times 2$ grid. Each of the four waves now owns a $128 \\times 128$ output tile, placing one wave on each SIMD. This sacrifices occupancy in exchange for greater operand reuse within each wave.\n\nThe register-operand pipeline expands from two slots to three. Two K=32 substeps are prefetched before the first WMMA, allowing later LDS loads and matrix operations to be interleaved even though there are no other resident waves to hide latency. For each K block, the wave waits for the current TDM stage, prefetches substeps 0 and 1 into two register slots, executes WMMA for substep 0 while loading a later substep into the free slot, rotates the three-slot ring through all four substeps, signals the barriers around the final WMMA, and then advances to the next stage.\n\n**Why it helps:** The larger wave-local tile increases register- and LDS-level data reuse.\nEach substep can issue 64 WMMA instructions while the three-slot pipeline maintains overlap.\n\nThe figure below illustrates the one-wave-per-SIMD schedule:\n\n#### Level 11 Pseudocode[#](#level-11-pseudocode)\n\n```\nsched::lock_simd();\n\nif (wave_id == 0)\n    tdm::load_async(A_LDS[current], A_global, maskA);\nif (wave_id == 1)\n    tdm::load_async(B_LDS[current], B_global, maskB);\nsync::wait_tdm<0>();\ncluster::sync();\n\nfor each K stage:\n    load(A_reg[0], A_LDS[current][0]);\n    load(B_reg[0], B_LDS[current][0]);\n    load(A_reg[1], A_LDS[current][1]);\n    load(B_reg[1], B_LDS[current][1]);\n\n    if (wave_id == 0)\n        tdm::load_async(A_LDS[next], A_global, maskA, count_or_zero);\n    if (wave_id == 1)\n        tdm::load_async(B_LDS[next], B_global, maskB, count_or_zero);\n\n    for substep = 0 .. 2:\n        if substep + 2 < 4:\n            load(A_reg[(substep + 2) % 3],\n                 A_LDS[current][substep + 2]);\n            load(B_reg[(substep + 2) % 3],\n                 B_LDS[current][substep + 2]);\n        sync::wait_ds<DS_SUB>();\n        mma_ABt(C, A_reg[substep % 3], B_reg[substep % 3]);\n\n    sync::wait_ds<0>();\n    sync::wait_tdm<0>();\n    sync::arrive();\n    if (wave_id == 0)\n        cluster::arrive();\n    mma_ABt(C, A_reg[final], B_reg[final]);\n    sync::wait();\n    cluster::wait();\n    swap(current, next);\n```\n\nFigure 28 shows how the orange LDS activity near cycles 100–350 primes the register ring. The\nthree long purple groups at cycles 328–787, 901–1,353, and 1,368–1,865 each contain 64\ndecoded WMMA instructions from an inner-loop `mma_ABt`. Royal blue at cycles 1,869–2,100 is\nthe TDM drain. After the two barrier signals, the final 64-instruction WMMA group runs at\ncycles 2,150–2,596 before the waits:\n\n### Level 12: Two Waves per SIMD ([gemm_two_waves.cpp](https://github.com/HazyResearch/HipKittens/blob/1602364f4f40b5caeec0ccbbaf9ca31f784f1599/kernels/cdna5/gemm/bf16fp32/gfx1250/12_gemm_two_waves.cpp#L119-L143))[#](#level-12-two-waves-per-simd-gemm-two-waves-cpp)\n\n- **Performance:** 6% faster than Level 11.\n\nThe final rung keeps the $256 \\times 256$ workgroup tile and replaces Level 11’s $2 \\times 2$ wave grid with a $4 \\times 2$ grid. Each wave owns a $64 \\times 128$ output tile, the workgroup grows from four to eight waves, and two waves run on each SIMD.\n\nA $64 \\times 128$ accumulator requires 256 registers instead of 512. With 256 threads, each lane has 512 registers available instead of 1024, so the operand ring drops from three slots to two. Level 11 needed the third slot to keep loads in flight; in Level 12, the second resident wave hides that latency more effectively.\n\nThe operand feed uses `sched_group_barrier` instead of `compiler_fence`. It requests six LDS\nreads followed by eight matrix operations, repeated four times to cover the 24 reads and\n32 matrix operations in one substep.\n\n**Why it helps:** Two co-resident waves let the hardware scheduler issue work from one wave\nwhile the other is waiting for data, preserving matrix utilization while improving latency\nhiding.\n\nThe figure below illustrates this schedule:\n\n#### Level 12 Helpers[#](#level-12-helpers)\n\n| Helper | Purpose | \n|---|---|\n| `mma_ABt_base(...)` | Computes one output fragment, allowing the final MMA to be split into groups of 12 and 20 instructions. | \n| `pin_interleave(...)` | A kernel helper around `sched_group_barrier` that pins LDS reads and WMMA operations into a specific issue order. | \n\n#### Level 12 Pseudocode[#](#level-12-pseudocode)\n\n```\nsched::lock_simd();\n\nif (wave_id == 0)\n    tdm::load_async(A_LDS[current], A_global, maskA);\nif (wave_id == 1)\n    tdm::load_async(B_LDS[current], B_global, maskB);\nsync::wait_tdm<0>();\ncluster::sync();\n\nload(A_reg[0], A_LDS[current][0]);\nload(B_reg[0], B_LDS[current][0]);\n\nfor each K stage:\n    if (wave_id == 0)\n        tdm::load_async(A_LDS[next], A_global, maskA, count_or_zero);\n    if (wave_id == 1)\n        tdm::load_async(B_LDS[next], B_global, maskB, count_or_zero);\n\n    for substep = 0 .. 2:\n        load(A_reg[(substep + 1) % 2],\n             A_LDS[current][substep + 1]);\n        load(B_reg[(substep + 1) % 2],\n             B_LDS[current][substep + 1]);\n        mma_ABt(C, A_reg[substep % 2], B_reg[substep % 2]);\n        pin_interleave();\n\n    mma_ABt_base(...) x 12;\n    sync::wait_ds<0>();\n    sync::wait_tdm<0>();\n    sync::arrive();\n    if (wave_id == 0)\n        cluster::arrive();\n    sync::wait();\n\n    load(A_reg[0], A_LDS[next][0]);\n    load(B_reg[0], B_LDS[next][0]);\n    mma_ABt_base(...) x 20;\n    pin_interleave<5, 6>();\n    cluster::wait();\n    swap(current, next);\n```\n\nThe trace in Figure 30 shows the WGP cleanly interleaving instructions from the two waves resident on the SIMD. It begins with interleaved TDM descriptor setup and issue, followed by a long sequence of WMMA and LDS operations alternating between the waves. This maintains the strong WMMA utilization of Level 11 while providing more opportunities to hide latency by switching waves:\n\n## Summary[#](#summary)\n\nMany kernel-scheduling patterns that delivered high performance on the AMD Instinct MI350 and MI355X GPUs—including four-wave interleaving and eight- or sixteen-wave ping-pong schedules—translate directly to Helios. Despite the architectural changes described here, kernel developers can retain the core scheduling ideas from earlier AMD GPU generations while taking advantage of partitioned LDS, TDM, and workgroup multicast.\n\nWe plan to continue updating [HipKittens](https://github.com/HazyResearch/HipKittens) with\nadditional Helios kernels, optimizations, and technical discussions. Testing was performed by the authors on early-access hardware. Results may vary based on\nconfiguration, usage, software version, firmware, and optimizations.\n\n## Test Configuration[#](#test-configuration)\n\n- GPU: AMD Instinct™ MI455X GPU\n- Workload: BF16 GEMM with $M=N=K=8192$\n- Methodology: 500 warm-up iterations and 100 measured iterations, with L2 cache flush\n- Kernel implementation: HipKittens HIP/C++\n- Profiling: AMD Advanced Thread Trace with the ROCm Systems Profiler\n\n## Acknowledgements[#](#acknowledgements)\n\nFinally, we thank the AMD University Partnerships team for supporting this work, including Hugo Andrade, Preethi Jayadev, and Tom Papatheodore, and AMD’s Triton and HipBLASLt/TensileLite teams. We also thank our AMD colleagues Lei Zhang, Stanley Winata, Xiaohu Guo, Kumar Deepak, Bryant Nelson, Alex Brown, Brad Nemanich, Brian Shi, Majed Sujon, Ahmed Eltantawy, and Kyle Wang for their feedback and support on this work.\n\n## Disclaimers[#](#disclaimers)\n\nThe information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. The information contained herein is subject to change and may be rendered inaccurate for many reasons, including but not limited to product and roadmap changes, component and motherboard version changes, new model and/or product releases, product differences between differing manufacturers, software changes, BIOS flashes, firmware upgrades, or the like. Any computer system has risks of security vulnerabilities that cannot be completely prevented or mitigated. AMD assumes no obligation to update or otherwise correct or revise this information. However, AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes. THIS INFORMATION IS PROVIDED ‘AS IS.” AMD MAKES NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE CONTENTS HEREOF AND ASSUMES NO RESPONSIBILITY FOR ANY INACCURACIES, ERRORS, OR OMISSIONS THAT MAY APPEAR IN THIS INFORMATION. AMD SPECIFICALLY DISCLAIMS ANY IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. IN NO EVENT WILL AMD BE LIABLE TO ANY PERSON FOR ANY RELIANCE, DIRECT, INDIRECT, SPECIAL, OR OTHER CONSEQUENTIAL DAMAGES ARISING FROM THE USE OF ANY INFORMATION CONTAINED HEREIN, EVEN IF AMD IS EXPRESSLY ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. © 2026 Advanced Micro Devices, Inc. All rights reserved\n\nThird-party content is licensed to you directly by the third party that owns the content and is not licensed to you by AMD. ALL LINKED THIRD-PARTY CONTENT IS PROVIDED “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF SUCH THIRD-PARTY CONTENT IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THIRD-PARTY CONTENT.", "url": "https://wpnews.pro/news/an-educational-gemm-ladder-for-helios-gpus", "canonical_source": "https://rocm.blogs.amd.com/software-tools-optimization/hipkittens-gemm-ladder/README.html", "published_at": "2026-09-14 00:00:00+00:00", "updated_at": "2026-09-14 17:24:24.245377+00:00", "lang": "en", "topics": ["ai-chips", "ai-infrastructure", "ai-research", "developer-tools"], "entities": ["AMD", "AMD Instinct MI455X", "Helios", "HipKittens", "HazyResearch", "Simon Boehm", "CDNA 5"], "alternates": {"html": "https://wpnews.pro/news/an-educational-gemm-ladder-for-helios-gpus", "markdown": "https://wpnews.pro/news/an-educational-gemm-ladder-for-helios-gpus.md", "text": "https://wpnews.pro/news/an-educational-gemm-ladder-for-helios-gpus.txt", "jsonld": "https://wpnews.pro/news/an-educational-gemm-ladder-for-helios-gpus.jsonld"}}