{"slug": "cuda-matrix-multiplication-optimization-from-naive-to-near-cublas", "title": "CUDA Matrix Multiplication Optimization: From Naive to Near-cuBLAS", "summary": "A naive CUDA matrix multiplication kernel on an NVIDIA RTX A6000 GPU achieves only 309 GFLOP/s, or 1.3% of the 23,270 GFLOP/s single-precision peak that NVIDIA's cuBLAS library reaches, according to an analysis of six optimization stages that close the gap to 93.7% of cuBLAS performance. The 4096×4096 single-precision benchmarks, run on the Ampere GA102 chip with 48 GB GDDR6X under CUDA 12.x, attribute the entire gap to data movement through the memory hierarchy rather than arithmetic, with each stage targeting global memory coalescing, shared memory reuse, register-level reuse, or vectorized loads.", "body_md": "Matrix multiplication is the single most important kernel in GPU computing.\nEvery linear layer, every convolution, every attention head in every transformer —\nall of them reduce to a general matrix multiply (GEMM) at their core.\nWhen you call `torch.mm()` or `tf.matmul()`, you’re invoking a kernel\nthat has been optimized for decades. NVIDIA’s cuBLAS library achieves over\n90% of a GPU’s theoretical peak floating-point throughput on this one\noperation. Understanding how it gets there is understanding how GPUs actually work.\n\nThe textbook algorithm is O(N³): computing the product of two N×N\nmatrices requires 2N³ floating-point operations. GPUs should be perfect\nfor this — thousands of cores running in parallel, hundreds of gigabytes\nper second of memory bandwidth. An A6000 GPU can deliver 23,270 GFLOP/s of\nsingle-precision throughput. But a naive CUDA implementation of matrix\nmultiplication achieves only about 309 GFLOP/s — **1.3% of peak**. The\nalgorithm is correct. The arithmetic is identical. The entire gap is about how\ndata moves through the memory hierarchy.\n\nThis article walks through six optimization stages that close that gap, from 1.3% to 93.7% of cuBLAS performance. Each stage targets a specific hardware bottleneck — global memory coalescing, shared memory reuse, register-level data reuse, vectorized loads — and each one comes with an interactive visualization so you can see exactly what changes and why it matters. By the end, you’ll have a concrete mental model for how high-performance GPU kernels are designed, and why the same principles apply to every operation in deep learning.\n\nAll performance numbers in this article are measured on an NVIDIA RTX A6000 GPU (Ampere architecture, GA102 chip) with 48 GB GDDR6X running CUDA 12.x. The matrices are 4096×4096 single-precision floats unless otherwise noted. The cuBLAS baseline for this configuration is 23,270 GFLOP/s.\n\n## [The Naive Triple Loop](#the-naive-triple-loop)\n\nThe textbook matrix multiplication algorithm is three nested loops. To compute C = A × B where A is M×K and B is K×N, each element of the output matrix C[i][j] is the dot product of row i of A and column j of B:\n\nThis requires exactly 2·M·N·K floating-point operations (one multiply and one add per iteration of the inner loop). For square matrices of size 4096, that’s 2 × 4096³ = 137.4 billion FLOPs. Each element of C requires K multiply-accumulate operations — a full dot product between a row of A and a column of B.\n\nOn a GPU, the naive approach maps naturally: assign each thread to compute one element of C. Thread (i, j) reads the entire row i of A and the entire column j of B, accumulates the dot product, and writes the result. With 4096 × 4096 = 16.7 million threads running across dozens of SMs, this seems like it should saturate the hardware. It doesn’t — not even close.\n\nThe CUDA kernel looks like this:\n\nEach thread loops through the entire K dimension, performing one multiply-add per iteration. The arithmetic is correct and the parallelism is massive. But the kernel achieves only 309 GFLOP/s — less than 2% of what this GPU can deliver.\n\nThe visualization above shows the access pattern for a single output element. To compute C[i][j], the thread must read K elements from row i of A and K elements from column j of B. When thousands of threads do this simultaneously, the total memory traffic is staggering: every thread independently reads the full K dimension from both A and B, with no data sharing between threads. The access pattern for B is especially wasteful — and the reason comes down to how matrices are stored in memory.\n\n## [Why Naive is Cache-Hostile](#why-naive-is-cache-hostile)\n\nA 2D matrix is stored in memory as a 1D array, laid out row by row. In row-major\norder, element [i][j] of an N-column matrix lives at memory address\n`base + i*N + j`. Consecutive elements within the same row are adjacent in\nmemory. Consecutive elements within the same column are N positions apart.\n\nThis layout has a profound consequence for the naive algorithm. When a thread\nreads row i of A, it accesses `A[i][0], A[i][1], A[i][2], ...` —\nconsecutive memory addresses that will be served from a single cache line fetch.\nBut when it reads column j of B, it accesses `B[0][j], B[1][j], B[2][j], ...`\n— each element is N positions apart in memory. Every single access to B\nmisses the cache and fetches a new cache line, of which only one element\n(4 bytes out of 64 or 128) is actually used. The remaining bytes in each cache\nline are loaded and immediately discarded.\n\nConsider a concrete example with N=4096. A cache line is 128 bytes (32 floats) on modern GPUs. When accessing column j of B:\n\n- `B[0][j]` loads a cache line covering`B[0][j]` through`B[0][j+31]` —\n32 elements from row 0. Only`B[0][j]` is needed. 31 elements wasted.\n- `B[1][j]` is 4096 positions away in memory — an entirely different cache\nline. Another 128-byte fetch for 4 useful bytes.\n- Repeat for all 4096 rows: 4096 cache line fetches, each using 3.1% of the loaded data.\n\nThis is the fundamental problem: the naive algorithm has a\n**compute-to-memory ratio of O(1)**. Each loaded value is used exactly once and\nthen evicted. For every multiply-add (2 FLOPs), we load at least one value from\nslow memory (8 bytes in FP32 round-trip for loads and stores). That puts us\nfirmly in the memory-bound regime, nowhere near the GPU’s compute\ncapability.\n\nTo quantify: the A6000 has 768 GB/s of memory bandwidth. At 8 bytes per FLOP pair, the maximum achievable throughput is 768/8 = 96 GFLOP/s if we’re purely memory-bound. The naive kernel gets 309 GFLOP/s only because the L2 cache absorbs some of the redundant reads. But we need to get to 23,270 GFLOP/s — that requires a 242× improvement in the compute-to-memory ratio.\n\nThe fix is tiling: divide the matrices into small blocks that fit in fast memory, load each block once, and reuse it for multiple output elements. If a tile is T×T, each loaded value gets reused T times instead of once. The compute-to-memory ratio rises from O(1) to O(T) — and that’s the difference between 1% and 90% of peak performance.\n\n## [Tiled Matrix Multiplication: The Fix](#tiled-matrix-multiplication-the-fix)\n\nThe tiling algorithm divides A, B, and C into tiles of size T×T. Instead of computing each output element independently, we compute an entire T×T block of the output at once. The output tile C[ti][tj] is the sum of partial products across k-slices:\n\nFor each k-slice, we load a T×T tile from A and a T×T tile from B into fast memory. Then we multiply these two small tiles together (a T×T matrix multiply using 2T³ FLOPs) and accumulate the result into the output tile. We repeat this for each k-slice until the entire dot product dimension has been covered.\n\nThe key insight: each tile of A and B contains T² elements and is loaded once from slow memory. But each element participates in T multiply-add operations during the tile multiply. So the data reuse factor is T — we get T times more compute per byte loaded.\n\nLet’s make this concrete with T=32. For one output tile of C:\n\n- **Data loaded** : 2 × 32² = 2,048 floats per k-step, times K/32\nk-steps = 2,048 × 128 = 262,144 floats total for 4096×4096\nmatrices\n- **Compute** : 2 × 32³ per k-step × 128 k-steps = 8.4 million\nFLOPs\n- **Compute-to-memory ratio** : 8.4M / (262K × 4 bytes) = 8.0 FLOPs per\nbyte — an 8× improvement over naive\n\nOn a CPU, \"fast memory\" means L1 cache. With 32 KB of L1 cache, we can fit tiles up to about 64×64 in FP32 (two tiles of 64×64×4 bytes = 32 KB). On a GPU, the equivalent is shared memory — an explicitly managed on-chip scratchpad that sits right next to the compute units. The tiling principle is identical on both architectures. What changes is the memory hierarchy and the programming model for exploiting it.\n\nThe transition from CPU to GPU tiling introduces several new concerns: how threads cooperate to load tiles, how memory accesses are coalesced across warps, and how the barrier synchronization between load and compute phases is managed. These GPU-specific details are what make the next three sections critical.\n\n## [Translation to GPU: Warps and Coalescing](#translation-to-gpu-warps-and-coalescing)\n\nBefore diving into shared memory tiling on the GPU, we need to understand how\nGPU memory access works at the hardware level. The critical concept is\n**memory coalescing**.\n\nA GPU organizes threads into **warps** of 32 threads that execute in lockstep.\nWhen threads in a warp issue memory requests, the hardware tries to combine\n(coalesce) their individual 4-byte requests into as few memory transactions as\npossible. If all 32 threads access 32 consecutive 4-byte addresses, the\nhardware can serve this as a single 128-byte transaction. If the 32 threads\naccess scattered addresses, each thread’s request may become a separate\ntransaction — up to 32 individual fetches for data that could have been\nserved in one.\n\nThe coalescing rules are straightforward:\n\n- **Best case** : thread i accesses address`base + i * 4` (32 consecutive\nfloats). One 128-byte transaction serves all 32 threads.\n- **Offset case** : thread i accesses address`base + i * 4 + offset` . If the\naddresses span two cache lines, two transactions are needed.\n- **Strided case** : thread i accesses address`base + i * stride * 4` . If stride\nis large (e.g., N=4096), each thread hits a different cache line. 32\ntransactions instead of 1 — a 32× bandwidth penalty.\n- **Random case** : addresses are unrelated. Up to 32 separate transactions, each\nfetching 128 bytes for 4 useful bytes.\n\nIn the naive GPU matmul, each thread computes one element of C. Threads within a warp compute adjacent columns of the same row: thread 0 computes C[i][0], thread 1 computes C[i][1], and so on. When these threads load their values from A, they all read the same row — this is fine (it can be broadcast). But when they load from B, thread 0 reads B[k][0], thread 1 reads B[k][1], etc. In row-major storage, B[k][0] and B[k][1] are adjacent in memory — so loads from B are actually coalesced.\n\nThe problem is subtler than it appears. While B accesses coalesce, the total\nnumber of global memory transactions is still enormous because there is\n**zero data reuse** across threads. Every thread independently loads the entire\nk-dimension from both A and B. With M×N threads each loading K elements\nfrom A and K elements from B, the total global memory traffic is\nO(M·N·K) — proportional to the number of FLOPs rather than\nthe matrix size. The memory bus becomes the bottleneck long before the compute\nunits are saturated.\n\nThis is why the naive CUDA kernel achieves only 309 GFLOP/s on an A6000 — 1.3% of its 23,270 GFLOP/s peak. The arithmetic units sit idle, waiting for data to arrive from global memory (HBM). The solution is the same as on CPU: tile the computation so that data loaded from global memory is reused many times. On GPU, the fast reuse buffer is shared memory.\n\n## [Shared Memory: The GPU Tile Cache](#shared-memory-the-gpu-tile-cache)\n\nEach streaming multiprocessor (SM) on a GPU has a bank of on-chip memory called\n**shared memory** — typically 48 to 228 KB depending on the GPU\ngeneration. This memory sits physically next to the compute cores, with bandwidth\nexceeding 12,000 GB/s — over 16× faster than global memory (HBM).\nUnlike L1/L2 caches, shared memory is explicitly managed by the programmer. You\ndecide exactly what goes in and when.\n\nThe shared memory tiling kernel follows a simple pattern:\n\n1. **Cooperative load** : All threads in a thread block work together to load a\ntile of A and a tile of B from global memory (HBM) into shared memory. Each\nthread loads one or a few elements.\n2. **Barrier synchronize** : Call`__syncthreads()` to ensure every thread’s\nload has completed before any thread starts reading from the shared buffer.\n3. **Compute from shared memory** : Each thread computes its portion of the output\ntile by reading from the fast shared memory arrays instead of slow global\nmemory.\n4. **Advance to next tile** : Move the tile window along the K dimension and\nrepeat steps 1–3.\n\nThe kernel looks something like this:\n\nThis pattern reduces global memory traffic by a factor of T (the tile size). Instead of each thread independently loading the full K dimension, the entire thread block cooperatively loads a T×T tile and every thread in the block reuses it. With T=32, the global memory traffic drops from O(M·N·K) to O(M·N·K / T) — a 32× reduction.\n\nFor more on how shared memory fits into the broader GPU memory hierarchy, see the [GPU Memory Hierarchy](https://www.abhik.ai/concepts/gpu-computing/memory-hierarchy) concept page. For details on how SMs are organized, see [Streaming Multiprocessor](https://www.abhik.ai/concepts/gpu-computing/shared-multiprocessor).\n\nShared memory tiling alone brings performance from around 1,987 to 2,980 GFLOP/s — a 1.5× improvement. That might seem modest given the 32× reduction in global memory traffic, but it’s because the bottleneck has shifted. Global memory bandwidth is no longer the primary constraint. Now the limiting factors are:\n\n- **Per-thread compute density** : each thread computes just one output element,\nso the ratio of shared memory loads to FLOPs is still low. Each iteration of\nthe inner loop performs 2 FLOPs but requires 2 shared memory reads.\n- **Shared memory bank conflicts** : shared memory is divided into 32 banks. When\nmultiple threads in a warp access the same bank simultaneously, their requests\nare serialized. In the naive tile layout, column-wise access patterns can\ntrigger 32-way bank conflicts.\n- **Thread block occupancy** : with 32×32 = 1,024 threads per block, only\none or two blocks can reside on each SM simultaneously, limiting the\nscheduler’s ability to hide memory latency by switching between warps.\n\nThe next optimization addresses the biggest of these: per-thread compute density.\n\n## [Register Tiling: More Work Per Thread](#register-tiling-more-work-per-thread)\n\nIn the basic shared memory kernel, each thread computes exactly one element of the output matrix. It loads one value from A’s tile and one value from B’s tile per iteration of the inner loop, performs one multiply-add, and moves on. That’s a 1:1 ratio of memory loads to FLOPs — still heavily memory-bound, just from shared memory instead of global memory.\n\nThe solution is **register tiling**: have each thread compute a small block of\noutput elements instead of just one. If a thread computes a TM×TN block\n(say, 8×8 = 64 elements), it loads TM values from A’s tile into\nregisters and TN values from B’s tile into registers. Then it computes\ntheir **outer product** — TM×TN = 64 multiply-adds — from\nthose TM+TN = 16 loaded values. Each loaded value is reused either TM or TN\ntimes.\n\nThe outer product approach works as follows. For each position k within the shared memory tile:\n\n1. Load `regM[0..7] = As[threadRow*8 + 0..7][k]` — 8 values from A’s tile\n2. Load `regN[0..7] = Bs[k][threadCol*8 + 0..7]` — 8 values from B’s tile\n3. Compute the 8×8 outer product: `result[m][n] += regM[m] * regN[n]` for\nall 64 pairs\n\nStep 3 performs 64 multiply-adds (128 FLOPs) from just 16 shared memory loads. That’s 8 FLOPs per load, compared to 2 FLOPs per load in the basic kernel.\n\nThe arithmetic intensity improvement is dramatic:\n\n- **1×1 tiling** (basic kernel): 2 shared memory loads per 2 FLOPs =\n1.0 FLOP/load\n- **8×8 tiling** : 16 shared memory loads per 128 FLOPs = 8.0 FLOPs/load\n- **Reduction in shared memory traffic** : 8× fewer loads per output element\n\nBut that’s only the shared memory side. The effect cascades to global memory too. With 8×8 register tiling, each thread block computes a much larger output tile with the same shared memory capacity. A 128×128 output tile is computed by 256 threads (16×16 blocks of 8×8 per thread), using the same 32×32 shared memory tiles as before. The global memory loads per output element drop by roughly 32× compared to the naive kernel.\n\nGoing from 1×1 to 8×8 per-thread output with optimized block\ndimensions (128×128 output tile, 8×8 per thread, 16×16 = 256\nthreads per block) pushes performance to 15,972 GFLOP/s —\n**68.7% of cuBLAS**. This single optimization delivers the largest performance\njump in the entire pipeline.\n\nThe register file on a GPU is the fastest memory available — zero latency, no bank conflicts, and massive aggregate bandwidth. An A6000 SM has 256 KB of register file, far larger than its shared memory. Each thread in our 8×8 tiling kernel uses 64 registers for the accumulator (one per output element), plus 16 for the loaded values, plus a few dozen for loop variables and addresses — roughly 100 registers per thread. With 256 threads per block, that’s 25,600 registers — well within the SM’s capacity.\n\nBy staging data into registers via the outer product pattern, we exploit this fast storage to its fullest. The bottleneck shifts once more: now the limiting factor is how efficiently we load data from global memory into shared memory.\n\n## [Vectorized Memory Access](#vectorized-memory-access)\n\nEven with register tiling, the kernel still loads data from global memory one float at a time — 32-bit (4-byte) loads. But the GPU’s memory bus is 128 bits wide per thread. Loading a single float uses only 25% of the available bus width. The remaining 75% is wasted on every transaction.\n\nThe fix is **vectorized loads**: instead of loading one `float` (4 bytes) per\ninstruction, load a `float4` (16 bytes) — four consecutive floats in a\nsingle 128-bit transaction. This requires the data to be 16-byte aligned and\ncontiguous in memory.\n\nThe code change is straightforward:\n\nFor tile loads from A, this works naturally in row-major layout: consecutive elements within a row are contiguous. But for the shared memory layout, there’s a subtlety. During the compute phase, each thread needs to read down a column of A’s shared memory tile (to gather TM values for the outer product). Column access in a row-major shared memory array triggers bank conflicts and prevents vectorized reads.\n\nThe solution is to **transpose A’s tile** in shared memory during the load\nphase. Instead of storing A’s tile in its natural row-major order, we store\nit transposed so that what was a column becomes a row. Now, column access in the\nlogical matrix becomes row access in physical shared memory — contiguous,\nbank-conflict-free, and vectorizable.\n\nThe transpose costs nothing extra: the data is being loaded from global memory anyway. We just write it to a different shared memory offset during the cooperative load phase. The global memory access pattern stays coalesced (reading rows of A), and the shared memory layout is rearranged for optimal compute-phase access.\n\nThe combination of `float4` global memory loads and shared memory transpose\npushes performance to 18,237 GFLOP/s — **78.4% of cuBLAS**. The final gap\nis closed by warp-level tiling optimizations: arranging the thread-to-data\nmapping within each warp so that threads within a warp access values that are\nphysically close in the register file, maximizing register-level locality.\n\nWith warp tiling, the kernel reaches 21,779 GFLOP/s —\n**93.7% of cuBLAS** on an A6000 GPU. Warp tiling ensures that the 32 threads in\na warp collectively work on a contiguous sub-tile of the output, so their shared\nmemory access patterns are maximally coalesced and their register usage patterns\nallow the compiler to minimize register spills.\n\n## [The Full Optimization Pipeline](#the-full-optimization-pipeline)\n\nFrom 309 GFLOP/s to 21,779 GFLOP/s — a 70× improvement on the exact same hardware, computing the exact same result. Each optimization targets a specific bottleneck in the memory hierarchy:\n\n| Kernel | GFLOP/s | % of cuBLAS | Key Optimization | \n|---|---|---|---|\n| Naive | 309 | 1.3% | Baseline — one thread per output element | \n| Global coalescing | 1,987 | 8.5% | Coalesced global memory reads | \n| Shared memory tiling | 2,980 | 12.8% | Cooperative tile loads into SMEM | \n| 1D block tiling | 8,474 | 36.4% | Multiple results per thread (1D) | \n| 2D block tiling | 15,972 | 68.7% | 8×8 outer product per thread | \n| Vectorized loads | 18,237 | 78.4% | float4 loads + SMEM transpose | \n| Warp tiling | 21,779 | 93.7% | Warp-level register locality | \n\nNotice the pattern: the biggest gains come from **reducing data movement**, not\nfrom increasing compute. The GPU had enough arithmetic throughput from the start.\nEvery optimization is about feeding data to the compute units more efficiently\n— loading it fewer times, loading it in larger chunks, and keeping it in\nfaster memory for longer.\n\nThe remaining 6.3% gap comes from cuBLAS’s collection of hundreds of specialized kernel variants. cuBLAS includes kernels hand-tuned for specific matrix sizes, data types, GPU architectures, and even specific aspect ratios (e.g., tall-skinny vs. square matrices). It uses autotuning to select the fastest variant at runtime. Reaching that last few percent requires the same kind of exhaustive, architecture-specific specialization — but the principles behind each optimization are exactly what we’ve covered here.\n\n## [Beyond Tiling: Async, Tensor Cores, and Scheduling](#beyond-tiling-async-tensor-cores-and-scheduling)\n\nThe techniques above reach 93.7% of cuBLAS on Ampere-generation GPUs using standard CUDA cores. Modern GPU architectures — Hopper (H100) and Blackwell (B200) — add dedicated hardware accelerators that push GEMM performance another 10–40× beyond what CUDA cores alone can deliver.\n\nThese accelerators change the performance landscape entirely:\n\n- **Tensor Cores** perform entire small matrix multiplies (e.g., 16×16×16\nin FP16) in a single instruction, delivering 15× the throughput of CUDA\ncores for supported data types. On H100, tensor cores provide 989 TFLOP/s in\nFP16, compared to 67 TFLOP/s for CUDA cores.\n- **TMA (Tensor Memory Accelerator)** offloads global-to-shared-memory copies to\na dedicated DMA engine, freeing the CUDA cores to compute while data transfers\nhappen asynchronously. This eliminates the \"load then compute\" serialization\nthat limits our shared memory kernel.\n- **Warp Group MMA** allows multiple warps (up to 4, forming a 128-thread warp\ngroup) to cooperatively feed data to tensor cores, keeping them saturated even\nas their throughput increases with each GPU generation.\n- **Persistent kernels** keep thread blocks alive across multiple output tiles,\namortizing launch overhead and enabling software pipelining of memory and\ncompute stages. A persistent kernel processes its entire tile assignment\nwithout returning to the host.\n\nThe pipeline on Hopper uses a multi-stage software pipeline: while one warp group computes the current tile using tensor cores, another warp group prefetches the next tile from global memory via TMA. This overlap of compute and memory access hides latency entirely — the compute units never stall waiting for data.\n\nFor details on tensor core architecture, see the [Tensor Cores](https://www.abhik.ai/concepts/gpu-computing/tensor-cores) concept page. For how high-bandwidth memory (HBM) feeds these accelerators, see [HBM Memory](https://www.abhik.ai/concepts/gpu-computing/hbm-memory).\n\nState-of-the-art GEMM kernels on H100 GPUs reach 764+ TFLOP/s in FP16 — over 35× faster than our best CUDA-core kernel on A6000. This performance comes not from better algorithms but from hardware that is purpose-built for the tiled matrix multiply pattern we’ve been studying. The tiling hierarchy (global → shared → register → tensor core) remains the same; the hardware just executes each level faster.\n\n## [What This Means for Deep Learning](#what-this-means-for-deep-learning)\n\nThis isn’t academic. Matrix multiplication is the computational bottleneck of every neural network, and the optimizations above directly determine how fast your models train and run inference.\n\nEvery major deep learning operation is a GEMM in disguise:\n\n- **Linear layers** :`output = input @ weights` is a direct matrix multiply. A\ntransformer with hidden size 4096 and batch size 32 performs a 32×4096\n× 4096×4096 GEMM for every linear layer. A 70B parameter model has\napproximately 80 such layers, each with multiple linear projections.\n- **Convolutions** : the im2col algorithm reshapes input patches into a matrix,\nconverting convolution into GEMM. A ResNet-50 forward pass performs over 3.8\nbillion multiply-adds, nearly all of which are GEMMs after im2col\ntransformation. Alternatively, Winograd and FFT-based methods also rely on\nmatrix multiplies at their core.\n- **Attention** : the self-attention mechanism requires two GEMMs per head —\nQ·K⊃T to compute attention scores and Attention·V to compute\nthe output. For a sequence length of 2048 with 32 heads, that’s 64\nseparate matrix multiplies per layer.\n- **Gradient computation** : backpropagation doubles the GEMM count. Each\nforward-pass GEMM produces two GEMMs in the backward pass (one for the input\ngradient, one for the weight gradient). Training is roughly 3× the\ncompute of inference.\n\nWhen you call `torch.mm()` or `F.linear()`, PyTorch dispatches to cuBLAS (or\ncuDNN for convolutions), which runs exactly the optimizations described in this\narticle. Understanding tiled matmul gives you practical intuition for:\n\n- **Why batch size affects GPU utilization** : larger batches produce larger\nmatrices, which tile more efficiently and keep more SMs occupied. A batch size\nof 1 may use only 10% of the GPU’s compute capability because the\nmatrices are too narrow to fill all SMs with work.\n- **Why sequence length matters for attention** : longer sequences produce larger\nQ·K⊃T matrices. The tiling efficiency improves with matrix size,\nwhich is why attention gets relatively cheaper (per token) as sequences grow\n— until the O(N²) memory footprint becomes the bottleneck.\n- **Why mixed precision is 2–8× faster** : FP16 and BF16 enable\ntensor cores, which execute matrix multiplies in dedicated hardware at\n15× the throughput of CUDA cores. This is why nearly all modern training\nruns use mixed precision.\n- **Why model parallelism works** : tensor parallelism splits weight matrices\nacross GPUs. Each GPU computes a partial GEMM on a smaller matrix, then the\nresults are combined via all-reduce. The communication cost is proportional to\nthe output size, not the FLOP count.\n\nFor how GPUs communicate during distributed training, see [NCCL Communication](https://www.abhik.ai/concepts/gpu-computing/nccl-communication). For the parallelism strategies that split GEMMs across devices, see [Distributed Parallelism](https://www.abhik.ai/concepts/gpu-computing/distributed-parallelism).\n\n### [Key Takeaways](#key-takeaways)\n\n1. \n**Memory access patterns determine GPU performance** — the naive kernel achieves 1.3% of peak because of non-coalesced global memory access. The algorithm is fine; the data movement is catastrophic.\n2. \n**Tiling is the fundamental optimization** — load a block into fast memory, reuse it many times. On CPU it’s L1 cache. On GPU it’s shared memory. The principle is identical.\n3. \n**Register tiling multiplies the gains** — computing an 8×8 block per thread instead of 1 element reduces memory loads by 32× and reaches 68.7% of cuBLAS.\n4. \n**Vectorization and warp tiling close the gap** — float4 loads and warp-level register locality push to 93.7% of cuBLAS.\n5. \n**Tensor cores change the game entirely** — dedicated matrix hardware delivers 15× the throughput of CUDA cores, making bf16 the default for training.\n\n## [Further Reading](#further-reading)\n\n- [How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance](https://siboehm.com/articles/22/CUDA-MMM) — Simon Boehm’s detailed worklog reaching 93.7% of cuBLAS on A6000\n- [Inside NVIDIA GPUs: Anatomy of High-Performance Matmul Kernels](https://www.aleksagordic.com/blog/matmul) — Aleksa Gordic’s deep dive into Hopper-era SOTA techniques reaching 764 TFLOP/s\n- [CUTLASS: CUDA Templates for Linear Algebra](https://github.com/NVIDIA/cutlass) — NVIDIA’s open-source templated GEMM library\n- [Triton: An Intermediate Language for GPU Programming](https://triton-lang.org/) — OpenAI’s DSL that generates optimized GPU kernels from Python\n- [CUDA C++ Best Practices Guide](https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/) — NVIDIA’s official optimization guide", "url": "https://wpnews.pro/news/cuda-matrix-multiplication-optimization-from-naive-to-near-cublas", "canonical_source": "https://www.abhik.ai/articles/cuda-matrix-multiplication-optimization", "published_at": "2026-10-16 14:03:02+00:00", "updated_at": "2026-10-16 14:15:18.363770+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "artificial-intelligence"], "entities": ["NVIDIA", "CUDA", "cuBLAS", "NVIDIA RTX A6000", "Ampere", "GA102", "PyTorch", "TensorFlow"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/cuda-matrix-multiplication-optimization-from-naive-to-near-cublas", "markdown": "https://wpnews.pro/news/cuda-matrix-multiplication-optimization-from-naive-to-near-cublas.md", "text": "https://wpnews.pro/news/cuda-matrix-multiplication-optimization-from-naive-to-near-cublas.txt", "jsonld": "https://wpnews.pro/news/cuda-matrix-multiplication-optimization-from-naive-to-near-cublas.jsonld"}}