{"slug": "kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead", "title": "Kernel Fusion in NVIDIA CUDA: Optimizing Memory Traffic and Launch Overhead", "summary": "NVIDIA published a technical blog post explaining how kernel fusion in CUDA can optimize GPU code by combining multiple operations into a single kernel, reducing memory traffic and launch overhead. The post demonstrates manual kernel fusion with a sum_abs example and notes that CUDA Graphs provide complementary optimization at a different layer. The techniques aim to improve memory bandwidth utilization and performance for GPU workloads.", "body_md": "There are many ways to optimize code for GPUs. In this post, you’ll learn how kernel fusion can improve memory bandwidth and reduce kernel launch overhead, along with multiple ways to apply it in NVIDIA CUDA code.\n\nA common bottleneck when writing GPU code is that GPU compute is so fast that even high-bandwidth device memory doesn’t use the GPU kernel fully. Kernel fusion addresses this by combining multiple GPU operations into a single device kernel, so intermediate results don’t need to round-trip through global memory or require separate kernel launches.\n\nThis post is about fusing kernel bodies so intermediate results stay in registers, and memory transfers are reduced. CUDA Graphs provide another kind of fusion, but at a different layer. A graph captures a sequence of kernel launches, memory copies, and synchronizations into one reusable object that the host can dispatch with a single call. It doesn’t fuse kernel bodies.\n\n## Kernel fusion in practice\n\nKernels inside a graph run separately and pass intermediate results through global memory. Wrapping a naive baseline in a graph shaves microseconds off the host side, but the one GiB round-trip through the intermediate buffer remains unchanged. The two approaches are complementary and can be used together.\n\nLet’s take a simple example: `sum(abs(x))`\n\n. We read an array, take the absolute value of each element, and return the sum.\n\nA naive implementation uses two kernels: one to compute `abs`\n\ninto a temporary intermediate buffer of the same size as the input, and another to reduce that buffer to a single number. This works, but it’s slow in a way that shows the broader problem.\n\n```\ntemplate <typename Config>\n__global__ void abs_kernel(Config config,\n                           cuda::std::span<const float> in,\n                           cuda::std::span<float> tmp)\n{\n    /* Base SIMT implementation using fsabs */\n}\n\ntemplate <typename Config>\n__global__ void sum_kernel(Config config,\n                           cuda::std::span<const float> tmp,\n                           cuda::std::span<float> out)\n{\n    /* CUB block-reduce + atomic add into out[0] */\n}\n\nint main()\n{\n    /* device, stream, and buffers via cuda::make_buffer\n     * See the CCCL Runtime post for the full setup. */\n\n    auto config = cuda::distribute<BLOCK_THREADS>(in.size());\n\n    cuda::launch(stream, config, abs_kernel<decltype(config)>, in,  tmp);\n    cuda::launch(stream, config, sum_kernel<decltype(config)>, tmp, out);\n}\n```\n\nThe C++ examples featured throughout this post use the latest NVIDIA CCCL runtime interfaces, including `cuda::std::span`\n\n, `cuda::launch`\n\n, and `cuda::make_buffer`\n\n—which debuted in CUDA 13.2. To explore these primitives further, refer to [CCCL Runtime: A Modern C++ Runtime for CUDA](https://developer.nvidia.com/blog/cccl-runtime-a-modern-c-runtime-for-cuda/).\n\nIf we look at an NVIDIA [Nsight Systems ](https://developer.nvidia.com/nsight-systems)profile of the full implementation, it looks like this:\n\n## Manual kernel fusion\n\nThe first thing we could do is rewrite these two kernels as a single `sum_abs_kernel`\n\n. This removes the intermediate buffer, which exists only to pass data between kernels. If both operations live in the same kernel, the buffer disappears, and the performance of the operation improves.\n\n```\ntemplate <typename Config>\n__global__ void sum_abs_kernel(Config config,\n                               cuda::std::span<const float> x,\n                               cuda::std::span<float> out)\n{\n    using BlockReduce = cub::BlockReduce<float, BLOCK_THREADS>;\n    __shared__ typename BlockReduce::TempStorage temp_storage;\n\n    // Grid-stride loop. abs() is computed inline -- no tmp buffer.\n    float thread_sum = 0.0f;\n    const auto tid    = cuda::gpu_thread.rank(cuda::grid, config);\n    const auto stride = cuda::gpu_thread.count(cuda::grid, config);\n    for (size_t i = tid; i < x.size(); i += stride) {\n        thread_sum += fabsf(x[i]);\n    }\n\n    // Block-level reduction in shared memory, via CUB.\n    float block_sum = BlockReduce(temp_storage).Sum(thread_sum);\n\n    // One thread per block atomically accumulates into the global result.\n    if (threadIdx.x == 0) {\n        cuda::atomic_ref<float, cuda::thread_scope_device> r(out[0]);\n        r.fetch_add(block_sum, cuda::memory_order_relaxed);\n    }\n}\n\nint main()\n{\n    /* device, stream, and buffers -- see the CCCL Runtime post. */\n\n    // Fixed grid of 1024 blocks for the grid-stride pattern.\n    auto config = cuda::make_config(cuda::make_hierarchy(\n        cuda::grid_dims(1024),\n        cuda::block_dims<BLOCK_THREADS>()));\n    cuda::launch(stream, config, sum_abs_kernel<decltype(config)>, x, out);\n}\n```\n\n`fabsf(x[i])`\n\nis computed inline, in a register, instead of being read from an intermediate buffer that another kernel wrote.- A grid-stride loop lets each thread accumulate multiple elements into a single register (\n`thread_sum`\n\n), so partial sums never touch global memory. `cub::BlockReduce`\n\nhandles the per-block reduction in shared memory, and`cuda::atomic_ref`\n\nlets one thread per block add its partial sum to the global result.\n\nIf we look at an Nsight Systems profile of the fused implementation, it collapses accordingly:\n\n| Metric | Naive | Manual fusion |\n|---|---|---|\n| Kernels Launched | 2 | 1 |\n| Intermediate buffer | Yes | No |\n| Bytes moved through global memory | 3 GB | 1 GB |\n| Time | 3.51 ms = 2.28ms + 1.23 ms | 1.18 ms |\n| Effective memory bandwidth | 855 GiB/s | 851 GiB/s |\n| Speedup vs. naive | – | 3 times |\n\nThese numbers are for an NVIDIA GeForce RTX 4090. The memory bandwidth sits at ~850 GiB/s—roughly 90% of its theoretical 1,008 GB/s peak.\n\nWe’re bandwidth-bound as the GPU is doing as much memory work as physics enables. The fused version is faster because there’s less memory work to do.\n\nA hand-written CUDA C++ kernel like this is sometimes the right solution, because you have full control and get as close as possible to peak performance. The drawback is the effort required to write the kernel, along with the high maintenance cost and limited portability across GPU architectures.\n\nDevice-side libraries such as CUB, `libcu++`\n\n, `nvmath.device`\n\n, and NVSHMEM can simplify manual fusion by providing reusable primitives for FFTs, matrix multiplications, reductions, scans, sorts, communication, and more.\n\n## Implicit kernel fusion\n\nWriting your own kernel works. In this case, the operation is essentially “applying `abs`\n\nwhile reducing.” Instead of expressing that manually in CUDA C++, we can describe the computation in plain Python and let the compiler generate the kernel.\n\nThat is what `torch.compile`\n\ndoes. You give it a function, it traces what the tensors are doing, and Torch Inductor, PyTorch’s compiler, generates a kernel that runs on the GPU.\n\n``` python\nimport torch\n\nx = torch.randn(N, device=\"cuda\")\n\ndef sum_abs(t):\n    return t.abs().sum()\n\ncompiled = torch.compile(sum_abs)\nresult = compiled(x)\n```\n\nBy wrapping the `sum_abs`\n\nfunction with `torch.compile`\n\nwe get a fused kernel. In the eager version (without `torch.compile`\n\n) we get what we expect: two kernels, one element-wise (`abs`\n\n) and one reduce (`sum`\n\n).\n\nIn the compiled version, we also get two kernels: `triton_red_fused_abs_sum_0`\n\nand `triton_red_fused_abs_sum_1`\n\n.\n\nWhy? Because that is what the Inductor compiler generates.\n\nTo understand what Inductor did, take a look at the generated Triton kernels:\n\n```\nTORCH_LOGS=\"output_code\" python implicit_fusion.py\n```\n\nThe interesting part from the first kernel:\n\n```\ntmp0 = tl.load(in_ptr0 + ...)   # load a chunk of input\ntmp1 = tl_math.abs(tmp0)        # apply abs in registers, inline\ntmp2 = _tmp3 + tmp1             # accumulate into the running sum\n```\n\nThe `abs`\n\nhappens in a register, right after the load, inside the reduction kernel. There’s no separate `abs`\n\nkernel and no per-element intermediate buffer. A second kernel, running in one block, reads the 2,048 partial results and reduces them to one final scalar. You can learn more about this in this [PyTorch blog post.](https://pytorch.org/blog/why-is-pytorch-compile-so-fast-kernel-fusion/)\n\nIn practice, `torch.compile`\n\nfused the operation but split the reduction into two stages: one kernel reduces the input to a small set of per-block partial sums, and a second kernel adds those partial sums into a single number. In the manual version, we made a different choice: using atomics to accumulate into one global float. Both versions move one GiB through global memory instead of 3 GiB. The compiler picked a reasonable strategy but not the same strategy we would choose.\n\nThis is both good and bad. The compiler gives us a fused kernel for free, but we trade away some predictability. A different dtype, a different shape, or a small operation added in the middle can produce a completely different set of kernels. Sometimes you get one kernel. Sometimes you get four. Sometimes a fusion you were counting on quietly stops happening, without warning.\n\n## Explicit kernel fusion: Callbacks, iterators, and epilogs\n\nWhat if we want the productivity of Python with the predictability of a hand-written kernel? That is what `cuda.compute`\n\nis for.\n\n`cuda.compute`\n\nexposes device-wide algorithms such as `reduce`\n\n, `scan`\n\n, `sort`\n\n, and `transform`\n\n, along with a set of iterators that you can compose to control how those algorithms read their input. You do not write a kernel directly. Instead, you compose a computation and pass it to an algorithm, and the fusion happens because you explicitly asked for it.\n\nThe same `sum(abs(x))`\n\noperation in `cuda.compute`\n\nlooks like this:\n\n``` python\nimport torch\nimport numpy as np\nfrom cuda.compute import (\n    Determinism, OpKind, TransformIterator, reduce_into,\n)\n\nx = torch.randn(N, device=\"cuda\")\nout = torch.empty(1, dtype=torch.float32, device=\"cuda\")\nh_init = np.array([0.0], dtype=np.float32)\n\nabs_it = TransformIterator(x, lambda a: abs(a))\nreduce_into(\n    d_in=abs_it, d_out=out, num_items=N,\n    op=OpKind.PLUS, h_init=h_init,\n    determinism=Determinism.NOT_GUARANTEED,\n)\n```\n\n`TransformIterator`\n\ndoesn’t allocate anything or launch a kernel. It’s a lazy view that says, “when someone asks me for element `i`\n\n, read `x[i]`\n\nand apply `abs`\n\nto it.” `reduce_into`\n\nthen performs a single-device-wide reduction, pulling elements through the iterator.\n\nThe `abs`\n\nruns inline, in registers, as each element is loaded from global memory. There’s no per-element intermediate buffer at any point. Also note that `cuda.compute`\n\nworks with PyTorch Tensors. It supports inputs that implement DLPack or the CUDA Array API, such as CuPy arrays.\n\nOnce you separate “what the data is” from “how to walk it,” producing element `i`\n\nbecomes an arbitrary inline computation: a load, a transform, a gather, or a struct unpack. The algorithm threads these operations into a single pass with no intermediate storage. This kind of solution is deterministic, portable, and composable, making it ideal for library authors and performance applications.\n\nIf we profile this with Nsight Systems, we get something similar to a single fused kernel produced by a few lines of Python:\n\nThe [CUB library](https://nvidia.github.io/cccl/unstable/cub/index.html), on which `cuda.compute`\n\nis built, produces a single fused kernel optimized for the operation we described. It provides battle-tested implementations for many algorithms, along with the guarantees CCCL provides, such as optimizations for new hardware.\n\nUnlike a compiler approach, it won’t silently change the fusion strategy because of a small code change, a different compiler version, or a compiler decision, because you explicitly decide what should be fused. This gives you more deterministic results while still enabling control over the final behavior, including details like [floating-point determinism](https://developer.nvidia.com/blog/controlling-floating-point-determinism-in-nvidia-cccl/).\n\n| Metric | Naive | Manual Fusion | `torch.compile` | `cuda.compute` |\n|---|---|---|---|---|\n| Kernels Launched | 2 | 1 | 2 | 1 |\n| Per-element intermediate buffer | yes | no | no | no |\n| Speedup vs. naive | – | 3x | 3x | 3x |\n| Who decides what gets fused? | – | You | Compiler | You |\n| Determinism | – | Full | Varies | Predictable |\n\nAll three fused methods end up in the same place with 1 GiB of memory traffic and about a 3x speedup over the naive version. With manual fusion, you write the kernel, so the fusion is explicit and under your control. With compiler fusion, the compiler chooses for you.\n\nWith `cuda.compute`\n\n, you write the composition using Python ergonomics. Like manual fusion, this provides an explicit guarantee that the transform is fused into the reduction. The added benefit is that this composition is backed by CUB, so you get a well-tested, highly optimized implementation under the hood instead of a custom kernel you have to maintain and tune yourself.\n\n`cuda.compute`\n\ngives Python developers access to the same iterator-based, deterministic fusion that Thrust and CUB have long provided in C++. It enables higher-level APIs and frameworks to perform as if they had custom kernels, without requiring developers to write those kernels themselves.\n\nYou can learn more about [cuda.compute](https://nvidia.github.io/cccl/unstable/python/compute/index.html) in docs and [cuda-samples](https://github.com/NVIDIA/cuda-samples/tree/master/python/2_CoreConcepts) for Python.", "url": "https://wpnews.pro/news/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead", "canonical_source": "https://developer.nvidia.com/blog/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead/", "published_at": "2026-07-10 16:41:03+00:00", "updated_at": "2026-07-10 17:09:00.429610+00:00", "lang": "en", "topics": ["developer-tools", "ai-infrastructure", "machine-learning"], "entities": ["NVIDIA", "CUDA", "CCCL Runtime", "Nsight Systems"], "alternates": {"html": "https://wpnews.pro/news/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead", "markdown": "https://wpnews.pro/news/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead.md", "text": "https://wpnews.pro/news/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead.txt", "jsonld": "https://wpnews.pro/news/kernel-fusion-in-nvidia-cuda-optimizing-memory-traffic-and-launch-overhead.jsonld"}}