{"slug": "the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough", "title": "The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough", "summary": "NVIDIA published a step-by-step tutorial demonstrating how to debug, benchmark, and optimize CUDA code using modern tools such as Compute Sanitizer, Nsight Systems, NVTX, CUB, and CCCL, applied to an RGB-to-grayscale image processing pipeline with median filtering. The tutorial covers six incremental improvements, including adopting the CCCL API, using CUB's optimized algorithms, managing GPU memory with pooled containers, and parallelizing work with per-thread streams, with code available on GitHub and Google Colab.", "body_md": "NVIDIA CUDA remains the foundation of GPU-accelerated computing, powering everything from scientific simulations to large-scale AI training.\n\nBut writing correct, maintainable, and performant CUDA code can be challenging: memory bugs hide in plain sight, performance bottlenecks can be invisible without the right instrumentation, and hand-rolled GPU algorithms rarely match the efficiency of optimized libraries. Fortunately, the modern CUDA toolchain has matured significantly, and many of these challenges now have straightforward solutions.\n\nIn this blog post, we will walk through the tools NVIDIA offers to debug, benchmark, and improve your code. With only small line changes each time, we are going to make the example code safer, easier to maintain, and faster.\n\nAcross six incremental steps, this post will cover:\n\n- How to easily find indexing bugs by adopting the modern CCCL API and Compute Sanitizer\n- How to improve Nsight Systems benchmarks with NVTX\n- How to use CUB’s optimized algorithms at the block and device level\n- How to manage GPU memory through pooled containers\n- How to speed up host-to-device transfers with pinned containers\n- How to parallelize GPU work by giving each thread its own stream and asynchronous transfers\n\nAs a companion to this blog post, we provide the [code](https://github.com/NVIDIA/accelerated-computing-hub/tree/main/resources/blogs/modern_cuda_cpp_blogpost/code_steps) and the option to run on [Google Colab](https://colab.research.google.com/github/NVIDIA/accelerated-computing-hub/blob/main/resources/blogs/modern_cuda_cpp_blogpost/code_steps/modern_cuda_toolbox_tutorial.ipynb).\n\n## Starting point: An image processing pipeline example\n\nFrom an input stream of red, green, and blue images, start by transferring the data from the CPU to the GPU. Then convert those from RGB to grayscale.\n\nThen, for each 32 by 32 pixel tile in the image, compute the median by sorting the pixels and selecting the middle value. Finally, copy the median of each tile back to the CPU.\n\n### The base code example\n\nBelow is the full starting code. Each step in this post improves on it.\n\n```\n#define CUDA_CHECK_ERROR(call) do { \\\n    cudaError_t err = call; \\\n    if (err != cudaSuccess) { \\\n        std::cerr << \"CUDA error in \" << __FILE__ << \" at line \" << __LINE__ << \": \" \\\n                  << cudaGetErrorString(err) << std::endl; \\\n        std::exit(EXIT_FAILURE); \\\n    } \\\n} while (0)\n// Alias for an image pixel\nusing pixel_t = uint8_t;\n// Kernel converting the red, green and blue images into a single gray image\n__global__ void computeRGBToGray(const pixel_t* d_image_r, const pixel_t* d_image_g, const pixel_t* d_image_b, pixel_t* d_image_gray, int width, int height) {\n    // Compute the thread global index in the grid\n    const int x = threadIdx.x + blockIdx.x * blockDim.x;\n    const int y = threadIdx.y + blockIdx.y * blockDim.y;\n    // Boundary check selecting only threads within the image boundary\n    if (x < width && y < height) {\n        // Compute the thread index in the image\n        const int i = x + y * width;\n        // Convert from rgb to grayscale and store the result in global memory\n        d_image_gray[i] = static_cast<pixel_t>(0.299f * d_image_r[i] + 0.587f * d_image_g[i] + 0.114f * d_image_b[i]);\n    }\n}\n// Kernel computing the median of each tile in the grayscale image\ntemplate <int TILE_WIDTH, int HISTO_SIZE>\n__global__ void computeMedian(pixel_t *d_image_gray, pixel_t *d_median, int width, int height) {\n    // Compute the thread global index in the grid\n    const int x = threadIdx.x + blockIdx.x * blockDim.x;\n    const int y = threadIdx.y + blockIdx.y * blockDim.y;\n    // Boundary check selecting only threads within the image boundary\n    if (!(x < width && y < height))\n        return;\n    // Allocate the shared memory in which we will store the tile\n    __shared__ pixel_t tile[TILE_WIDTH * TILE_WIDTH];\n    // Compute the thread index in the image\n    const int index = x + y * width;\n    // Load the tile's grayscale value from global memory into shared memory\n    tile[index] = d_image_gray[index];\n    // Synchronize to make sure all threads have loaded their data\n    __syncthreads();\n    // Sort the tile array using a single threaded bubble sort\n    if (threadIdx.x == 0 && threadIdx.y == 0) {\n        for (int i = 0; i < TILE_WIDTH * TILE_WIDTH; ++i)\n            for (int j = i + 1; j < TILE_WIDTH * TILE_WIDTH; ++j)\n                if (tile[i] > tile[j])\n                    cuda::std::swap(tile[i], tile[j]);\n        // Each thread block stores the median, found in the middle index after sorting, in the global median array\n        const int medianIndex = (TILE_WIDTH * TILE_WIDTH) / 2;\n        d_median[blockIdx.x + blockIdx.y * gridDim.x] = tile[medianIndex];\n    }\n}\nint main() {\n    // Define all the example constants\n    constexpr auto TILE_WIDTH = 32;\n    constexpr auto HISTO_SIZE = 256;\n    constexpr auto NB_TILE_X = 250;\n    constexpr auto NB_TILE_Y = NB_TILE_X;\n    constexpr auto IMAGE_LENGTH = TILE_WIDTH * NB_TILE_X;\n    constexpr auto IMAGE_SIZE = IMAGE_LENGTH * IMAGE_LENGTH;\n    constexpr auto NB_IMAGES = 3;\n    constexpr auto INIT_VALUE = 4;\n    // Allocate the CPU memory to store the images tiles medians and for the red, green, blue and grayscale images\n    std::vector<std::vector<pixel_t>> h_images_r(NB_IMAGES, std::vector<pixel_t>(IMAGE_SIZE, 4));\n    std::vector<std::vector<pixel_t>> h_images_g(NB_IMAGES, std::vector<pixel_t>(IMAGE_SIZE, 4));\n    std::vector<std::vector<pixel_t>> h_images_b(NB_IMAGES, std::vector<pixel_t>(IMAGE_SIZE, 4));\n    std::vector<std::vector<pixel_t>> h_images_gray(NB_IMAGES, std::vector<pixel_t>(IMAGE_SIZE, 0));\n    std::vector<std::vector<pixel_t>> h_medians(NB_IMAGES, std::vector<pixel_t>(NB_TILE_X * NB_TILE_Y));\n    // Run the image processing pipeline for each image, in parallel\n    #pragma omp parallel for\n    for (int i = 0; i < NB_IMAGES; ++i)\n    {\n        pixel_t *d_image_r, *d_image_g, *d_image_b, *d_image_gray, *d_median;\n        // Allocate the GPU memory for each container\n        CUDA_CHECK_ERROR(cudaMalloc(&d_image_r, IMAGE_SIZE * sizeof(pixel_t)));\n        CUDA_CHECK_ERROR(cudaMalloc(&d_image_g, IMAGE_SIZE * sizeof(pixel_t)));\n        CUDA_CHECK_ERROR(cudaMalloc(&d_image_b, IMAGE_SIZE * sizeof(pixel_t)));\n        CUDA_CHECK_ERROR(cudaMalloc(&d_image_gray, IMAGE_SIZE * sizeof(pixel_t)));\n        CUDA_CHECK_ERROR(cudaMalloc(&d_median, (NB_TILE_X * NB_TILE_Y) * sizeof(pixel_t)));\n        // Copy the memory of each container from CPU to GPU\n        CUDA_CHECK_ERROR(cudaMemcpy(d_image_r, h_images_r[i].data(), IMAGE_SIZE * sizeof(pixel_t), cudaMemcpyHostToDevice));\n        CUDA_CHECK_ERROR(cudaMemcpy(d_image_g, h_images_g[i].data(), IMAGE_SIZE * sizeof(pixel_t), cudaMemcpyHostToDevice));\n        CUDA_CHECK_ERROR(cudaMemcpy(d_image_b, h_images_b[i].data(), IMAGE_SIZE * sizeof(pixel_t), cudaMemcpyHostToDevice));\n        // Launch a GPU kernel to convert the RGB images to grayscale\n        dim3 blockSize(TILE_WIDTH, TILE_WIDTH);\n        dim3 gridSize(cuda::ceil_div(IMAGE_LENGTH, blockSize.x), cuda::ceil_div(IMAGE_LENGTH, blockSize.y));\n        computeRGBToGray<<<gridSize, blockSize>>>(d_image_r, d_image_g, d_image_b, d_image_gray, IMAGE_LENGTH, IMAGE_LENGTH);\n        CUDA_CHECK_ERROR(cudaGetLastError());\n        // Launch the GPU kernel to compute the median of every tile in the image\n        computeMedian<TILE_WIDTH, HISTO_SIZE><<<gridSize, blockSize>>>(d_image_gray, d_median, IMAGE_LENGTH, IMAGE_LENGTH);\n        CUDA_CHECK_ERROR(cudaGetLastError());\n        // Copy the GPU median memory back to the CPU\n        CUDA_CHECK_ERROR(cudaMemcpy(h_medians[i].data(), d_median, (NB_TILE_X * NB_TILE_Y) * sizeof(pixel_t), cudaMemcpyDeviceToHost));\n        // Free the GPU memory\n        CUDA_CHECK_ERROR(cudaFree(d_image_r));\n        CUDA_CHECK_ERROR(cudaFree(d_image_g));\n        CUDA_CHECK_ERROR(cudaFree(d_image_b));\n        CUDA_CHECK_ERROR(cudaFree(d_image_gray));\n        CUDA_CHECK_ERROR(cudaFree(d_median));\n    }\n    return 0;\n}\n```\n\nThis code starts by defining two kernels:\n\n`computeRGBToGray`\n\nloads the values of the red, green, and blue input images to convert and write them to the grayscale output image.`computeMedian`\n\ncomputes the median of each tile of an input grayscale image. Each thread block loads the tile from global memory to shared memory. Then, a single thread is used to sort the array and write the value found at the middle index, corresponding to the median, in the global output median array.\n\nIn the main, after defining the constants used for the example, the CPU memory is allocated for each image and for the medians.\n\nThe image processing pipeline is then run for each of the three images, in parallel, using OpenMP. The pipeline starts by allocating the required memory on the GPU before transferring the data from the CPU to the GPU. The two kernels to convert from RGB to grayscale and to compute the median are launched afterward. Finally, we copy back the median results to the CPU before freeing the memory.\n\nThis code has several flaws that will be addressed, step by step.\n\n## 1. Compute Sanitizer and the CCCL API: Find bugs easily and write safer code\n\nLet’s start by running the code.\n\n```\ncode_steps$ ./build/0_base_error_example\nCUDA error in 0_base_error_example.cu at line 105: an illegal memory access was encountered\n```\n\nWhile the code has some error checking in it, when you get an error message like this “illegal memory access,” you should start by using compute-sanitizer to investigate further.\n\nUsing [Compute Sanitizer](https://developer.nvidia.com/compute-sanitizer), the NVIDIA functional correctness checking suite, we can directly identify a bug that can be hard to spot:\n\n``` bash\n$ compute-sanitizer ./build/0_base_error_example \n========= COMPUTE-SANITIZER\n========= Invalid __shared__ write of size 1 bytes\n=========     at void computeMedian<(int)32, (int)256>(unsigned char *, unsigned char *, int, int)+0x170 in 0_base_error_example.cu:55\n=========     by thread (0,3,0) in block (20,0,0)\n=========     Access at 0x6440 is out of bounds\n```\n\nRunning the above directly shows that the code suffers from an out-of-bound shared write at line 55 of `0_base_error_example.cu`\n\n.\n\n```\ntile[index] = d_image_gray[index];\n```\n\nThe line incorrectly loads data in shared memory, using a global index. Since shared memory is defined at the thread block level, we need to change the indexing. To avoid indexing mistakes, a new API was introduced in CCCL to distinguish global and block-level indexing. To use it, you first need to launch your kernel using the new `cuda::launch`\n\nAPI:\n\n```\nauto config = cuda::make_config(cuda::block_dims(...), cuda::grid_dims(...));\ncuda::launch(stream, config, kernel_name<decltype(config)>, input)\n```\n\nThen, using the new indexing API inside the kernel:\n\n```\ntemplate <typename Configuration>\n__global__ void kernel_name(Configuration config, ...) {\n    // Retrieve and expand each global index\n    const auto [x, y, z] = cuda::gpu_thread.index(cuda::grid, config);\n\n    // Retrieve the block index structure (containing block_idx.x, .y, .z)\n    const auto block_idx = cuda::gpu_thread.index(cuda::block, config);\n}\n```\n\nWithout using compute-sanitizer or the new API, this mistake could also have been directly spotted by using `cuda::std::span`\n\nor its n-dimensional variant `cuda::std::mdspan`\n\ninstead of raw pointers. `cuda::std::span`\n\nand `cuda::std::mdspan`\n\nare non-owning views over contiguous memory and are useful to abstract the exact container away. Accessing data through spans is safer than through raw pointers in part because in debug mode, out-of-bounds accesses will trigger an assertion.\n\nThe kernel should be updated as:\n\n```\n// Alias for a 2-dimensional mdspan\ntemplate <typename T>\nusing span_2d = cuda::std::mdspan<T, cuda::std::dims<2>>;\n\ntemplate <typename Configuration>\n__global__ void computeMedian(..., span_2d<const pixel_t> d_image_gray, ...)\n```\n\nIf you run the code with these changes you’ll get something like the following:\n\n```\n$ ./build/1_span \nlibcudacxx/include/cuda/std/__mdspan/mdspan.h:436: operator(): block: [16,0,0], thread: [0,30,0] Assertion `mdspan: operator() out of bounds access` failed.\n```\n\nMemory accesses in shared memory should also be protected by using a `cuda::shared_memory_mdspan`\n\nas in the following snippet:\n\n```\n__shared__ pixel_t shared[TILE_WIDTH * TILE_WIDTH];\ncuda::shared_memory_mdspan tile_2d(shared, TILE_WIDTH, TILE_WIDTH);\n```\n\nNow you can run and everything should execute properly without errors.\n\nUsing the new launch API and its indexing mechanism, spans over raw pointers, and compute-sanitizer, out-of-bounds accesses either do not happen or are caught right away. For more information on compute-sanitizer, see [Efficient CUDA Debugging: How to Hunt Bugs with NVIDIA Compute Sanitizer](https://developer.nvidia.com/blog/debugging-cuda-more-efficiently-with-nvidia-compute-sanitizer/).\n\n## 2. Nsight Systems and NVTX: Benchmark your code properly\n\nNow the code is bug-free, it is ready to be benchmarked using NVIDIA Nsight Systems. It allows you to visualize the program timeline: know when each function is called and for how long.\n\nTo make the timeline visualization easier, we wrap every interesting code section using NVTX:\n\n```\nvoid image_compute(...)\n{\n  // NVTX range for the scope of the whole function\n  nvtx3::scoped_range fun_scope(\"Image compute\");\n\n // NVTX range that is pushed and then popped for a specific code section\n  nvtxRangePushA(\"Kernel median\");\n\n  // Launch the GPU kernel to compute the median of every tile in the image\n  ...\n\n  // Pop the range at the end of the specific code section\n  nvtxRangePop();\n}\n```\n\nWhich yields the following result:\n\nIn the GPU hardware (CUDA HW) section of the profiler output in Figure 3, above, it is reported that the GPU is mainly busy with kernels (98.5% of the GPU time) while the memory operations only take 1.5% of the GPU time.\n\nOf the two kernels, the one computing the medians is taking the majority of the runtime with 2.1 seconds for each image (see the yellow box on the right, where the stats for `computeMedian`\n\nare shown, and the elapsed time is 2.142s).\n\nFrom the CPU (thread) section, we see the image computation takes 6.8 seconds in total, with most of the time being spent on computing the medians for the three grayscale images.\n\nWe now know the first operation to optimize in order to have the greatest impact. For more information on Nsight Systems, see [Optimizing CUDA Memory Transfers with NVIDIA Nsight Systems](https://developer.nvidia.com/blog/optimizing-cuda-memory-transfers-with-nsight-systems/). For more information on NVTX, see [CUDA Pro Tip: Generate Custom Application Profile Timelines with NVTX](https://developer.nvidia.com/blog/cuda-pro-tip-generate-custom-application-profile-timelines-nvtx/).\n\n## 3. CUB: Express algorithms directly on the GPU\n\nWhen dealing with common algorithms, writing custom kernels is error-prone and will most likely result in an inefficient implementation. Whenever possible, both for device-side patterns and for in-kernel primitives, it is recommended to use CUB.\n\nCUB is the NVIDIA parallel algorithm library shipped through CCCL. It exposes highly optimized routines at several granularities: device-wide (`cub::Device*`\n\n), block-level (`cub::Block*`\n\n), and warp-level (`cub::Warp*`\n\n).\n\nFor the RGB to grayscale step, we can replace the custom kernel with `cub::DeviceTransform::Transform`\n\n. It applies a user-provided function to a tuple of input iterators and writes the result to an output iterator, executing on the GPU:\n\n```\n// Use CUB to convert the RGB images to grayscale\ncub::DeviceTransform::Transform(\n    cuda::std::make_tuple(d_image_r, d_image_g, d_image_b),  // inputs\n    d_image_gray,                                            // output\n    IMAGE_SIZE,                                              // size\n    [] __host__ __device__ (pixel_t r, pixel_t g, pixel_t b) // functor\n    {\n        return static_cast<pixel_t>(0.299f * r + 0.587f * g + 0.114f * b);\n    },\n    stream);\n```\n\nFor the median, programming a parallel block-level sort by hand is complex and slow. Instead, we directly leverage CUB’s block-level radix sort inside the kernel:\n\n```\n// Declare and allocate the storage for CUB BlockRadixSort\nusing BlockRadixSort = cub::BlockRadixSort<...>;\n__shared__ typename BlockRadixSort::TempStorage temp_storage;\n\n// Load the tile's grayscale value from global memory\npixel_t thread_keys[1];\nthread_keys[0] = d_image_gray(y, x);\n\n// Perform the thread-block-level radix sort\nBlockRadixSort(temp_storage).Sort(thread_keys);\n\n// Select the thread found at the middle index\n// Write its value which is, after sorting, the median, in the global median array\nif (block_idx.x == TILE_WIDTH / 2 && block_idx.y == TILE_WIDTH / 2)\n    d_median(grid_block_idx.y, grid_block_idx.x) = thread_keys[0];\n```\n\nFollowing this change, we benchmark again using Nsight Systems:\n\nThe time required to compute the median is now only 773 microseconds (again, look at the elapsed time in the `computeMedian `\n\nyellow pop-out image), 2717x faster. The overall time to compute all three images is now 635 milliseconds, 10x faster.\n\nIf we reassess the current bottleneck: the time spent on memory allocations represents around 83% of the total image compute runtime.\n\nThis can be greatly improved.\n\n## 4. Pooled Memory Containers: Convenient and faster memory management\n\nAllocating GPU memory using `cudaMalloc`\n\ncan have unexpected negative effects: leaks by forgetting to call `cudaFree`\n\nand costly memory operations in critical parts of your code.\n\nInstead, we recommend using CCCL’s asynchronous memory containers, `cuda::device_buffer`\n\n. Like C++ `std::vector`\n\n, the memory is automatically deallocated once the container goes out of scope.\n\nAdditionally, a memory pool backs the buffer, so repeated allocations and deallocations do not pay the full cost of `cudaMalloc`\n\n/ `cudaFree`\n\nevery time.\n\nTo use the GPU memory containers, we update the code accordingly:\n\n```\n// Resource to handle the GPU memory allocations\ncuda::device_memory_pool_ref device_resource = cuda::device_default_memory_pool(cuda::device_ref{0});\n\n// Explained at a later stage, unimportant for now\ncuda::stream stream{cuda::device_ref{0}};\n\n// Allocate the GPU memory using uninitialized containers\ncuda::device_buffer<pixel_t> d_image_r = cuda::make_buffer<pixel_t>(stream, device_resource, IMAGE_SIZE, cuda::no_init);\n...\n```\n\nAfter this change, we analyze the timeline again:\n\nThe time spent on memory allocation is now almost nonexistent; the time it takes to compute an image has improved by 2.6x.\n\nThe GPU time is now memory-dominated. Almost all the time to compute all images is spent on first copying the red, green and blue, for the three images, from CPU to GPU.\n\nIt is possible to greatly speed up the host-to-device memory transfers.\n\n## 5. Pinned memory: Faster host-to-device memory transfers\n\nCPU data allocations are pageable by default, which the GPU cannot access directly. The CUDA driver must first allocate a temporary page-locked, or pinned, host array, copy the host data to the pinned array, and then transfer the data from the pinned array to device memory.\n\nWhen it is known in advance that CPU memory will be copied to the GPU, it is advised to directly allocate using pinned memory.\n\nCCCL exposes a pinned-memory host container, `cuda::host_buffer`\n\n, that can be built through the `cuda::make_pinned_buffer`\n\nfactory:\n\n```\n// Allocate the CPU memory to store the image tiles, medians, and for the red, green, blue, and grayscale images\n// Those CPU containers, contrary to std::vector, are allocated using pinned memory\nstd::vector<cuda::host_buffer<pixel_t>> h_images_r(NB_IMAGES, cuda::make_pinned_buffer<pixel_t>(stream, IMAGE_SIZE, ...));\n...\n```\n\nFollowing those changes, we can benchmark again:\n\nThe time it takes to do the host-to-device memory transfers has been significantly reduced; it now only takes 25 ms to compute all images, 10x faster.\n\nFor more information on pinned memory, see [How to Optimize Data Transfers in CUDA C/C++](https://developer.nvidia.com/blog/how-optimize-data-transfers-cuda-cc/).\n\nOne surprising behavior might have caught the eye of some readers since the beginning:\n\nThough we are using different CPU threads, all operations (memory and kernels) are being executed sequentially on the GPU.\n\nLet’s fix it.\n\n## 6. Streams: Parallelize operations on the GPU\n\nBy default, all operations (kernels, memory allocations, or transfers) are launched on what we call the default stream: it can be viewed as a queue of tasks the GPU needs to execute in order.\n\nIn this example, we need a stream for each image/thread. CCCL provides `cuda::stream`\n\n, an owning self-managed version of CUDA streams. It can simply be constructed inside the parallel for loop, so each OpenMP thread gets its own queue of GPU work.\n\nTo efficiently leverage streams, we also need to use the asynchronous API: each GPU operation launched by the CPU should not be waited upon until completion. To saturate the GPU, each CPU thread should launch as many operations as possible, as fast as possible, without waiting for them to first complete. Kernels and CUB device calls are already asynchronous by default and are launched on the passed stream. To launch asynchronous copies between the host and the device, we use the new CCCL `cuda::copy_bytes`\n\nAPI.\n\nIt is advised in any modern CUDA code never to rely on the default stream and to always rely on streams.\n\nWe update the code accordingly:\n\nA dedicated `init_stream`\n\nis used for the initial allocations of the pinned host buffers. Each iteration of the parallel for loop now owns its own `cuda::stream`\n\nfor the computation pipeline:\n\n```\n// Stream used for initial host buffer allocations\ncuda::stream init_stream{cuda::device_ref{0}};\n\n...\n\n// Allocate the host pinned buffers on init_stream:\nstd::vector<cuda::host_buffer<pixel_t>> h_images_r(NB_IMAGES, cuda::make_pinned_buffer<pixel_t>(init_stream, IMAGE_SIZE, ...));\n\n...\n\n// Sync before launching operations on another stream:\ninit_stream.sync();\n\n#pragma omp parallel for\nfor (int i = 0; i < NB_IMAGES; ++i)\n{\n    // One different stream per thread\n    cuda::stream stream{cuda::device_ref{0}};\n\n    ...\n\n    // GPU buffer allocations using the stream owned by each thread\n    cuda::device_buffer<pixel_t> d_image_r = cuda::make_buffer<pixel_t>(stream, device_resource, IMAGE_SIZE, cuda::no_init);\n\n    ...\n\n    // Copy the memory of each container from CPU to GPU asynchronously using the stream owned by each thread\n    cuda::copy_bytes(stream, h_images_r[i], d_image_r);\n    \n    ...\n\n    // Use CUB to convert the RGB images to grayscale asynchronously using the per thread stream\n    cub::DeviceTransform::Transform(..., stream.get());\n\n    // Launch the GPU kernel to compute the median of every tile in the image using the per thread stream\n    cuda::launch(stream, ...);\n\n    // Copy the GPU median memory back to the CPU\n    cuda::copy_bytes(stream, d_median, h_medians[i]);\n    \n    ...\n\n    // To make sure the copy bytes is finished before accessing results on the host\n    stream.sync();\n}\n```\n\nFollowing those changes, we can take a final look at the timeline:\n\nWe now have a complete overlap between our kernels and memory copies.\n\nThe final duration to compute all three images following all our improvements is 23 milliseconds, starting from 6.8 seconds.\n\n## Your turn\n\nUsing the CUDA Developer’s Toolbox, we made the code safer, easier to maintain and faster. No low-level optimizations were used, yet the code is 300x faster.\n\nTry out this [code](https://github.com/NVIDIA/accelerated-computing-hub/tree/main/resources/blogs/modern_cuda_cpp_blogpost/code_steps) yourself, and run on [Google Colab](https://colab.research.google.com/github/NVIDIA/accelerated-computing-hub/blob/main/resources/blogs/modern_cuda_cpp_blogpost/code_steps/modern_cuda_toolbox_tutorial.ipynb) if you like.\n\nWe have also built a full class to learn how to use those tools in detail. It is freely available on [YouTube](https://www.youtube.com/playlist?list=PL5B692fm6--vWLhYPqLcEu6RF3hXjEyJr) alongside links to practice on [Google Colab](https://github.com/NVIDIA/accelerated-computing-hub/tree/main/tutorials/cuda-cpp).", "url": "https://wpnews.pro/news/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough", "canonical_source": "https://developer.nvidia.com/blog/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough/", "published_at": "2026-09-02 17:15:57+00:00", "updated_at": "2026-09-02 17:25:36.718432+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["NVIDIA", "CUDA", "Compute Sanitizer", "Nsight Systems", "NVTX", "CUB", "CCCL", "Google Colab"], "alternates": {"html": "https://wpnews.pro/news/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough", "markdown": "https://wpnews.pro/news/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough.md", "text": "https://wpnews.pro/news/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough.txt", "jsonld": "https://wpnews.pro/news/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkthrough.jsonld"}}