{"slug": "a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda", "title": "A Pedagogical Introduction to Porting a Conjugate Gradient Solver to CUDA", "summary": "A developer has published a pedagogical walkthrough of incrementally porting HPCCG, a Conjugate Gradient mini-application from the Mantevo project, from CPU to CUDA. Profiling a 100^3 problem showed the sparse matrix-vector multiply kernel consumed 83.5% of the 3.378-second runtime, motivating a staged migration of SpMV, WAXPBY, and DDOT kernels, with a parallel Kokkos port run on both CPU and CUDA backends for comparison. The accompanying code and intermediate commits are available in the author's GitHub repository.", "body_md": "GPU programming tutorials often begin with isolated examples: vector addition, reductions, matrix multiplication, and memory coalescing. These are useful for learning individual CUDA concepts, but there is another question that quickly arises when working with real scientific software:\n\n**How do we actually port an existing CPU application to a GPU?**\n\nIn this article, I will walk through a small but realistic example using HPCCG, a Conjugate Gradient mini-application from the Mantevo project. Rather than rewriting the entire solver for CUDA at once, we will migrate it incrementally:\n\nAs a final experiment, we will return to the original CPU implementation and port it independently to Kokkos, then run essentially the same Kokkos source using both CPU and CUDA backends. This gives us a simple way to compare an explicitly written CUDA implementation with a performance-portability approach.\n\nThe purpose is not to produce the fastest possible Conjugate Gradient implementation. Instead, the goal is to make the **porting and performance-engineering process itself visible**: profile, form a hypothesis, change one part of the application, measure again, and follow the bottleneck.\n\nAll of the code and intermediate CUDA-porting commits used in this article are available in my [HPCCG GitHub repository](https://github.com/junsik45/HPCCG), so the individual stages can be checked out and reproduced while following along.\n\nHPCCG solves a linear system arising from a 27-point stencil on a structured 3D grid, using an unpreconditioned Conjugate Gradient method. The solver is deliberately simple, which makes it a good porting target: almost all of the work is done by three kernels.\n\n`HPC_sparsemv`): computes `Ap = A * p`\n`waxpby`): computes `w = alpha * x + beta * y`\n`ddot`): computes the dot product `x · y`\nOne CG iteration strings these together roughly as follows:\n\n```\nfor (int k = 1; k < max_iter && normr > tolerance; k++) {\n  if (k == 1) {\n    waxpby(nrow, 1.0, r, 0.0, r, p);          // p = r\n  } else {\n    oldrtrans = rtrans;\n    rtrans = ddot(nrow, r, r);                // global reduction\n    double beta = rtrans / oldrtrans;\n    waxpby(nrow, 1.0, r, beta, p, p);         // p = r + beta * p\n  }\n\n  normr = sqrt(rtrans);                       // convergence check\n\n  HPC_sparsemv(A, p, Ap);                     // Ap = A * p\n\n  double alpha = ddot(nrow, p, Ap);           // global reduction\n  alpha = rtrans / alpha;\n\n  waxpby(nrow, 1.0, x,  alpha, p,  x);        // x = x + alpha * p\n  waxpby(nrow, 1.0, r, -alpha, Ap, r);        // r = r - alpha * Ap\n}\n```\n\nKeep this loop in mind. Every porting decision later in the article is really a decision about **where each line of this loop runs and where its data lives**.\n\nBefore writing a single line of CUDA, we need to know where the time goes. Conveniently, HPCCG already reports a per-kernel timing breakdown.\n\nFor a `100^3` problem, corresponding to one million rows, the original single-process CPU implementation took:\n\n| KernelTime (s)Fraction of total |  |  | \n|---|---|---|\n| SpMV | 2.819 | 83.5% | \n| WAXPBY | 0.342 | 10.1% | \n| DDOT | 0.215 | 6.4% | \n| **Total** | **3.378** | **100%** | \n\nThe solver required 149 iterations and reached a final residual of approximately `8.0 × 10⁻²¹`.\n\nSpMV is clearly the dominant operation, accounting for more than 80% of the CPU runtime. This is not particularly surprising. Each matrix row accesses neighboring elements through indexed loads, while performing relatively little arithmetic per byte moved.\n\nThis gives us our first hypothesis:\n\n**If SpMV dominates, moving SpMV to the GPU should be the highest-leverage first step.**\n\nCommit `51fdbe5` — *Starting off CUDA migration -- sparsemv is hotspot*\n\nThe first obstacle is not actually the kernel. It is the matrix representation.\n\nHPCCG stores the matrix as arrays of per-row pointers:\n\n``` js\nfor (int i = 0; i < nrow; i++) {\n  double sum = 0.0;\n\n  const double* const cur_vals = A->ptr_to_vals_in_row[i];\n  const int* const cur_inds = A->ptr_to_inds_in_row[i];\n  const int cur_nnz = A->nnz_in_row[i];\n\n  for (int j = 0; j < cur_nnz; j++)\n    sum += cur_vals[j] * x[cur_inds[j]];\n\n  y[i] = sum;\n}\n```\n\nThis is convenient on the CPU, but copying the top-level structure to device memory would not make its host pointers magically become valid GPU pointers.\n\nI therefore flattened the matrix into **Compressed Sparse Row (CSR)** form:\n\n`row_offsets[nrow + 1]`` cols[nnz]``vals[nnz]`\nEach array is contiguous and can be allocated and copied to the device independently.\n\nCommit `05ace3b` — *sparsemv device copy and free routine*\n\nThe initial CUDA kernel deliberately uses the simplest possible mapping: **one CUDA thread per matrix row**.\n\n``` js\n__global__\nvoid spmv_kernel(DeviceCSRMatrix A,\n                 const double* x,\n                 double* y)\n{\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n\n    if (i < A.nrow) {\n        double sum = 0.0;\n\n        for (int j = A.row_offsets[i];\n             j < A.row_offsets[i + 1]; ++j)\n            sum += A.vals[j] * x[A.cols[j]];\n\n        y[i] = sum;\n    }\n}\n```\n\nAt this point I intentionally did **not** optimize the sparse-memory access pattern. The goal was first to establish a correct GPU baseline.\n\nThe rest of CG still ran on the CPU, so each SpMV required the input vector to reach the GPU and the result to return to the CPU.\n\nCommit `ef5c262` — *Add CUDA CSR SpMV path to HPCCG*\n\nThis is an important intermediate stage. A GPU kernel can be much faster than its CPU counterpart while the application as a whole remains limited by everything surrounding that kernel.\n\nWAXPBY is the easiest operation to port. Each output element is independent:\n\n``` js\n__global__\nvoid waxpby_kernel(int n,\n                   double alpha, const double* x,\n                   double beta,  const double* y,\n                   double* w)\n{\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n\n    if (i < n)\n        w[i] = alpha * x[i] + beta * y[i];\n}\n```\n\nDuring the migration I first compared the CPU and GPU results before switching the actual solver call sites over.\n\nCommits `4b78d30` — *Added waxpby CUDA kernel and correctness check*\n\n`870b4bf` — *Finished migrating WAXPBY*\n\nDDOT is different. WAXPBY maps one input element to one output element; DDOT collapses an entire vector into a single scalar.\n\nI implemented it as a simple two-stage reduction.\n\nIn the first kernel, each thread accumulates a local sum using a grid-stride loop. Threads within each block then reduce those values through shared memory, producing one partial result per block.\n\nConceptually:\n\n```\nx[i] * y[i]\n      ↓\nthread-local accumulation\n      ↓\nshared-memory block reduction\n      ↓\none partial / block\n      ↓\nsecond reduction\n      ↓\ndevice scalar\n```\n\nA second kernel reduces the block-level partial results to the final device scalar.\n\nI deliberately used a custom reduction rather than immediately calling a library routine such as CUB or cuBLAS. For this exercise, seeing the reduction and its synchronization structure explicitly was part of the point.\n\nCommit `1101bf2` — *DDOT also is on GPU*\n\nBut this creates a new problem.\n\nCG needs these scalar results to compute `alpha`, `beta`, and the convergence criterion. If the vector operations live on the GPU but every reduction result immediately returns to the CPU, the algorithm repeatedly forces the two processors to synchronize.\n\nWe have accelerated all three kernels, but we have not yet fixed the application.\n\nAt this point, SpMV, WAXPBY, and DDOT all have CUDA implementations.\n\nIt is tempting to declare the port complete.\n\nIt isn't.\n\nNsight Systems showed that the hybrid implementation was repeatedly entering `cudaMemcpy`. In one intermediate profile I observed:\n\n```\ncudaMemcpy        304 calls\ncudaMalloc         10 calls\ncudaLaunchKernel 1195 calls\n```\n\nThe CUDA API attributed roughly 405 ms to the `cudaMemcpy` calls, compared with roughly 29 ms of host API time spent launching kernels.\n\nThere is an important subtlety here: **this does not mean that copying a few bytes over PCIe literally takes milliseconds.**\n\nA blocking `cudaMemcpy` is also a synchronization point. Its API duration can include time spent waiting for previously queued GPU work to complete.\n\nSo the important observation was not simply:\n\nPCIe is slow.\n\nIt was:\n\n**The application is repeatedly forcing the CPU and GPU to synchronize.**\n\nThis is a different level of performance problem.\n\nWe started with a **kernel-level** bottleneck: SpMV.\n\nAfter accelerating the kernels, we exposed an **application-dataflow** bottleneck.\n\nThe next optimization therefore does not change the mathematics of SpMV, DDOT, or WAXPBY at all.\n\nIt changes **where the data lives**.\n\nThe main CG working set consists of:\n\n```\nx\nr\np\nAp\nA\n```\n\nIn the GPU-resident version, these vectors and the CSR matrix are allocated on the GPU before entering the iteration loop. The iterative solver then operates directly on device pointers.\n\n```\n// Allocate and initialize the GPU working set once.\n\nspmv_cuda(A, d_x, d_Ap);\nwaxpby_cuda(n, 1.0, d_b, -1.0, d_Ap, d_r);\nddot_cuda(n, d_r, d_r, d_partial, d_rtrans);\n\nfor (int k = 1; k < max_iter; ++k) {\n\n    // update p on the GPU\n    ...\n\n    spmv_cuda(A, d_p, d_Ap);\n\n    ddot_cuda(n, d_p, d_Ap, d_partial, d_pAp);\n\n    // compute alpha and update x and r on the GPU\n    ...\n}\n\n// Copy final solution back.\ncudaMemcpy(x, d_x, ...);\n```\n\n\"GPU-resident\" here does **not** mean that the entire CG solver has been turned into one giant CUDA kernel.\n\nThe host still orchestrates a sequence of kernels. What changed is that the large working vectors no longer travel between CPU and GPU after every operation.\n\nCommits `a4c1be8` — *Fully GPU resident CG kernel*\n\n`bfedb51` — *Add GPU-resident native CUDA HPCCG baseline*\n\nThis produced an interesting result.\n\nFor the small `20^3` problem:\n\n| `20^3` Serial CPUGPU-resident CUDA |  |  | \n|---|---|---|\n| Total time | **16.9 ms** | 51.6 ms | \n| CG iterations | 149 | 149 | \n\nThe GPU implementation is about **3× slower**.\n\nBut for `100^3`:\n\n| `100^3` Serial CPUGPU-resident CUDA |  |  | \n|---|---|---|\n| Total time | 3.378 s | **0.468 s** | \n| CG iterations | 149 | 149 | \n\nNow the GPU implementation is approximately **7.2× faster end-to-end**.\n\nThis is one of the most useful results of the exercise.\n\nBoth problems require the same 149 CG iterations. But `20^3` contains only 8,000 rows, whereas `100^3` contains one million — **125 times as many rows**.\n\nFor the small problem, there is simply not enough useful work to amortize repeated kernel launches, reductions, and synchronization.\n\nFor the larger problem, GPU throughput dominates those fixed costs.\n\nSo rather than saying simply that \"the GPU is faster,\" I find it more useful to think in terms of an **application-level crossover**:\n\n```\nsmall problem\n    useful GPU work < GPU overhead\n              ↓\n          CPU wins\n\nlarge problem\n    useful GPU work >> GPU overhead\n              ↓\n          GPU wins\n```\n\nThere is another subtle lesson hidden in these results.\n\nHPCCG's original timing infrastructure was written for synchronous CPU functions. When it surrounds a CUDA kernel launch, however, the host can return before the GPU has actually finished executing the kernel.\n\nFor example, the GPU-resident `100^3` run reports:\n\n```\nTotal:     0.468121 s\nDDOT:      0.002686 s\nWAXPBY:    0.002234 s\nSPARSEMV:  0.002139 s\n```\n\nThose per-operation numbers should **not** be interpreted as accumulated GPU execution times. They largely reflect asynchronous host-side launch behavior.\n\nI therefore use the application's total wall-clock time for the end-to-end comparison, and Nsight Systems or Nsight Compute when reasoning about individual CUDA operations.\n\nPorting an application to an asynchronous execution model can change not only its performance, but also the meaning of its existing profiler.\n\nOnce the large vector transfers were gone, the smaller synchronization points became visible.\n\nOne particularly simple example was the residual scalar.\n\nInitially, before computing the next residual, I preserved the previous value using a device-to-device copy.\n\nBut there was no reason to move the data at all.\n\nI already had two scalar buffers. They could simply exchange roles:\n\n```\nstd::swap(d_rtrans, d_oldrtrans);\n\nddot_cuda(n, d_r, d_r, d_partial, d_rtrans);\n\ncompute_division<<<1,1>>>(\n    d_rtrans,\n    d_oldrtrans,\n    d_beta\n);\n```\n\nThe swap changes two host-side pointer values. No GPU data moves.\n\nThat one change reduced the number of `cudaMemcpy` calls in the profile from:\n\n```\n304 → 156\n```\n\nExactly **148 copies disappeared**.\n\nThe interesting part is what happened next.\n\nThe total API time attributed to `cudaMemcpy` barely changed, and end-to-end runtime improved only modestly, from roughly 0.55 s to around 0.49 s in those runs.\n\nWhy?\n\nBecause the removed copies were cheap device-to-device scalar copies. The expensive calls that remained were primarily host-visible synchronization points.\n\nThis is another useful profiling lesson:\n\n**Removing many operations is not necessarily the same thing as removing the expensive operations.**\n\nOnce application dataflow was no longer dominating the discussion, I returned to the original hotspot: SpMV.\n\nNsight Compute showed the following picture for the `100^3` one-thread-per-row CSR kernel:\n\n| MetricMeasurement |  | \n|---|---|\n| Achieved occupancy | 90.9% | \n| Theoretical occupancy | 100% | \n| DRAM throughput | 63.3% of peak | \n| SM compute throughput | 11.7% | \n| Active warps / scheduler | ~11.1 | \n| Eligible warps / scheduler | ~0.06 | \n| L1 hit rate | 54.0% | \n| L2 hit rate | 63.1% | \n\nAt first glance, the occupancy looks excellent.\n\nBut occupancy only tells us how many warps can reside on the SM. It does not tell us how many of those warps are actually ready to issue an instruction.\n\nHere, roughly 11 warps were active per scheduler, while only about **0.06 warps were eligible** on average. The scheduler had no eligible warp during approximately **98% of sampled cycles**.\n\nThe source counters provided another clue: Nsight Compute reported approximately **43.7 million excessive memory sectors**, about **72% of the total sectors**, associated with uncoalesced global accesses.\n\nSo increasing occupancy would be the wrong first optimization target. There are already plenty of resident warps.\n\nThe more interesting problem is **memory access efficiency**.\n\nThe indirect access\n\n```\nx[A.cols[j]]\n```\n\nis an obvious candidate, but it is not necessarily the only culprit. In a one-thread-per-row CSR mapping, neighboring threads also walk different segments of `vals` and `cols`.\n\nNsight Compute tells us that the current memory access pattern is inefficient. Determining exactly which loads dominate would be the next controlled experiment.\n\nAnd that is where I intentionally stop optimizing this CUDA implementation for now.\n\nThe point of this exercise was not to squeeze every last percent out of CSR SpMV. We now have enough information to ask a different question:\n\n**What happens if we solve the same porting problem using a performance-portability abstraction instead of writing CUDA directly?**", "url": "https://wpnews.pro/news/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda", "canonical_source": "https://dev.to/junsik_yoo/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda-aih", "published_at": "2026-09-16 20:39:48+00:00", "updated_at": "2026-09-16 20:53:10.823844+00:00", "lang": "en", "topics": ["mlops", "developer-tools", "ai-infrastructure"], "entities": ["HPCCG", "Mantevo", "CUDA", "Kokkos", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda", "markdown": "https://wpnews.pro/news/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda.md", "text": "https://wpnews.pro/news/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda.txt", "jsonld": "https://wpnews.pro/news/a-pedagogical-introduction-to-porting-a-conjugate-gradient-solver-to-cuda.jsonld"}}