{"slug": "testing-cuda-kernel-execution-with-gpu-correctness-harness", "title": "Testing CUDA kernel execution with GPU correctness harness", "summary": "A new CUDA kernel correctness harness compiles kernels with nvcc, loads them into Python via ctypes, and checks outputs against NumPy baselines, with timing via CUDA events. All 23 tests pass on an NVIDIA T4 GPU, covering vector add, matrix multiplication with tile sizes including 257, reduction, and softmax overflow cases.", "body_md": "CUDA kernels don't throw exceptions when they're wrong. A misaligned tile boundary, a warp reading half the reduction, exp() overflowing to inf - the output is just silently incorrect. This harness surfaces those failure modes.\n\nKernels compile via nvcc, load into Python through ctypes, and get checked against NumPy baselines. Timing measurements use CUDA events inside the compiled library, not wall-clock.\n\n```\n+-----------------------------------------------------------------------+\n|                          PyTest Framework                             |\n|      (Boundary Analysis, Softmax Overflow, Edge-Case Vectors)         |\n+-----------------------------------------------------------------------+\n                                  |\n                        (ctypes C-ABI Wrapper)\n                                  v\n+-----------------------------------------------------------------------+\n|                  C++/CUDA Shared Library (.so)                        |\n|  +---------------------------------------------------------------+    |\n|  | CUDA Event Timer (Start → Launch Kernel → Stop)               |    |\n|  +---------------------------------------------------------------+    |\n|  | Kernels: vector_add | matrix_mul | reduction | softmax        |    |\n|  +---------------------------------------------------------------+    |\n+-----------------------------------------------------------------------+\n                                  |\n                          (Device Execution)\n                                  v\n+-----------------------------------------------------------------------+\n|                      NVIDIA GPU (T4)                                  |\n|         Warp-Shuffle Intrinsic (__shfl_down_sync) Execution           |\n+-----------------------------------------------------------------------+\n```\n\nN=256 and N=257 are not the same test. One fills a block exactly. The other has a partial block where off-by-one in ceiling division silently drops the last element.\n\nA tiled GEMM parameterized over tile sizes [16, 32, 64, 128, 257]. Everything passes at powers of two. 257 is where index math breaks.\n\nShared memory tree down to 32 lanes, then __shfl_down_sync for the final warp. No bank conflicts, no unnecessary syncs.\n\nTwo-pass stable algorithm. exp(1000) is inf in FP32. The naive kernel fails this test.\n\n- CUDA Toolkit 11.x or higher\n- Python 3.10 or higher\n- Make build system\n- NVIDIA GPU (tested on T4)\n\nClean build:\n\n```\nmake clean && make\n```\n\nRun the full test suite:\n\n```\npytest tests/ -v\n```\n\nRun performance benchmarks:\n\n```\npython benchmarks/bench.py\n```\n\nAll 23 tests passing on NVIDIA T4:\n\n```\ntests/test_correctness.py::TestVectorAdd::test_equivalence_random            PASSED\ntests/test_correctness.py::TestVectorAdd::test_single_element                PASSED\ntests/test_correctness.py::TestVectorAdd::test_non_power_of_two_length       PASSED\ntests/test_correctness.py::TestVectorAdd::test_exact_block_boundary          PASSED\ntests/test_correctness.py::TestVectorAdd::test_mismatched_shapes_raises      PASSED\ntests/test_correctness.py::TestVectorAdd::test_wrong_dtype_raises            PASSED\ntests/test_correctness.py::TestMatrixMul::test_matrix_mul_equivalence[16]    PASSED\ntests/test_correctness.py::TestMatrixMul::test_matrix_mul_equivalence[32]    PASSED\ntests/test_correctness.py::TestMatrixMul::test_matrix_mul_equivalence[64]    PASSED\ntests/test_correctness.py::TestMatrixMul::test_matrix_mul_equivalence[128]   PASSED\ntests/test_correctness.py::TestMatrixMul::test_matrix_mul_equivalence[257]   PASSED\ntests/test_correctness.py::TestReduction::test_sum_random                    PASSED\ntests/test_correctness.py::TestReduction::test_sum_single_element            PASSED\ntests/test_correctness.py::TestReduction::test_sum_exact_block_boundary      PASSED\ntests/test_correctness.py::TestReduction::test_sum_non_power_of_two          PASSED\ntests/test_correctness.py::TestReduction::test_sum_multi_block               PASSED\ntests/test_correctness.py::TestReduction::test_all_zeros                     PASSED\ntests/test_correctness.py::TestSoftmax::test_equivalence_random              PASSED\ntests/test_correctness.py::TestSoftmax::test_output_sums_to_one             PASSED\ntests/test_correctness.py::TestSoftmax::test_numerical_stability_large_values PASSED\ntests/test_correctness.py::TestSoftmax::test_uniform_input_gives_uniform_output PASSED\ntests/test_correctness.py::TestSoftmax::test_single_element                  PASSED\ntests/test_correctness.py::TestSoftmax::test_oversized_input_raises          PASSED\n\n23 passed in 0.45s\n```\n\nCUDA event timing with warmup=5, 20 runs per kernel. NVIDIA T4 peak bandwidth is approximately 320 GB/s.\n\n| Kernel | Input Size | Avg Time | Bandwidth |\n|---|---|---|---|\n| vector_add | 16M elements | 0.765 ms | 263.2 GB/s |\n| reduce_sum | 16M elements | 0.629 ms | 106.7 GB/s |\n| matrix_mul | 512x512 | 0.342 ms | 9.2 GB/s |\n| softmax | 1024 | 0.018 ms | 0.9 GB/s |\n\nVector add achieves 263 GB/s, which is 82% of T4 peak. This is expected for a memory-bound kernel.\n\nReduce sum achieves 107 GB/s. The two-phase design adds overhead. Specialized libraries like CUB achieve closer to 250 GB/s with fused algorithms.\n\nMatrix multiply bandwidth is intentionally low. This is a naive GEMM with poor arithmetic intensity. The relevant metric here is correctness at edge cases like N=257, not peak performance.\n\n- FP16 and BF16 kernel variants\n- Online softmax algorithm for sequences beyond N=1024\n- NVTX markers for Nsight timeline profiling\n- CI integration with GPU runner\n\nThis harness validates CUDA kernel correctness across:\n\n- Non-power-of-two dimensions\n- Boundary conditions and partial blocks\n- Numerical stability under extreme values\n- Warp-level synchronization and memory safety\n- Real-world edge case dimensions (257 is not random)\n\nThe goal is to catch silent failures before they reach production, validating both the kernel logic and the infrastructure that wraps it.", "url": "https://wpnews.pro/news/testing-cuda-kernel-execution-with-gpu-correctness-harness", "canonical_source": "https://github.com/IshwaryaRavichandran/gpu-correctness-harness", "published_at": "2026-08-21 05:55:00+00:00", "updated_at": "2026-08-21 06:13:43.179734+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning"], "entities": ["NVIDIA T4", "CUDA", "NumPy", "PyTest"], "alternates": {"html": "https://wpnews.pro/news/testing-cuda-kernel-execution-with-gpu-correctness-harness", "markdown": "https://wpnews.pro/news/testing-cuda-kernel-execution-with-gpu-correctness-harness.md", "text": "https://wpnews.pro/news/testing-cuda-kernel-execution-with-gpu-correctness-harness.txt", "jsonld": "https://wpnews.pro/news/testing-cuda-kernel-execution-with-gpu-correctness-harness.jsonld"}}