Testing CUDA kernel execution with GPU correctness harness

A new CUDA kernel correctness harness compiles kernels with nvcc, loads them into Python via ctypes, and checks outputs against NumPy baselines, with timing via CUDA events. All 23 tests pass on an NVIDIA T4 GPU, covering vector add, matrix multiplication with tile sizes including 257, reduction, and softmax overflow cases.

CUDA kernels don't throw exceptions when they're wrong. A misaligned tile boundary, a warp reading half the reduction, exp overflowing to inf - the output is just silently incorrect. This harness surfaces those failure modes. Kernels compile via nvcc, load into Python through ctypes, and get checked against NumPy baselines. Timing measurements use CUDA events inside the compiled library, not wall-clock. +-----------------------------------------------------------------------+ | PyTest Framework | | Boundary Analysis, Softmax Overflow, Edge-Case Vectors | +-----------------------------------------------------------------------+ | ctypes C-ABI Wrapper v +-----------------------------------------------------------------------+ | C++/CUDA Shared Library .so | | +---------------------------------------------------------------+ | | | CUDA Event Timer Start → Launch Kernel → Stop | | | +---------------------------------------------------------------+ | | | Kernels: vector add | matrix mul | reduction | softmax | | | +---------------------------------------------------------------+ | +-----------------------------------------------------------------------+ | Device Execution v +-----------------------------------------------------------------------+ | NVIDIA GPU T4 | | Warp-Shuffle Intrinsic shfl down sync Execution | +-----------------------------------------------------------------------+ N=256 and N=257 are not the same test. One fills a block exactly. The other has a partial block where off-by-one in ceiling division silently drops the last element. A tiled GEMM parameterized over tile sizes 16, 32, 64, 128, 257 . Everything passes at powers of two. 257 is where index math breaks. Shared memory tree down to 32 lanes, then shfl down sync for the final warp. No bank conflicts, no unnecessary syncs. Two-pass stable algorithm. exp 1000 is inf in FP32. The naive kernel fails this test. - CUDA Toolkit 11.x or higher - Python 3.10 or higher - Make build system - NVIDIA GPU tested on T4 Clean build: make clean && make Run the full test suite: pytest tests/ -v Run performance benchmarks: python benchmarks/bench.py All 23 tests passing on NVIDIA T4: tests/test correctness.py::TestVectorAdd::test equivalence random PASSED tests/test correctness.py::TestVectorAdd::test single element PASSED tests/test correctness.py::TestVectorAdd::test non power of two length PASSED tests/test correctness.py::TestVectorAdd::test exact block boundary PASSED tests/test correctness.py::TestVectorAdd::test mismatched shapes raises PASSED tests/test correctness.py::TestVectorAdd::test wrong dtype raises PASSED tests/test correctness.py::TestMatrixMul::test matrix mul equivalence 16 PASSED tests/test correctness.py::TestMatrixMul::test matrix mul equivalence 32 PASSED tests/test correctness.py::TestMatrixMul::test matrix mul equivalence 64 PASSED tests/test correctness.py::TestMatrixMul::test matrix mul equivalence 128 PASSED tests/test correctness.py::TestMatrixMul::test matrix mul equivalence 257 PASSED tests/test correctness.py::TestReduction::test sum random PASSED tests/test correctness.py::TestReduction::test sum single element PASSED tests/test correctness.py::TestReduction::test sum exact block boundary PASSED tests/test correctness.py::TestReduction::test sum non power of two PASSED tests/test correctness.py::TestReduction::test sum multi block PASSED tests/test correctness.py::TestReduction::test all zeros PASSED tests/test correctness.py::TestSoftmax::test equivalence random PASSED tests/test correctness.py::TestSoftmax::test output sums to one PASSED tests/test correctness.py::TestSoftmax::test numerical stability large values PASSED tests/test correctness.py::TestSoftmax::test uniform input gives uniform output PASSED tests/test correctness.py::TestSoftmax::test single element PASSED tests/test correctness.py::TestSoftmax::test oversized input raises PASSED 23 passed in 0.45s CUDA event timing with warmup=5, 20 runs per kernel. NVIDIA T4 peak bandwidth is approximately 320 GB/s. | Kernel | Input Size | Avg Time | Bandwidth | |---|---|---|---| | vector add | 16M elements | 0.765 ms | 263.2 GB/s | | reduce sum | 16M elements | 0.629 ms | 106.7 GB/s | | matrix mul | 512x512 | 0.342 ms | 9.2 GB/s | | softmax | 1024 | 0.018 ms | 0.9 GB/s | Vector add achieves 263 GB/s, which is 82% of T4 peak. This is expected for a memory-bound kernel. Reduce sum achieves 107 GB/s. The two-phase design adds overhead. Specialized libraries like CUB achieve closer to 250 GB/s with fused algorithms. Matrix multiply bandwidth is intentionally low. This is a naive GEMM with poor arithmetic intensity. The relevant metric here is correctness at edge cases like N=257, not peak performance. - FP16 and BF16 kernel variants - Online softmax algorithm for sequences beyond N=1024 - NVTX markers for Nsight timeline profiling - CI integration with GPU runner This harness validates CUDA kernel correctness across: - Non-power-of-two dimensions - Boundary conditions and partial blocks - Numerical stability under extreme values - Warp-level synchronization and memory safety - Real-world edge case dimensions 257 is not random The goal is to catch silent failures before they reach production, validating both the kernel logic and the infrastructure that wraps it.