{"slug": "gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption", "title": "gemmcheck3: self-checking rocBLAS GEMM test (RX 7900 XTX corruption)", "summary": "A developer released gemmcheck3, a self-checking rocBLAS GEMM stress test for AMD GPUs targeting RDNA3/gfx11 hardware such as the RX 7900 XTX. The tool runs five rocblas_gemm_ex calls in LLM layer shapes (fp16-in/fp32-out and int8-in/int32-out) back to back on a single stream, comparing each output bit-for-bit against a startup reference and recording launch number, word index, and values on any mismatch. It supports fault injection for self-testing and exits with distinct codes for clean runs, mismatches, HIP/rocBLAS errors, and failed determinism checks.", "body_md": "|  | // gemmcheck3: a self-checking rocBLAS GEMM stress test for AMD GPUs (RDNA3 / gfx11 tested). | \n|  | // | \n|  | // Five rocblas_gemm_ex calls in LLM layer shapes (fp16 in / fp32 out, and int8 in / int32 out), random inputs, | \n|  | // atomics disabled, launched back to back on one stream. A reference is computed once at startup and a | \n|  | // determinism check must pass. After every GEMM a compare kernel checks the output bit for bit; on a mismatch it | \n|  | // records the launch number, word index, the value read, the reference, and both re-read through an atomic | \n|  | // (served by L2). Every few seconds the host prints a status line and describes any bad launch: GEMM | \n|  | // coordinates (rows, columns), checker coordinates, magnitudes, and whether the re-read agrees. | \n|  | // | \n|  | // On a healthy GPU with a deterministic library, a mismatch should never happen. | \n|  | // | \n|  | //   hipcc -O2 --offload-arch=gfx1100 gemmcheck3.hip -o gemmcheck3 -L$ROCM_PATH/lib -lrocblas -Wl,-rpath,$ROCM_PATH/lib | \n|  | //   GEMMCHECK_STOP=1 ./gemmcheck3 600 5                run up to 600 s, report every 5 s, stop at the first mismatch | \n|  | //   GEMMCHECK_INJECT=7 ./gemmcheck3 10 5               self-test: plants a known 64x16 fault at launch 7 (exit 1) | \n|  | //   GEMMCHECK_GRID=512,128 ./gemmcheck3 ...            checker geometry (blocks,threads), default 1024,256 | \n|  | //   GEMMCHECK_TRACE=1 AMD_LOG_LEVEL=3 ./gemmcheck3 1 1  one launch per case between markers, then exit | \n|  | // Exit codes: 0 clean, 1 mismatches found, 2 HIP/rocBLAS error, 3 determinism check failed. | \n|  | #include <hip/hip_runtime.h> | \n|  | #include <hip/hip_fp16.h> | \n|  | #include <rocblas/rocblas.h> | \n|  | #include <algorithm> | \n|  | #include <chrono> | \n|  | #include <cmath> | \n|  | #include <cstdint> | \n|  | #include <cstdio> | \n|  | #include <cstdlib> | \n|  | #include <cstring> | \n|  | #include <map> | \n|  | #include <random> | \n|  | #include <set> | \n|  | #include <utility> | \n|  | #include <vector> | \n|  |  | \n|  | #define CK(x) do { hipError_t e_ = (x); if (e_ != hipSuccess) { fprintf(stderr, \"HIP %s at %s:%d\\n\", hipGetErrorString(e_), __FILE__, __LINE__); exit(2); } } while (0) | \n|  | #define RB(x) do { rocblas_status s_ = (x); if (s_ != rocblas_status_success) { fprintf(stderr, \"rocBLAS %s at %s:%d\\n\", rocblas_status_to_string(s_), __FILE__, __LINE__); exit(2); } } while (0) | \n|  |  | \n|  | struct Rec { | \n|  | unsigned long long launch; | \n|  | unsigned int ci, idx, got, want, got2, want2; // got2/want2: the same words read again through an atomic | \n|  | }; | \n|  |  | \n|  | // counts 32-bit words that differ and records each one (up to cap); first keeps one differing index. | \n|  | // got2/want2 are atomic read-modify-writes of 0: served by L2 past the per-CU caches, but an RMW, not a passive probe | \n|  | __global__ void compare_words(uint32_t * a, uint32_t * ref, size_t n, unsigned long long * count, unsigned long long * first, | \n|  | Rec * recs, unsigned long long * nrec, unsigned long long cap, unsigned long long launch, int ci) { | \n|  | size_t i = (size_t) blockIdx.x * blockDim.x + threadIdx.x; | \n|  | size_t stride = (size_t) gridDim.x * blockDim.x; | \n|  | unsigned long long local = 0; | \n|  | for (; i < n; i += stride) { | \n|  | const uint32_t got = a[i], want = ref[i]; | \n|  | if (got != want) { | \n|  | // re-read first, before any counter traffic can delay it | \n|  | const uint32_t got2 = atomicAdd(&a[i], 0u), want2 = atomicAdd(&ref[i], 0u); | \n|  | if (local == 0) { atomicCAS(first, ~0ull, (unsigned long long) i); } | \n|  | local++; | \n|  | const unsigned long long slot = atomicAdd(nrec, 1ull); | \n|  | if (slot < cap) { | \n|  | Rec r; | \n|  | r.launch = launch; r.ci = (unsigned int) ci; r.idx = (unsigned int) i; r.got = got; r.want = want; | \n|  | r.got2 = got2; r.want2 = want2; | \n|  | recs[slot] = r; | \n|  | } | \n|  | } | \n|  | } | \n|  | if (local) { atomicAdd(count, local); } | \n|  | } | \n|  |  | \n|  | // test hook (GEMMCHECK_INJECT=launch): flips bit 0 of rows 100-163, columns 5-20 of that launch's output | \n|  | __global__ void inject(uint32_t * c, int m) { | \n|  | const int row = 100 + threadIdx.x, col = 5 + blockIdx.x; | \n|  | c[(size_t) col * m + row] ^= 1u; | \n|  | } | \n|  |  | \n|  | struct Case { | \n|  | const char * name; | \n|  | int m, n, k; | \n|  | bool int8; | \n|  | void * a = nullptr; void * b = nullptr; void * c = nullptr; void * ref = nullptr; | \n|  | size_t cwords = 0; | \n|  | unsigned long long * d_count = nullptr; unsigned long long * d_first = nullptr; | \n|  | unsigned long long launches = 0; | \n|  | }; | \n|  |  | \n|  | static void run_gemm(rocblas_handle h, Case & cs, void * out) { | \n|  | if (cs.int8) { | \n|  | int32_t alpha = 1, beta = 0; | \n|  | RB(rocblas_gemm_ex(h, rocblas_operation_transpose, rocblas_operation_none, cs.m, cs.n, cs.k, &alpha, | \n|  | cs.a, rocblas_datatype_i8_r, cs.k, cs.b, rocblas_datatype_i8_r, cs.k, &beta, | \n|  | out, rocblas_datatype_i32_r, cs.m, out, rocblas_datatype_i32_r, cs.m, | \n|  | rocblas_datatype_i32_r, rocblas_gemm_algo_standard, 0, rocblas_gemm_flags_none)); | \n|  | } else { | \n|  | float alpha = 1.0f, beta = 0.0f; | \n|  | RB(rocblas_gemm_ex(h, rocblas_operation_transpose, rocblas_operation_none, cs.m, cs.n, cs.k, &alpha, | \n|  | cs.a, rocblas_datatype_f16_r, cs.k, cs.b, rocblas_datatype_f16_r, cs.k, &beta, | \n|  | out, rocblas_datatype_f32_r, cs.m, out, rocblas_datatype_f32_r, cs.m, | \n|  | rocblas_datatype_f32_r, rocblas_gemm_algo_standard, 0, rocblas_gemm_flags_none)); | \n|  | } | \n|  | } | \n|  |  | \n|  | static float as_float(uint32_t u) { float f; memcpy(&f, &u, 4); return f; } | \n|  |  | \n|  | // one bad launch of one case: where the wrong words sit and what they look like | \n|  | static void describe(const std::vector<Case> & cases, unsigned long long launch, unsigned int ci, std::vector<Rec> & rs, | \n|  | int blocks, int threads) { | \n|  | const Case & cs = cases[ci]; | \n|  | const size_t T = (size_t) blocks * threads; | \n|  | std::sort(rs.begin(), rs.end(), [](const Rec & x, const Rec & y) { return x.idx < y.idx; }); | \n|  | std::set<int> cols, blks, iters; | \n|  | int rmin = cs.m, rmax = -1, cmin = cs.n, cmax = -1; | \n|  | size_t reread_differs = 0, reread_eq_want = 0, reread_eq_want2 = 0, ref_reread_differs = 0, nonfinite_got = 0, nonfinite_want = 0, ulp_small = 0; | \n|  | size_t xor1 = 0; | \n|  | std::set<unsigned int> uniq; | \n|  | double maxdiff = 0, maxref = 0; | \n|  | for (const Rec & r : rs) { | \n|  | const int row = (int) (r.idx % cs.m), col = (int) (r.idx / cs.m); | \n|  | rmin = std::min(rmin, row); rmax = std::max(rmax, row); cmin = std::min(cmin, col); cmax = std::max(cmax, col); | \n|  | cols.insert(col); | \n|  | const size_t tid = r.idx % T; | \n|  | blks.insert((int) (tid / threads)); iters.insert((int) (r.idx / T)); | \n|  | uniq.insert(r.idx); | \n|  | if ((r.got ^ r.want) == 1u) { xor1++; } | \n|  | if (r.got2 != r.got) { reread_differs++; } | \n|  | if (r.got2 == r.want) { reread_eq_want++; } | \n|  | if (r.got2 == r.want2) { reread_eq_want2++; } | \n|  | if (r.want2 != r.want) { ref_reread_differs++; } | \n|  | if (cs.int8) { | \n|  | maxdiff = std::max(maxdiff, std::fabs((double) (int32_t) r.got - (double) (int32_t) r.want)); | \n|  | maxref = std::max(maxref, std::fabs((double) (int32_t) r.want)); | \n|  | } else { | \n|  | const float g = as_float(r.got), w = as_float(r.want); | \n|  | const bool gf = std::isfinite(g), wf = std::isfinite(w); | \n|  | if (!gf) { nonfinite_got++; } | \n|  | if (!wf) { nonfinite_want++; } | \n|  | if (gf && wf) { | \n|  | maxdiff = std::max(maxdiff, std::fabs((double) g - (double) w)); | \n|  | if (((r.got ^ r.want) & 0x80000000u) == 0 && (r.got > r.want ? r.got - r.want : r.want - r.got) <= 16) { ulp_small++; } | \n|  | } | \n|  | if (wf) { maxref = std::max(maxref, std::fabs((double) w)); } | \n|  | } | \n|  | } | \n|  | printf(\"\\n  EVENT launch %llu, %s: %zu wrong words recorded\\n\", launch, cs.name, rs.size()); | \n|  | printf(\"    GEMM coordinates: rows %d-%d, columns %d-%d (%zu distinct columns)\\n\", rmin, rmax, cmin, cmax, cols.size()); | \n|  | printf(\"    checker coordinates (%d x %d): %zu distinct blocks (%d-%d), %zu distinct iterations (%d-%d)\\n\", blocks, threads, | \n|  | blks.size(), *blks.begin(), *blks.rbegin(), iters.size(), *iters.begin(), *iters.rbegin()); | \n|  | printf(\"    second read (atomic, L2): differs from first read %zu, equals want %zu, equals want2 %zu; want2 differs from want %zu\\n\", | \n|  | reread_differs, reread_eq_want, reread_eq_want2, ref_reread_differs); | \n|  | printf(\"    checks: unique indices %zu, got^want==1 %zu, got2==got %zu, want2==want %zu, records %zu\\n\", | \n|  | uniq.size(), xor1, rs.size() - reread_differs, rs.size() - ref_reread_differs, rs.size()); | \n|  | if (cs.int8) { | \n|  | printf(\"    largest \\|got - want\\| %.0f (largest \\|want\\| %.0f)\\n\", maxdiff, maxref); | \n|  | } else { | \n|  | printf(\"    NaN/Inf in got %zu, in want %zu; within 16 ulp (both finite) %zu; largest finite \\|got - want\\| %.4g (largest \\|want\\| %.4g)\\n\", | \n|  | nonfinite_got, nonfinite_want, ulp_small, maxdiff, maxref); | \n|  | } | \n|  | for (size_t j = 0; j < rs.size() && j < 6; j++) { | \n|  | const Rec & r = rs[j]; | \n|  | printf(\"    word %u (row %u col %u, checker block %zu iter %zu): got 0x%08x want 0x%08x got2 0x%08x want2 0x%08x\\n\", r.idx, | \n|  | r.idx % cs.m, r.idx / cs.m, (r.idx % T) / threads, (size_t) (r.idx / T), r.got, r.want, r.got2, r.want2); | \n|  | } | \n|  | fflush(stdout); | \n|  | } | \n|  |  | \n|  | int main(int argc, char ** argv) { | \n|  | const double seconds = argc > 1 ? atof(argv[1]) : 600; | \n|  | const double every   = argc > 2 ? atof(argv[2]) : 30; | \n|  | int blocks = 1024, threads = 256; | \n|  | if (const char * g = getenv(\"GEMMCHECK_GRID\")) { | \n|  | if (sscanf(g, \"%d,%d\", &blocks, &threads) != 2 \\|\\| blocks < 1 \\|\\| blocks > 65535 \\|\\| threads < 32 \\|\\| threads > 1024 \\|\\| threads % 32) { | \n|  | fprintf(stderr, \"bad GEMMCHECK_GRID '%s' (want blocks,threads with 1..65535 blocks and 32..1024 threads, a multiple of 32)\\n\", g); | \n|  | return 2; | \n|  | } | \n|  | } | \n|  | const unsigned long long inject_at = getenv(\"GEMMCHECK_INJECT\") ? strtoull(getenv(\"GEMMCHECK_INJECT\"), nullptr, 10) : 0; | \n|  | const bool trace = getenv(\"GEMMCHECK_TRACE\") != nullptr; | \n|  | const bool stop_on_mismatch = getenv(\"GEMMCHECK_STOP\") != nullptr; | \n|  | const unsigned long long cap = 1ull << 16; | \n|  |  | \n|  | // Qwen3.8-27B at ubatch 512: ffn gate/up, ffn down, fused QKV; hidden 5120, ffn 17408, qkv 12288 | \n|  | std::vector<Case> cases = { | \n|  | {\"f16 ffn_up   17408x512x5120\",  17408, 512, 5120,  false}, | \n|  | {\"f16 ffn_down  5120x512x17408\",  5120, 512, 17408, false}, | \n|  | {\"i8  ffn_up   17408x512x5120\",  17408, 512, 5120,  true}, | \n|  | {\"i8  ffn_down  5120x512x17408\",  5120, 512, 17408, true}, | \n|  | {\"i8  qkv      12288x512x5120\",  12288, 512, 5120,  true}, | \n|  | }; | \n|  |  | \n|  | rocblas_handle h; RB(rocblas_create_handle(&h)); | \n|  | RB(rocblas_set_atomics_mode(h, rocblas_atomics_not_allowed)); | \n|  | hipStream_t st; CK(hipStreamCreate(&st)); RB(rocblas_set_stream(h, st)); | \n|  |  | \n|  | // allocation order identical to gemmcheck.hip; the record buffers come after all of it | \n|  | std::mt19937 rng(12345); | \n|  | for (auto & cs : cases) { | \n|  | size_t na = (size_t) cs.m * cs.k, nb = (size_t) cs.k * cs.n, nc = (size_t) cs.m * cs.n; | \n|  | size_t es = cs.int8 ? 1 : 2; | \n|  | std::vector<uint8_t> ha(na * es), hb(nb * es); | \n|  | if (cs.int8) { | \n|  | std::uniform_int_distribution<int> d(-127, 127); | \n|  | for (auto & v : ha) v = (uint8_t) (int8_t) d(rng); | \n|  | for (auto & v : hb) v = (uint8_t) (int8_t) d(rng); | \n|  | } else { | \n|  | std::uniform_real_distribution<float> d(-1.0f, 1.0f); | \n|  | __half * pa = (__half *) ha.data(); __half * pb = (__half *) hb.data(); | \n|  | for (size_t i = 0; i < na; i++) pa[i] = __float2half(d(rng)); | \n|  | for (size_t i = 0; i < nb; i++) pb[i] = __float2half(d(rng)); | \n|  | } | \n|  | CK(hipMalloc(&cs.a, na * es)); CK(hipMalloc(&cs.b, nb * es)); | \n|  | CK(hipMalloc(&cs.c, nc * 4)); CK(hipMalloc(&cs.ref, nc * 4)); | \n|  | CK(hipMemcpy(cs.a, ha.data(), na * es, hipMemcpyHostToDevice)); | \n|  | CK(hipMemcpy(cs.b, hb.data(), nb * es, hipMemcpyHostToDevice)); | \n|  | cs.cwords = nc; | \n|  | CK(hipMalloc(&cs.d_count, sizeof(unsigned long long))); CK(hipMemset(cs.d_count, 0, sizeof(unsigned long long))); | \n|  | CK(hipMalloc(&cs.d_first, sizeof(unsigned long long))); CK(hipMemset(cs.d_first, 0xff, sizeof(unsigned long long))); | \n|  | } | \n|  | // reference results, then a determinism check: recompute each and require bit-identical output. The record | \n|  | // buffers are allocated only after rocBLAS's first GEMMs, to reduce perturbation of its lazily allocated workspace (identical | \n|  | // addresses are not guaranteed) | \n|  | for (auto & cs : cases) { run_gemm(h, cs, cs.ref); } | \n|  | CK(hipStreamSynchronize(st)); | \n|  | Rec * d_recs; unsigned long long * d_nrec; | \n|  | CK(hipMalloc(&d_recs, sizeof(Rec) * cap)); CK(hipMalloc(&d_nrec, sizeof(unsigned long long))); CK(hipMemset(d_nrec, 0, sizeof(unsigned long long))); | \n|  | for (size_t ci = 0; ci < cases.size(); ci++) { | \n|  | auto & cs = cases[ci]; | \n|  | run_gemm(h, cs, cs.c); | \n|  | compare_words<<<blocks, threads, 0, st>>>((uint32_t *) cs.c, (uint32_t *) cs.ref, cs.cwords, cs.d_count, cs.d_first, d_recs, d_nrec, cap, 0, (int) ci); | \n|  | } | \n|  | CK(hipStreamSynchronize(st)); | \n|  | for (auto & cs : cases) { | \n|  | unsigned long long c = 0; CK(hipMemcpy(&c, cs.d_count, sizeof c, hipMemcpyDeviceToHost)); | \n|  | if (c) { printf(\"NONDETERMINISTIC: %s differs from its own reference (%llu words) before the stress starts; results would be meaningless\\n\", cs.name, c); return 3; } | \n|  | } | \n|  | CK(hipMemset(d_nrec, 0, sizeof(unsigned long long))); | \n|  | printf(\"determinism check passed for %zu cases; checker %d x %d; stressing for %.0f s\\n\", cases.size(), blocks, threads, seconds); fflush(stdout); | \n|  |  | \n|  | if (trace) { | \n|  | for (size_t ci = 0; ci < cases.size(); ci++) { | \n|  | fprintf(stderr, \"MARK begin case %zu %s\\n\", ci, cases[ci].name); | \n|  | run_gemm(h, cases[ci], cases[ci].c); | \n|  | CK(hipStreamSynchronize(st)); | \n|  | fprintf(stderr, \"MARK end case %zu\\n\", ci); | \n|  | } | \n|  | return 0; | \n|  | } | \n|  |  | \n|  | const auto t0 = std::chrono::steady_clock::now(); | \n|  | auto last = t0; | \n|  | unsigned long long iters = 0, total_bad = 0, launch = 0; | \n|  | unsigned long long seen = 0; | \n|  | std::vector<unsigned long long> prev(cases.size(), 0); | \n|  | while (true) { | \n|  | for (size_t ci = 0; ci < cases.size(); ci++) { | \n|  | auto & cs = cases[ci]; | \n|  | run_gemm(h, cs, cs.c); | \n|  | if (++launch == inject_at) { inject<<<16, 64, 0, st>>>((uint32_t *) cs.c, cs.m); } | \n|  | compare_words<<<blocks, threads, 0, st>>>((uint32_t *) cs.c, (uint32_t *) cs.ref, cs.cwords, cs.d_count, cs.d_first, d_recs, d_nrec, cap, launch, (int) ci); | \n|  | cs.launches++; | \n|  | } | \n|  | iters++; | \n|  | if ((iters & 63) == 0) { | \n|  | CK(hipStreamSynchronize(st)); | \n|  | const auto now = std::chrono::steady_clock::now(); | \n|  | const double el = std::chrono::duration<double>(now - t0).count(); | \n|  | const bool done = el >= seconds; | \n|  | if (std::chrono::duration<double>(now - last).count() >= every \\|\\| done) { | \n|  | last = now; | \n|  | total_bad = 0; | \n|  | printf(\"t=%6.0fs iters=%llu\", el, iters); | \n|  | for (size_t i = 0; i < cases.size(); i++) { | \n|  | unsigned long long c = 0; | \n|  | CK(hipMemcpy(&c, cases[i].d_count, sizeof c, hipMemcpyDeviceToHost)); | \n|  | total_bad += c; | \n|  | if (c != prev[i]) { printf(\"\\n  MISMATCH %s: +%llu words (%llu so far)\", cases[i].name, c - prev[i], c); prev[i] = c; } | \n|  | } | \n|  | printf(\" \\| total mismatched words %llu\\n\", total_bad); | \n|  | unsigned long long nr = 0; CK(hipMemcpy(&nr, d_nrec, sizeof nr, hipMemcpyDeviceToHost)); | \n|  | const unsigned long long upto = std::min(nr, cap); | \n|  | if (upto > seen) { | \n|  | std::vector<Rec> rs(upto - seen); | \n|  | CK(hipMemcpy(rs.data(), d_recs + seen, sizeof(Rec) * rs.size(), hipMemcpyDeviceToHost)); | \n|  | std::map<std::pair<unsigned long long, unsigned int>, std::vector<Rec>> groups; | \n|  | size_t invalid = 0; | \n|  | for (const Rec & r : rs) { | \n|  | if (r.ci >= cases.size() \\|\\| r.idx >= cases[r.ci].cwords) { invalid++; continue; } | \n|  | groups[{r.launch, r.ci}].push_back(r); | \n|  | } | \n|  | if (invalid) { printf(\"  %zu records with an invalid case or index skipped\\n\", invalid); } | \n|  | for (auto & g : groups) { describe(cases, g.first.first, g.first.second, g.second, blocks, threads); } | \n|  | seen = upto; | \n|  | } | \n|  | if (nr > cap) { printf(\"  record buffer full: %llu mismatches seen, %llu recorded; event geometry above is TRUNCATED\\n\", nr, cap); } | \n|  | if (stop_on_mismatch && total_bad) { printf(\"STOPPED at the first mismatch, t=%.0fs\\n\", el); fflush(stdout); break; } | \n|  | fflush(stdout); | \n|  | } | \n|  | if (done) { break; } | \n|  | } | \n|  | } | \n|  | unsigned long long launches = 0; for (auto & cs : cases) launches += cs.launches; | \n|  | printf(\"RESULT %s: %llu GEMM launches, %llu mismatched words\\n\", total_bad ? \"MISMATCHES\" : \"CLEAN\", launches, total_bad); | \n|  | return total_bad ? 1 : 0; | \n|  | } |", "url": "https://wpnews.pro/news/gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption", "canonical_source": "https://gist.github.com/roverdrubber/7cd347a9340abdad2f707f2252aaaee0", "published_at": "2026-09-18 18:12:03+00:00", "updated_at": "2026-09-20 01:54:15.547199+00:00", "lang": "en", "topics": ["ai-chips", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["AMD", "rocBLAS", "RX 7900 XTX", "RDNA3", "gfx11", "HIP", "ROCm"], "alternates": {"html": "https://wpnews.pro/news/gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption", "markdown": "https://wpnews.pro/news/gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption.md", "text": "https://wpnews.pro/news/gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption.txt", "jsonld": "https://wpnews.pro/news/gemmcheck3-self-checking-rocblas-gemm-test-rx-7900-xtx-corruption.jsonld"}}