While working on an OpenACC GPU port of a CFD solver with OpenACC, I wanted to maintain bit-reproducibility between existing CPU calculations and GPU-accelerated kernels — particularly for mathematical transcendental functions (sin
, cos
, exp
, etc.).
I tested the bitwise-reproducible algorithms from Arteaga et al. (2014). By wrapping the original bitrep
C++ source with a Fortran interface and adding OpenACC/OpenMP directives (plus disabling compilers fused multiply-add optimizations) I achieved reproducibility between CPU and GPU in pure transcendental kernels.
Here a few note:
- Bit-reproducible results have been tested on several architectures of NVIDIA and AMD GPUs.
- Performance penalty ~3-4× slowdown compared to standard math libraries
- Equivalent performance on OpenACC/OpenMP with NVHPC SDK compiler (as expected)
- Haven’t yet tested directly in CUDA/HIP kernels to determine if the performance loss could be reduced by removing the OpenACC/OpenMP overhead
The GPU Fortran interface and the tests I used are available on: GitHub - loicreynier/bitrep: Bit-reproducible CPU/GPU math transcendental functions with Fortran interface · GitHub
I’m curious if anyone has explored similar of different approaches for achieving bit-reproducibility in GPU or even CPU codes?
2 Likes
MetMan 2 Hi, LoicReynier.
I’m also porting a weather model to a GPU platform project, using the bitrep library to achieve bitwise reproducibility verification across CPU and GPU platforms. Tests passed with the NVIDIA HPC SDK (CPU vs GPU) and with Flang (Sugon CPU vs GPU). My bitrep github repo: qingu/bitrep: reproducible CPU/GPU transcendental functions with Fortran interface
Have you tested bitwise reproducibility across different compilers on the CPU platform?
CPU IEEE float and GPU IEEE float are different, so a cross-backend mismatch is actually expected. A good way to (kinda) fight it is through same-backend regression and analysis instead.