NVIDIA GPU & CUDA Benchmark Suite
Automate CUDA workloads • HPL & HPCG • MLPerf inference • Custom kernels • Interactive reports
pip install nvprobe && nvprobe setup && nvprobe run --local
Bandwidth | MatMul / Attention | Conv2D | |---|---|---| | H2D / D2H / D2D across buffer sizes | fp32, fp16, int8 custom CUDA kernels | 2D convolution benchmarks | HPL (FP64 Linpack) | HPCG | MLPerf Inference | | Datacenter GPUs: A100, H100, B200, L40S… | Conjugate Gradients | ONNX Runtime via cmx4mlperf |
Bundled CUDA runtime— CuPy[ctk]
via pip, no system toolkit requiredAuto-downloaded HPC tools— NVIDIA HPC Benchmarks cached in~/.nvprobe/tools/
Interactive HTML reports— Chart.js charts with GPU / transfer / precision dropdowns and oscilloscope-style glow** A/B comparison**— compare two result sets side-by-side** Slurm integration**— generate, submit, monitor, collect from HPC clusters** SQLite storage**— all results persisted; CSV / JSON export
| Step | Command | What it does |
|---|---|---|
| 1 | pip install nvprobe |
|
| Install the package | ||
| 2 | nvprobe setup |
|
| Install CuPy, download HPL/HPCG, generate configs | ||
| 3 | nvprobe env |
|
| Verify GPU detection, driver, CUDA version | ||
| 4 | nvprobe run --local |
|
| Run all benchmarks locally | ||
| 5 | nvprobe report --open |
|
| Generate & open interactive HTML report |
Or from source:
git clone https://github.com/SergioZ3R0/nvprobe.git && cd nvprobe
pip install -e . && nvprobe setup && nvprobe run --local
| Command | Description |
|---|---|
nvprobe compare --a results/run1 --b results/run2 |
|
| Compare two runs | |
nvprobe run --config configs/cluster.yaml |
|
| Run with custom YAML config | |
nvprobe slurm submit --config configs/cluster.yaml |
|
| Submit Slurm job | |
nvprobe slurm status |
|
| Check Slurm job status | |
nvprobe setup --cuda 13 |
|
| Setup with specific CUDA version |
Chart.js canvas-based charts with interactive controls:
Bandwidth— filter by GPU and transfer type (H2D / D2H / D2D)** MatMul / Attention**— filter by GPU and precision (fp32 / fp16)** Range slider**— zoom into any x-axis region** Moving average**— smoother trend lines for dense data
name: my-run
gpu:
models: ["L40S", "B200"]
slurm:
enabled: true
partition: gpu
gpus_per_node: 8
precisions: [fp32, fp16]
benchmarks:
- name: bandwidth
params:
sizes_mb: [1, 4, 16, 64, 256, 1024]
- name: custom
params:
kernels: [matmul, attention]
nvprobe/
├── nvprobe/
│ ├── cli.py # CLI entry point
│ ├── config.py # YAML config
│ ├── runner.py # Benchmark orchestration
│ ├── slurm.py # Slurm job management
│ ├── reporter.py # Plotly HTML report generator
│ ├── db.py # SQLite storage + CSV/JSON export
│ └── benchmarks/
│ ├── base.py # Base class, GPU detection, diagnostics
│ ├── bandwidth.py # Memory bandwidth tests
│ ├── custom.py # Custom CUDA kernels
│ ├── hpl.py # HPL wrapper
│ ├── hpcg.py # HPCG wrapper
│ ├── mlperf.py # MLPerf via cmx4mlperf
│ └── _cuda/ # Raw CUDA kernels
├── configs/
│ ├── default.yaml
│ └── local.yaml
├── nvprobe.svg
├── index.html
├── README.md
└── pyproject.toml
HPL / HPCG— NVIDIA HPC Benchmarks binaries are validated for datacenter GPUs (A100, H100, B200, L40S…). They may crash (SIGSEGV) on RTX series. Bandwidth and custom kernels work on any CUDA GPU.MLPerf cuDNN—mlcr
discovers cuDNN via system CUDA paths. If installed viapip install nvidia-cudnn-cuXX
, pre-register with:mlcr get,cudnn,nvidia --input=$(python3 -c 'import nvidia.cudnn; print(nvidia.cudnn.__path__[0]'))
Python 3.10+ • NVIDIA GPU with CUDA drivers • nvidia-smi
in PATH • Slurm (optional)