# Native vLLM and ROCm 7.15 for RX 6000 (RDNA2) on Windows 11 – 26 Tflops FP16

> Source: <https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2>
> Published: 2026-08-18 10:11:51+00:00

Native vLLM + ROCm 7.15 (TheRock) for the whole AMD Radeon RDNA2 family on
Windows — **no WSL2, no NVIDIA, no compiler**. One-click installer, everything
prebuilt, and an OpenAI-compatible chat server that looks and works like the
NVIDIA stack.

**Verified on AMD Radeon RX 6750 XT 12 GB (gfx1031) — Windows 11 native — August 2026**

| Result | Number |
|---|---|
| rocBLAS FP16 GEMM (native bench) | 25 674 Gflops ≈ 26 TFLOPS |
| vLLM decode, Qwen3.5-4B 4-bit | ~58-62 tok/s (8.3 → 62.5 = 7.5× optimized) |

| Item | Requirement |
|---|---|
| OS | Windows 10/11 (Windows 11 recommended; `tar` must support zstd — automatic on Win11) |
| GPU | AMD RDNA2 — RX 6400 / 6500 / 6600 / 6650 / 6700 / 6750 / 6800 / 6900 / 6950 (all XT/M variants), 8+ GB VRAM for the 4B model |
| Driver | AMD Software: Adrenalin Edition (the normal gaming driver) |
| Disk | ~25 GB free on `C:` |
| Internet | only during install (~6 GB: stack ~2.3 GB + model ~3.8 GB) |
| Admin | one UAC click (installer auto-elevates) |

No compiler, no ROCm installer, no manual setup — everything ships prebuilt.

-
Download the repository (ZIP or

`git clone`

). You do**not** need to download the release archives (`*.tar.zst`

) manually — the installer fetches them automatically from the[Releases](/sebastianmechno-sys/vllm-rocm-windows-rdna2/releases)tab. -
**Double-click**— by default it downloads from this repo's releases; pass a GitHub username to use your own fork instead. It checks GPU + disk, then installs everything it does not already have (re-run is always safe and fast):`INSTALL.bat`

Step Action 1/6 GPU detection (warns if not RDNA2) + Python 3.11.9 2/6 4 archives from GitHub Releases → `C:\Python311`

,`C:\TheRock`

,`C:\vw_*_build`

3/6 venv fix + torch self-check 4/6 Qwen3.5-4B 4-bit model (skipped if already in HuggingFace cache) 5/6 writes `config.bat`

(the file to edit to change model later)6/6 verification benchmark -
→ opens the web chat in your browser (starts the model server automatically the first time). While the model thinks you see a small`CHAT.bat`

*Thinking…*spinner; when it's done, only the final answer streams in — the internal reasoning stays hidden. Tokens stream live with a tok/s counter, everything local on your AMD GPU. -
→ starts the model server on its own (OpenAI-compatible API on`SERVE.bat`

`http://127.0.0.1:8000/v1`

, like`vllm serve`

on NVIDIA). Use it with any OpenAI client, or just run`CHAT.bat`

. -
→ all 3 verification checks in one run: ROCm GPU detection, native rocBLAS FP16 power (`VERIFY.bat`

**~26 TFLOPS**) and the full 512-token vLLM benchmark (**~58-62 tok/s**).

| Test | Config | Result |
|---|---|---|
| rocBLAS FP16 GEMM | 4096×4096×4096, rocblas-bench.exe | 25 674 Gflops (≈26 TFLOPS) |
| vLLM Qwen3.5-4B decode | 512 tok, greedy, CUDA graphs | 59.4 tok/s (up to 62.5) |
| Optimization progression | eager fp16 baseline | 8.3 → 62.5 tok/s (7.5×) |

Full optimization history:

| # | configuration | tok/s |
|---|---|---|
| 1 | fp16 eager (baseline) | 8.3 |
| 2 | + CUDA graphs + skinny GEMV | 24.4 |
| 3 | + AWQ 4-bit quantization | 29.9 |
| 4 | + native HIP W4 GEMV kernel | 35.9 |
| 5 | + M=1 GEMV for lm_head | 58.1 |
| 6 | + direct-store kernel path | 59.1 |
| 7 | + weight-cast caching | 62.5 |

**TheRock** builds ROCm (HIP runtime, rocBLAS, Tensile) as native Windows binaries — this is what makes ROCm exist on Windows at all.`HSA_OVERRIDE_GFX_VERSION=10.3.0`

presents any RDNA2 GPU as gfx1030; the HIP kernel ships as a**fat binary (gfx1030 + gfx1031 + gfx1032)** so the whole RX 6000 series runs native code.- PyTorch 2.12
`+rocm7.15`

links against that runtime →`torch.cuda.is_available() == True`

on RDNA2 Windows. - vLLM plugin
`vllm_windows_rocm`

registers the tuned kernels: native HIP W4 GEMV for quantized linears, M=1 skinny GEMV for dense ones (including the huge tied lm_head), CUDA-graph safe (registered as real torch ops). `INSTALL.bat`

(engine:`INSTALL.ps1`

, manifest:`MANIFEST.json`

) downloads the 4 prebuilt archives from GitHub Releases and the model from HuggingFace, installs base Python, fixes the venv, verifies with a benchmark. Idempotent: it only downloads what is missing.

```
C:\Python311                                 Python 3.11.9
C:\TheRock\.venv                             torch 2.12+rocm7.15 venv (vLLM 0.19.1)
C:\TheRock\build\dist\rocm                   ROCm runtime libraries
C:\TheRock\ROCM_VLLM_RUNTIME                 vLLM + plugin + rocBLAS + rocblas-bench
C:\vw_cext_build, C:\vw_hipgemv_build        native HIP kernels
%USERPROFILE%\.cache\huggingface             model weights
```

Release archives (this repo's **Releases** tab, tag `v2.0`

):

| Archive | Size | Content |
|---|---|---|
`the-rock-venv.tar.zst` |
1.34 GB | torch ROCm venv |
`therock-rocm-dist.tar.zst` |
0.85 GB | ROCm runtime |
`vllm-stack.tar.zst` |
0.14 GB | vLLM + plugin + rocBLAS + rocblas-bench.exe |
`native-kernels.tar.zst` |
~1 MB | HIP GEMV kernels (fat binary) |

```
vllm-rocm-windows-rdna2-oneclick/
├── INSTALL.bat               one-click installer (entry point)
├── INSTALL.ps1               installer engine (downloads, extracts, verifies)
├── CHAT.bat                  opens the web chat (auto-starts the server)
├── SERVE.bat                 starts the model server alone (OpenAI API)
├── VERIFY.bat                all 3 verification checks in one run
├── chat.html                 the browser chat UI (Thinking spinner + tok/s)
├── MANIFEST.json             release archive names + extract targets
├── plugin_overrides/         tuned plugin modules (awq_gemv, bf16_gemv)
├── kernels/                  native HIP W4 GEMV source + prebuilt fat binary
├── scripts/                  model server entry + benchmark + rebuild helper
├── assets/                   verification screenshots
└── results/                  raw benchmark logs + progression chart
```

Edit `config.bat`

(written by the installer): set `SERVED_MODEL`

to the model
folder and `MODEL_NAME`

to the name shown in the chat / used by the API, then
run `CHAT.bat`

again. The benchmark uses `BENCH_MODEL`

(same file):

``` python
C:\TheRock\.venv\Scripts\python.exe -c "from huggingface_hub import snapshot_download; print(snapshot_download('<owner>/<repo>'))"
```

prints the snapshot folder to put in `config.bat`

.

| Symptom | Fix |
|---|---|
`No module named 'vllm._C'` warnings |
expected — the Windows plugin loads native kernels instead |
| Python installer exit 1601 | automatic NuGet fallback kicks in; nothing to do |
| extraction "Can't unlink" errors | close stray Python processes, re-run `INSTALL.bat` (it resumes) |
| low tok/s | close other GPU workloads; verify `overall_tok_s` ≥ 55 on a cold GPU |
| non-RDNA2 GPU | installer warns; unvalidated outside RDNA2 |
`tar` says "Unrecognized archive format" on the `.zst` files |
old Windows 10 without zstd — install Windows updates (bsdtar 3.5+ bundles libzstd; Windows 11 is fine) |
| chat page says "Server error" | the model is still loading — wait for "Application startup complete" in the SERVE window (~1 minute the first time) |
| custom debugging | `set BENCH_MODE=eager` , `set VLLM_WIN_HIPGEMV=0` , `set VLLM_WIN_BF16_GEMV=0` |

`scripts\rebuild_kernel.py`

recompiles `kernels/src/gemv_w4.cu`

(needs HIP SDK

- MSVC). Only for non-RDNA2 targets or exotic torch ABIs; the shipped fat binary covers all RDNA2.

Built on ROCm/TheRock, PyTorch ROCm and the vLLM project. Not affiliated with AMD.