[SOLVED] AMD RX 6600 GPU passthrough: VFIO NULL pointer dereference on VM shutdown — one-line AMDGPU fix

A user on an ASUS X99-WS/IPMI workstation with an Intel Xeon E5-2680 v4 and an AMD Radeon RX 6600 (Navi 23) reported that a one-line patch to AMDGPU's device cleanup code stopped a kernel NULL pointer dereference in vfio_pci_core_sriov_configure that crashed Ubuntu 26.04 (kernel 7.0.0-38-generic) whenever a Windows 11 QEMU/KVM guest using VFIO-PCI passthrough was shut down. Instrumented vfio-pci logging showed the VFIO device pointer was valid at probe and registration (vdev=ffff8c1a90c46800) but both vdev and drvdata were null on removal, and the author said the crash also occurred on a 6.17 kernel. The author stated the diagnosis and fix relied heavily on AI assistance and are not fully verified, and presented the result as an experiment rather than an authoritative kernel debugging guide.

Hi everyone I’ve been working on getting GPU passthrough running reliably on my Linux workstation, allowing me to switch between my Ubuntu desktop and a Windows 11 VM using the same physical GPU. The Windows VM itself was working perfectly. The RX 6600 was successfully passed through, Windows recognized it, and all my monitors switched over. The problem appeared when shutting down the VM. Instead of returning the GPU to the Linux host, the kernel would crash with a NULL pointer dereference inside vfio pci core . After spending quite some time debugging this, I found a potential root cause in AMDGPU’s device cleanup code and successfully tested a one-line patch. I thought I’d share the findings in case someone else runs into the same problem. I should probably mention that I did most of this with a fairly superficial understanding of what was actually happening inside the Linux kernel. I relied heavily on AI throughout basically the entire process, from analyzing kernel logs and reading source code to identifying the suspected cause and building the patched module. My contribution was mostly running commands, providing feedback, and testing the results. While the fix worked on my machine, I wouldn’t automatically assume that every technical explanation in this post is 100% accurate. A significant part of it is based on AI-assisted analysis that I am not personally qualified to fully verify. I also wouldn’t experiment with something like this on a system I actually depend on. This is my personal workstation, and I’m perfectly willing to break things, reinstall Linux, or spend a weekend figuring out why something stopped working. So please treat this as an interesting experiment and a potentially useful starting point, rather than an authoritative kernel debugging guide. Corrections from people who understand the internals better than I do are very welcome Motherboard: ASUS X99-WS/IPMI CPU: Intel Xeon E5-2680 v4 GPU: AMD Radeon RX 6600 Navi 23 Host OS: Ubuntu Linux Kernel: 7.0.0-38-generic Ubuntu 26.04 Virtualization: QEMU/KVM with libvirt and virt-manager Guest: Windows 11 GPU passthrough: VFIO-PCI, managed by libvirt The workstation has no integrated GPU. The RX 6600 is the only graphics card, so returning it to the Linux host after VM shutdown is essential. The GPU and its audio function are passed through as PCI devices: 0000:03:00.0 — AMD Radeon RX 6600 0000:03:00.1 — AMD HDMI/DP audio Starting the Windows VM worked. Libvirt detached the RX 6600 from AMDGPU, VFIO took ownership, and Windows operated normally. However, when shutting down Windows, the kernel would crash while VFIO was being removed. The relevant error was: BUG: kernel NULL pointer dereference, address: 0000000000000470 RIP: vfio pci core sriov configure+0x19/0x1f0 CR2: 0000000000000470 Call Trace: vfio pci core unregister device vfio pci remove pci device remove device remove device release driver internal device driver detach unbind store The interesting part was that the crash consistently happened during device removal, not during GPU initialization or while Windows was running. I had also encountered the problem with a 6.17 kernel. I downloaded the matching Ubuntu kernel source and built a modified vfio-pci module containing additional logging. The instrumentation recorded the VFIO device pointer during initialization, after successful registration, and when the removal callback was entered. During VM startup, everything looked correct: 18:24:25 vfio-pci 0000:03:00.0: VFIO-DEBUG: probe initialized vdev=ffff8c1a90c46800 drvdata=ffff8c1a90c46800 18:24:25 vfio-pci 0000:03:00.0: VFIO-DEBUG: registration succeeded vdev=ffff8c1a90c46800 drvdata=ffff8c1a90c46800 Both pointers were valid and identical. However, during VM shutdown: 18:32:08 vfio-pci 0000:03:00.0: VFIO-DEBUG: remove entered vdev null=1 drvdata null=1 Immediately afterward, the kernel crashed. So something had cleared the PCI device’s driver-private data after VFIO had successfully registered the GPU. That pointed toward a driver lifecycle problem rather than a GPU reset problem. While examining the AMDGPU driver source, I found this function in: drivers/gpu/drm/amd/amdgpu/amdgpu kms.c void amdgpu driver release kms struct drm device dev { struct amdgpu device adev = drm to adev dev ; amdgpu device fini sw adev ; pci set drvdata adev- pdev, NULL ; } The last line immediately caught my attention: php pci set drvdata adev- pdev, NULL ; This clears the PCI device’s driver-private pointer. The important detail is that amdgpu driver release kms is a DRM device release callback, not the PCI driver’s removal callback. DRM devices are reference-counted, meaning their final release can happen some time after the PCI driver has already been detached. In my logs, AMDGPU cleanup was apparently still happening after VFIO had registered the device: 18:24:25 VFIO successfully registers the GPU 18:24:39 "ttm finalized" 18:32:08 VFIO removal receives NULL drvdata The ttm finalized message was especially interesting because it appeared under the vfio-pci device prefix, despite originating from AMDGPU cleanup code. My working theory is that an old AMDGPU DRM object was released after VFIO had already attached to the same PCI device. When that happened, AMDGPU’s release callback could clear the driver-private pointer now belonging to VFIO. Later, when VFIO attempted to unregister the device, it retrieved NULL and crashed. I haven’t captured the exact write with a hardware watchpoint, so the timing relationship is not conclusively proven. However, the source code, the logs, and the result of the fix strongly support this explanation. The proposed fix is surprisingly simple: remove the PCI driver-data clearing operation from AMDGPU’s delayed release callback. diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu kms.c --- a/drivers/gpu/drm/amd/amdgpu/amdgpu kms.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu kms.c @@ -1608,7 +1608,6 @@ struct amdgpu device adev = drm to adev dev ; amdgpu device fini sw adev ; - pci set drvdata adev- pdev, NULL ; } The Linux driver core already clears driver-private data when unbinding a PCI driver, so the additional clearing in this delayed callback appears unnecessary and potentially unsafe. After identifying this, I discovered that Matthew Auld had already proposed essentially this exact change in May 2024, in a patch titled: PATCH 02/20 drm/amdgpu: don’t trample pdev drvdata The patch describes the same driver-data lifetime issue. It appears that the problematic statement is still present in my Ubuntu kernel source. Reference: Original upstream patch discussion https://www.mail-archive.com/search?f=1&l=amd-gfx%40lists.freedesktop.org&o=newest&q=from%3A%22Matthew+Auld%22 I applied the change to the matching Ubuntu linux-source-7.0.0 source and rebuilt the AMDGPU kernel module against the installed kernel headers. For this kernel, I used an external-module build with an additional include path for AMDGPU’s tracing headers: make -j4 \ -C /lib/modules/$ uname -r /build \ M="$PWD/drivers/gpu/drm/amd/amdgpu" \ KCFLAGS="-I$PWD/drivers/gpu" \ modules The resulting amdgpu.ko compiled successfully. I then installed it as a module override under: /lib/modules/7.0.0-38-generic/updates/amdgpu.ko Ubuntu’s module search configuration prioritizes updates/ , allowing the patched module to load while preserving the original distribution module for rollback. After rebuilding the module dependency database and rebooting, I repeated the GPU passthrough test. It worked The Windows 11 VM started, the RX 6600 was passed through successfully, and Windows operated normally. After shutting down the VM, the GPU returned to the Linux host without the VFIO NULL pointer crash. This was the first successful test with the patched driver. I still want to perform additional start/stop cycles before calling it completely stable, but the original failure did not occur. This is a custom kernel module modification, not an official Ubuntu fix. I tested it on my particular hardware and kernel combination. I would not recommend blindly installing a custom AMDGPU module built for another kernel version. If you want to experiment with this, make sure you have: The exact matching kernel source and headers A backup of the original module SSH access or another recovery mechanism A way to recover if the GPU driver fails to initialize In my case, having only one GPU made remote access particularly important. Initially, I was convinced the problem was somewhere inside VFIO because that’s where the kernel crashed. The debugging logs revealed something more interesting: VFIO had correctly initialized its device context, but the pointer disappeared before removal. That eventually led me to AMDGPU’s delayed DRM cleanup callback. The actual change was just one deleted line. I’m posting this both to help anyone experiencing a similar issue and to get feedback from people who know the DRM and PCI driver lifecycle better than I do. Has anyone else encountered this behavior, or does anyone know why the original 2024 patch hasn’t made it into the kernel source I’m using? I’d be especially interested in knowing whether this is already addressed by another upstream change or whether the original patch could be revisited. Thanks for reading