VFIO Pass through working on 9070XT

Hi All,

I just wanted to share some updates, that might help anyone struggling with their 9070XT.

My initial process was:

#!/bin/bash

systemctl stop lightdm.service
sleep 2

echo 0000:10:00.0 > /sys/bus/pci/drivers/amdgpu/unbind
sleep 2

 0000:10:00.1 > /sys/bus/pci/drivers/snd_hda_intel/unbind
#sleep 2

echo 3 > /sys/bus/pci/devices/0000:10:00.0/resource2_resize
sleep 2

systemctl start lightdm.service

I haven’t actually used my VMs for several months, fast forward to a couple of days ago, and when i tried it, it no longer works the same. I can use the script, start the VM, and the VM starts with the GPU attached, but upon exiting, the driver would no longer grab the GPU back, meaning the only way i could run another VM or reset my GPU was by doing a hard reset. According to Google AI this is because between 6.12 and 7.2 there have been changes to the hardware scheduler and power management.

However, with google’s help I was able to come up with the following script, which has been added to the libvirt/hooks directory. And now it automatically does everything. I can start and stop VMs without a hard reset. I have used it a bunch of times and so far, it has worked every time.

#!/bin/bash

GPU_PCI="0000:10:00.0"

ACTION="$2"
PHASE="$3"

if [ "$ACTION" = "prepare" ] && [ "$PHASE" = "begin" ]; then
    systemctl stop lightdm.service
    sleep 2

    if [ -e "/sys/bus/pci/devices/$GPU_PCI/driver" ]; then
        echo "$GPU_PCI" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null
        sleep 2
    fi

    echo 3 > /sys/bus/pci/devices/$GPU_PCI/resource2_resize 2>/dev/null
    sleep 1
    
    systemctl start lightdm.service
fi

if [ "$ACTION" = "release" ] && [ "$PHASE" = "end" ]; then
    systemctl stop lightdm.service
    sleep 2

    echo "amdgpu" > /sys/bus/pci/devices/$GPU_PCI/driver_override 2>/dev/null
    echo "$GPU_PCI" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
    sleep 2

    systemctl start lightdm.service
fi

So, I wanted to share it in case it can help anyone else.