LLM under Proxmox with GPU passthrough?

Forum users on Level1Techs report that running local LLMs on a Proxmox host with an Nvidia RTX 3090 passed through to a VM works without known blockers, with one user saying their local agents run in a Proxmox VM with "slightly higher average TPS than expected." A second user describes two approaches: installing drivers on the host and passing through device paths to multiple machines, or passing through the entire PCIe device so one VM gets exclusive access, noting they have done both for Jellyfin transcodes and a virtual gaming PC. A third user runs a 3090 in a vLLM container on Unraid, with agents reaching it over a URL and port via Tailscale.

Hello fellow nerds, I have a Proxmox machine with a 5950X and 128GB of DDR4. I am interested in putting a 3090 in, passing it through to a VM, and running Strata with Qwen 3.8 Flash Next IQ3 S. Are there any known blockers for this? Or potential Proxmox related performance bottlenecks? Thanks Don’t believe so, should work just fine My local agents are all setup in a VM on my proxmox server and it was a pretty seamless experience. I am getting a slightly higher average TPS than expected as well 1 Like nand https://forum.level1techs.com/u/nand 3 Yeah worked fine for me with Nvidia. You could even do drivers on the host and pass through paths to machines that need the card. I’ll call it method 1 I did method 1 for Jellyfin transcodes in the past. You can also pass through the entire pcie device method 2 if you’d like a vm to get it exclusively, I’ve done that too for a virtual gaming pc. I haven’t tried cuda specifically but I think it’s at a higher level of the stack and should work. So tl;dr: +1 to @ItTakes2ToMango https://forum.level1techs.com/u/ittakes2tomango I use unraid, and have my 3090 in a vllm container. Then my agents hit that url and port to facilitate. Tailscale is based.