Hi all,
I’m pretty new to linux, and saw a video, by @wendell using a framework Desktop with an external GPU, running an LLM considerably faster (and with more RAM)
I have a GMKtec Evo-X2 and have my external gpu working through oculink on my m.2 drive port.
I’m likely using the wrong tool (LM Studio), but when I activate my 2080ti (as a proof of concept) it seems like the 8060S cannot be used, and even with a LLM like a30-3B, it runs terribly (2-3 t/s) and cannot seem to see a way to get them working in parallel? On the 8060s, alone, it goes 10x that speed…
Any help or thought would be greatly appreciated, for a Linux novice!!
Cheers in advance, all!!
1 Like
ik_llama.cpp is what you want to use here. you can use a vulkan backend but for certain mixture of expert models you can run the dense layers on gpu and everything else is running from CPU.
you can kinda use ik_llama.cpp under wall if you’re using lm studio on windows.
5 Likes
Thats really helpful, thanks @wendell for the quick reply!! Are there any guides or other threads for doing this (sorry, I’m really new to all this and to “ik_llama.cpp”)
Does anyone think it’s possible to create a llama.cpp setup where we run dense layers on external GPU and we use the APU for the rest instead of CPU?
thats what some of our videos are about? not with strix halo but with ik_llama.cpp and this general approach. So yes, the general approach works fine.
with strix halo the problem becomes gpu bar space. I’ve had okay luck with 16/24gb vram gpus but rtx pro 6000 doesnt always work because resize bar, strix halo igpu, etc.
the minisforum pc not wanting to do the A1000 via pcie slot was also disappointing but it “can” work with fiddling. just obvious no one on the bios team (amd?) tested that or through strix halo would be used with a gpu.
the 20gb SFF 4000 plus these big-mem PCs are great tho
1 Like
I’m just now seting up my Framework Desktop (128GB) and the plan is to attach the 5080 that’s on my dock right now (USB4 and then Oculink when the Redriver PCIe adapter arrives), and hopefulyl soon replace it with a 5090.
Is 32GB too much for this to address?
I, too, will be trying to make a local LLM application work with both the 8060S and the Nvidia card, depending on the model. But anything more involved than LMStudio will be new and a learning experience for me so I’m hoping to figure all this out
Did you have any luck in the end for this? I am going to try and get this working in Windows / LM Studio over this weekend as a mini-project, so if I have success I will try and write up a full guide.
I managed to get only one or the other showing up. annoyingly, when I run through Vulcan, my RTX 2080 ti is the only GPU that is picked up… however, the RAM that is allocated to the GPU for the Strix halo is “invisible” all of a sudden… However, when setting the GPU ram to 512mb via bios, I can use the (paltry) 11gb ram from the e-GPU along with the “full” 128gb of available vram… just that the models run predictably about 35% to 40% of the speed they normally do…
Seems to be an issue for Vulkan… [LM Studio 0.3.16 Does not detect iGPU when dGPU present · Issue #705 · lmstudio-ai/lmstudio-bug-tracker] (LM Studio 0.3.16 Does not detect iGPU when dGPU present · Issue #705 · lmstudio-ai/lmstudio-bug-tracker · GitHub)
It sure would be amazing if someone as skilled as @wendell could make a vid / point us in the right direction, as I think there are quite a few of us who’d be very interested in matching an eGPU with their machine
Cheers (and to you, too, Wendell!!) So, an update…
I have had some small success in Jan.ai!!
Both GPU’s are visible, and “seem” to work in parallel (both gpu’s in Task Manager show utilisation) but still breaks with models over the GPU ram limit size i.e. if I load to the 2080ti first, then it goes bang… but fine if I load to the 8060S iGPU first…
1 Like
acerb 10 Any update on this topic?
4dj 11 Hey everyone, just coming on board here!
I currently run a DEG2 with a 5060 Ti 16GB on my MS-02 Ultra, but I just ordered the MS-S1 Max.
Researching the S1 Max is actually how I found this thread—I’m getting ready to set it up next week when it arrives.
I’m here to soak up all the Strix Halo knowledge this site has, and hopefully contribute back.
I won’t be pushing the absolute bleeding edge just yet :), but my workload involves a lot of heavy dense image matching and image models for GIS.
I’ve already done some POC testing with my Ultra/DEG2 setup.
Once the S1 Max gets here and I have the base system backed up, I’ll post some benchmarks and results in this thread.
I’m also planning to swap to an AMD GPU on the DEG2 shortly after for testing.
I wanted to bump this because it’s a great thread for eGPU and Strix setups, and I didn’t want a 9-month-old post to get locked or disappear.
If I missed a newer, more active thread about this exact setup, please point me in the right direction!
Thanks,
[gapl20](https://forum.level1techs.com/u/gapl20)
12 I just got the MS-02 Ultra but it doesn’t recognize my DEG2. How did you make it work? I am using an RTX 3090 and the DEG2 provided TB5 cable. I had tested with all 3 USB-C ports on the MS-02 Ultra just to verify but I get the same result.