DWO 1 I am currently comparing CUDA and ROCm on my local network. I am not finding as many issues with ROCm as I had read about even a couple months ago. I am considering going all in on AMD when zen 6 comes out for consumers. Does anyone think this is a bad idea? I know CUDA and NVIDIA is easier. But for just some small local models, the gap seems to be pretty small now. What do you guys think?
lambda 2 Depends on the GPUs that you’re planning to buy?
In my experiments with the AMD hardware that I own (Dual 7900XT), it’s pretty decent for inference although the prompt processing speed is a fair bit slower than comparable Nvidia hardware (RTX 3090).
ROCm isn’t that big of a headache for generic inference. I’m not sure about training.
Depending upon your local pricing, 2x R9700 are a good deal for the VRAM capacity for pure inference workloads.
If you need performance or better ecosystem, Nvidia is the only option. Lastly, I’m not sure what has this got to do with Zen6. 1 Like
DWO 3 Yea, thanks for that. At the moment I have 3 different machines. My workstation which has a 4060 ti only 8gb of vram so I have been playing around with some small models on that. My server has a 2070 also with 8gb of vram. And I built a steam machine with a radeon 9060 xt with 16 gb of vram. So that is my best gpu. I have only just started playing with that. I realized it is also good as a test bed for trying out ROCm and also getting more familiar with Linux and testingout some python scripts. I run Bazzite on that. Ubuntu on the server. Everything is connected via my 2.5gb network, so I am going to try out a MoE kind of set up to see how it all works together. The point is to experiment with it and get better with Linux and using models on ROCm. I should explain what I meant by zen 6. If all goes well with this little experiment and ROCm isn’t too painful an experience then I am willing to upgrade my system to something like a dual r9700 rig with a strix halo as the orchastator. Or even wait for gorgen or medusa halo. Since i may not upgrade to mid to late 2027, zen 6 will come out so i am thinking of waiting for that before upgrading. I guess I’d like to know other people’s experience with ROCm vs CUDA and where they land on the issue. AMD is just cheaper, which is why I am kinda hoping ROCm isn’t as painful as I’ve heard so I can get off the NVIDIA ecosystem and save myself a little money. Sorry, long post but I’m new to all this and there are a LOT of opinions online. Please share any experiences you guys have had with your builds. I’m very interested. Thanks for replying mate!
Personally, I don’t think the CPU will actually matter here. Everything that matters with inference happens on the GPU.
If you bought an R9700 today, it would just be faster than your 9060 XT, basically everywhere, and it would work fine with whatever CPU that you have. But ultimately you should set goals for what you want to do with the models locally, whether it’s a gentle coating or something like open claw, where it’s going to read and summarize your messages, or send messages for you, or watch stock tickers, whatever it may be. It’s important to understand your scope as that will help you to take out hardware that will fit your needs. 1 Like
DWO 5 Yes you’re absolutely right. A different CPU wouldn’t really matter here. I have been playing around with a lot of little models over my little network. It’s been fun. Getting a better picture of what kind of hardware I will be aiming for in 2027… Looking forward to CES next year. Lots of cool hardware hopefully coming out
Don’t be too hyped for CES next year; likely we get NOTHING at all as there is 0 incentive currently for companies to release “Peon” hardware. Everything is based around AI.
The most that might come out is some Zen6 CPUs sometime next year; which won’t do anything for inference… the AI Bubble has to pop in order for prices to start to come back down to where any normal people would be able to afford them.
Since we have been in this thread; the price of a R9700 has gone up 25%… and it likely will continue to rise now that the AMD platform is mature.
DWO 7 Yea… I’m still excited mate… The tech will still come out. The prices will eventually go down, and in the meantime I can still plan for when they do. The AI boom is forcing the tech companies to try and compete at least… Sure it sux right now but the tech will still get better, and that’s a good thing. It ain’t all doom and gloom. It’s forced me to use a bunch of old motherboards and shitty ram and an old case .. like old old and build a steam machine… Which is better than the steam machine… The only new component was a 9060 xt..so a small victory for me
How are tech companies competing? They are only building products for enterprise. There has never been a time before where it is worse for consumers since the beginning of computers existence.
Reusing and recycling hardware is fine, there is nothing wrong with it.
I’m just saying your timeline isn’t correct and there is no reason for a normal person to be hyped.